Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work presents a study on applying 1) combining pseudo-labeling with entropy-based label filtering for representation learning and 2) novel-novel and base-novel manifold mixup with entropy-based filtering for adapting base representation to novel classes for improving few-shot image recognition tasks. Under appropriate hyperparameter settings, the proposed approach achieves competitive performance on standard few-shot image recognition benchmarks. Ablation studies are conducted to investigate the gains brought by individual techniques.
strengths_and_weaknesses: Strengths:

1) The paper is well-written and easy to follow. Approaches, experiment settings and implementation details are clearly described, in a way that helps reproducibility of the proposed work. Experiment results are well organized.
2) Experiments and ablation studies seem thorough. Standard benchmarks are used. Latest works that follow the same experimental settings are included as baselines. Necessary baselines are included. Ablation studies included all components of the system.
3) Experiment results seem to show highly competitive performance (without transductive learning).

Weaknesses:

1) While it is clear that the proposed approach worked, it is not very clear how it worked. Here are my recommendations
- For entropy-based filtering, try to fit the qualitative examples in the main paper
- Show some hard examples selected by mixup. Maybe something like (0.9x image A + 0.1x image B)
- Which classes often benefit from pseudo-labeing, mixup and entropy filtering?

2) While not necessary, it would be interesting to learn about the sensitivity of the proposed approach to hyperparameters. The use of different hyperparameters for 1- and 5-shot might be a deviation from classical setups? Although it's not entirely surprising that different hyperparameters might be necessary as the dataset size changes. It would help if measures are taken to control overfitting to specific benchmarks.


=============================

Author response adequately addresses my concerns.

questions: See weaknesses.
limitations: Limitations are addressed adequately.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper presents a hard manifold mix up process to augment the few-shot samples during fine-tuning for improving accuracy performance. The mix-up is carried out in different settings: novel-novel, novel-base and the hard samples from the mix up based on margin are utilized during fine-tuning.
strengths_and_weaknesses: The papers' idea of using manifold mix up is complemented by the use of hard sampling for further use during fine-tuning, which is simple and interesting.
questions: 1.  The paper says that the pretraining is done with  self supervised loss in addition to cross entropy, but the eq. 1 just shows cross entropy where is the full loss according to the description?
2. How exactly the entropy threshold for filtering pseudolabels for base classes is selected?
3. How this method is compared with the "Charting the Right Manifold: Manifold Mixup for Few-shot Learning" paper by Mangla et al. in WACV 2020?
4. Why the other standard benchmarks such as tiered-Imagenet or CUB is missing also different architechtures such as WRN? Also, are the methods compared are evaluated based on the same pretrained model? Over, how many episodes the average accuracies are reported?
5. What about domain-shift results for example mini to CUB setting as in previous papers such as "A closer look at few-shot learning" by Chen et al. 2020.

limitations: I do not see much improvement from this method in comparison to previous methods. In fact in many settings the improvement is negligible such as in Table 1. Moreover, sometimes it is not practical to have access to the base class samples during fine-tuning stages and the assumption. Therefore source free fine-tuning is not practical with this method.

The pseudolabel creation for base class samples with a classifier initially built on novel target samples is cumbersome as the label set do not match in most cases. Therefore, the filtering step on top of this assumption with entropy criterion does not make sense with a reasonably big domain between base classes and novel classes. Also, many standard benchmark datasets are missing in the evaluation such as tieredImagenet, CUB, domain shift experiments such as mini-imagenet to CUB.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents FeLMi, a type of mixup method for few-shot learning, with margin-based uncertainty criteria. The work aims to augment new data in a mixup form to tackle the overfitting problem in a few-shot classification setting. With this, the presented method consists of six phases. Overall, after pretraining on the base classes and obtaining the pseudo labels of the base examples, both novel-novel and base-novel mixup samples are generated for data augmentation. The pretraining method employs self-supervised-based Invariant and Equivariant Representation learning (IER) [19]. Finally, the evaluation is presented using three few-shot learning benchmarks.   
strengths_and_weaknesses: In general, the idea of the paper is interesting. Specifically, I like how the presented method is inspired by several recent works of base examples and mixtup method for few-shot image classification to present novel data-augmentation methods. Additionally, the paper is well-written and easy to follow. However, I think the method contains the following weaknesses:
- The overall model is complex (having several stages with different criteria).  
- The method uses many tricks, such as self-supervised learning for pretraining and active learning for the hard mixtup. Therefore, the method's evaluation and justification became more complicated compared to standard methods such as ProtoNet. 
- The proposed method is evaluated on three small datasets, but I think some large datasets are required.   
- Having a 5-way classification problem, the FeLMi can not gain significant accuracy in CIFAR-FS dataset. I think this is ok, but some extra evaluation might help us understand the proposed method's classification gain/loss. 
questions: - While understanding the use of standard transfer learning instead of an episodic training approach, I would like to see more discussion on why fine-tuning and why not episodic pre-training? 
- What about standard pretraining without using a self-supervised [19] training method?
- What about the time complexity of the method?
- Could you please elaborate more on “k-smallest margin values as hard examples” (line 194)?
- What about the fine-tuning implementation details? Also, do the literature uses MLP on top of ResNet12? (“we train a two layer MLP” ll. 214). If so, please cite. 
- Why “temperature coefficient” is set to 4.0 (line 217)? Is this obtained through cross-validation?
limitations: Though the presented method is proposed for few-shot learning, it is only evaluated with image classification. Additionally, evaluation with large datasets (such as Tieredimagenet) is missing. While the presentation of the paper is good, I think the author(s) can improve the related work by clearly discussing the difference between the approaches (such as Base examples for few-shot learning) with the proposed method. I think the conclusion can include some limitations and future works too.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, Few shot Learning with hard Mixup (FeLMi) is developed to mitigate the issue of data scarcity. The proposed method is composed of 6 steps. (1) Pretrain the model on the base dataset using the cross-entropy loss with auxiliary self-supervised loss. (2) Train a linear logistic regression model on the top of the learned feature extractor using the novel data to generate pseudolabels for the base dataset. (3) Filter the pseudolabels based on entropy. (4) Do novel-novel and base-novel mixup to generate more data. (5) Choose hard mixup samples based on the margins in classification probability. (6) Finetune the model using the filtered base data, novel data, and hard mixup data. Experiments indicate that the proposed method leads to improved performance in few-shot learning.
strengths_and_weaknesses: Strengths:

The paper is well organized and easy to follow.

The proposed method is straightforward and easy to understand. 

Sufficient ablation studies in the experiments.


Weaknesses:

Among the 6 steps in FeLMi, only Step 5 is original (choosing hard mixup samples). However, Table 4 indicates that Step 5 only leads to marginal improvement in few-shot classification accuracy.

The model is pretrained by cross-entropy loss and self-supervised loss. It is known that pretraining with auxiliary self-supervised loss leads to improved performance in transfer learning for downstream tasks. Although FeLMi achieves the best performance in Tables 1, 2, and 3, it relies on a stronger pretraining method. It does not demonstrate the effectiveness of hard mixup.
questions: According to Table 5, base-novel mixup leads to negligible performance improvement. I wonder if this step is necessary for your proposed method.
limitations: Compared with the simplest fine-tuning methods in few-shot learning (e.g. RFS-simple, ECCV 2020), the proposed method is more computationally expensive. Specifically, the proposed method must assign pseudo labels for the base data. It becomes an issue when the base dataset is huge. It’s better to provide an empirical or theoretical analysis of the time and space complexity.
review_title: null

Meta-review: metareview: The submission introduces an approach to few-shot learning called Few-Shot Learning with Hard Mixup (FeLMi) which, as its name suggests, applies hard manifold mixup as an augmentation strategy for adapting a pre-trained model to a small training set of downstream examples. The model is first trained on the base classes using a combination of supervised learning and Invariant and Equivariant Representation learning (IER), then a linear classifier is trained on top of the frozen backbone using the novel classes' support set and pseudolabels are generated for the entire base class dataset. Base class examples are filtered to exclude ones with low pseudolabel entropy (using a thresholding hyperparameter). Feature-level mixup is applied on base-novel and novel-novel example pairs, and the resulting examples are subsampled to the N hardest ones based on the difference in top-2 probabilities. The model is then fine-tuned on the pseudolabeled base examples, novel examples, and hard-mixup examples.

Results are presented on two CIFAR100-based few-shot classification benchmarks (CIFAR-FS, FC-100) and mini-ImageNet in the 5-way, 1-shot and 5-way, 5-shot settings. FeLMi is shown to outperform competing approaches. Ablation analyses are also presented to assess the contribution of various components on performance improvements.

Reviewers highlight the submission's writing quality and clarity (7gPu, 2Qz6, WHpk). Opinions are split on how straightforward the proposed approach is, with Reviewers 3CtS and WHpk noting its simplicity, and Reviewer 2Qz6 expressing concerns over its many moving parts. Opinions are also split on the significance of the performance improvements; Reviewer 7gPu finds FeLMi's performance competitive with competing approaches, and Reviewers 3CtS and WHpk are concerned that the improvements are modest. The authors respond by emphasizing that FeLMi is simple and effective, but Reviewer 3CtS remains eager to see a clearer performance gap. Reviewer 3CtS is also concerned that the approach is not source-free, to which the authors respond that the unlabeled data could also come from another source than the upstream training dataset.

Following the discussions, opinions remain divided among reviewers, although the majority is either leaning towards or strongly recommending acceptance. Reviewer 3CtS still recommends rejection, but is open to an acceptance recommendation. I therefore recommend acceptance.
confidence: Certain
Decision: Accept