FeLMi : Few shot Learning with hard Mixup
Authors: Aniket Roy
Anshul Shah
Ketul Shah
Prithviraj Dhar
Anoop Cherian
Rama Chellappa
Venue: NeurIPS 2022 Accept (2022)
Keywords: few-shot learning
mixup

Abstract: Learning from a few examples is a challenging computer vision task. Traditionally,
meta-learning-based methods have shown promise towards solving this problem.
Recent approaches show benefits by learning a feature extractor on the abundant
base examples and transferring these to the fewer novel examples. However, the
finetuning stage is often prone to overfitting due to the small size of the novel
dataset. To this end, we propose Few shot Learning with hard Mixup (FeLMi)
using manifold mixup to synthetically generate samples that helps in mitigating
the data scarcity issue. Different from a naïve mixup, our approach selects the hard
mixup samples using an uncertainty-based criteria. To the best of our knowledge,
we are the first to use hard-mixup for the few-shot learning problem. Our approach
allows better use of the pseudo-labeled base examples through base-novel mixup
and entropy-based filtering. We evaluate our approach on several common few-shot
benchmarks - FC-100, CIFAR-FS, miniImageNet and tieredImageNet and obtain
improvements in both 1-shot and 5-shot settings. Additionally, we experimented on
the cross-domain few-shot setting (miniImageNet → CUB) and obtain significant
improvements.

: Few shot Learning with hard Mixup
Aniket Roy
Anshul Shah Johns Hopkins University
Ketul Shah
Johns Hopkins University Baltimore, USA aroy28@jhu.ede edu
Johns Hopkins University Baltimore, USA cssah33@jhu. edu
Baltimore, USA ashah95@jhu edu
Prithviraj Dhar
Anoop Cherian MERL Cambridge, MA cherian@merl.com
Rama Chellappa
Reality Labs, Meta Sunnyvale, USA prithvirj$5tmeta.comm com
Johns Hopkins University Baltimore, USA rchella4ßjhu.en edu
Abstract
Learning from a few examples a challenging computer vision task. Traditionally, meta-learning-basee methods have shown promise towards solving this problem. Recent approaches show benefits by learning feature extractor on the abundant base examples and transferring these the fewer novel examples. However, the finetuning stage is often prone to overfitting due to the small size of the novel dataset. To this end, we propose Few shot Learning with hard Mixup (FeLMi) using manifold mixup to synthetically generate samples that helps in mitigating the data scareity issue. Different from a naïve mixup, our approach selects hard mixup samples using uncertainty- based criteria. To the best of our knowledge, we are the first to use hard-mixup for the few-shot learning problem. Our approach allows better use of the pseudo-labeled base examples through base-novel mixup and entropy-basee filtering. We evaluate our approach on several common few-shot benchmarks FC- 100, CIFAR-FS, minilmageNet and ticredImageNet and obtain improvements in both 1-shot and 5-shot settings, Additionally, we experimented on the cross-domain few-shot setting (minilmageNet CUB) and obtain significant improvements. Code: https: bttpps://githbb com/ an meiiet000/Felmm
Introduction
Supervised deep learning has revolutionized data-driven computer vision tasks, e.g., object detection, segmentation [5], etc. However, it is expensive to collect and annotate large amounts of data, especially in some fields like medical image diagnosis where one needs rely on experts for annotating the data. Despite recent progress in learning from large data, learning robust models using
few labeled examples still remains quite challenging. Few-shot learning (FSL) [1 28 33 deals
with this problem, where there exists base classes with plenty of training examples, and novel classes with only few training samples. The goal is to learn a representation from the base classes, then quickly and efficiently adapt to the novel classes from few examples.
There
paradigms for training few-shot learner (1) episodic training and (2) transfer learning. During episodic training, in each episode, a few examples from the training data are sampled to form the "support set", on which training is performed and evaluated test dataset. Generally, the model is trained for large number of episodes to get better generalization. On the one hand, this approach is both time consuming and also incurs inductive (prior knowledge driven bias) and sampling bias. On the other hand, recent approaches use simple transfer learning the novel classes and obtained comparative performance. Tian et al. [31] show that pretraining on the large scale base dataset to learn good representation, followed by finetuning on the few
the "query set" drawn from the
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

novel class examples using simple linear classifier significantly outperforms the more complicated meta-learning approaches in most of the FSL benchmarks. The performance further boosted using self-distillation and distillation across base and novel class representations [21].
Data scareity is fundamental issue in few-shot learning. To circumvent this, AssoAllin 1 uses the nearest neighbor base examples in addition to the novel examples with adversarial augmentation. Jian al. [1] generates pseudo-labels for the base class samples and retrains on the entire dataset by "hallucinating" the labels of the base class samples. However, literature in semi-supervised learning has shown that poorly aseudo-labeled samples degrade the performance [2T..
We overcome the above issues in these prior approaches using entropy filtering on the noisy pseudo- labels. This step helps improve the training by discarding noisy pscudo-labeled base samples during training. Next, we propose a solution to the data scarcity issue by synthetically generating training examples with both novel and base examples using a mixup strategy. The mixup samples generated with minimal overhead while adding the much needed regularization by synthesizing new points close the novel distribution. Motivated by the efficacy of hard example mining in computer vision [26]. we propose to improve the naive mixup step with uncertainty guided hard-mixup. Further, we propose a technique to make use of base samples for mixup in addition to the novel samples. Note that Mangla et. al [16] has also used mixup for few-shot learning, but the motivation is completely different in our setting. Their approach does not add any more samples training and use as a regularizer across layers. On the contrary, we are the first to propose generating hard mixup samples to tackle the data scarcity issue in FSL.
Another contribution of our work is that we redefine hard-mixup for the few-shot learning setup. Note that hard-mixup has been used in contrastive learning, where positive and negative classes already defined [2]. In contrast, for N-way K-shot problem, where there exists N classes, the notion of hard examples not straightforward. While, hard mixup similar to these approaches could be extended in our setup by mixing and assigning a hard label, we instead work with soft labels and select the optimal examples
mix with the appropriate lambda values. To this end, we define hard example mixing based on a margin- mased uncertainty measure, inspired by the concept of active learning. We use the margin to estimate the hardness of sample, thereby measuring the uncertainty in predicting a particular class with high probability.
In summary, the contributions of this work are as follows:
1. To handle the data scarcity problem in few-shot learning, we use mixuup-based data augmen- tation.
2. We perform hard mixup using a novel margin-based uncertainty measure, which further elevates performance. To the best of our knowledge, we are the first to use hard-mixup for few-shot learning.
3. We propose an approach MFi to make use of both base and novel examples for generating more samples for novel training.
4. We validate our approach through extensive experiments on standard few-shot benchmarks FC100, CIFAR-FS, minilmageNet and tieredImageNee demonstrating ttateoff-the-ar results. Our approach also demonstrates promising improvements on the standard minilmageNet + CUB cross domain benchmark.
2 Related Work
Few-shot learning methods can be divided into two broad categories, (1) Meta-learning, (2) Transfer learning methods.
Meta Learning. Traditionally, meta-learning based methods have been effective for few-shot learning tasks and usually consists of meta- meta-training and meta-testing phase. In both meta-training and testing phases, multiple episodes are sampled from a task distribution and the model is trained on the support samples of the episode and evaluated on the query set. Thus, the model is trained on unseen classes via a learning to learn approach. Meta-learning methods can also be subdivided into two categories, viz., metric-based meta learning and optimization- based meta-learning.
Metric-hased meta-learning methods predict the label of the query as a weighted sum the labels over the support samples. Popular metric-based meta-lcarning methods include Prototypical Networks 28).

---

17 etc. Optimization- based meta- learning methods adapt the model parameters using small number of gradient steps, e.g. MAML (9) LEO [3], etc. MetaOptNet [99 solves differentiable convex optimization problem for few-shot learning with better generalization. Other meta-learning methods use earth-mover's 's distance 38
Relation Networks [30], Matching networks [33]. TADAM
set-to-set functions [35,, human interpretable concepts [5], etc.
Transfer learning methods. Recent methods have shown that contrastive pretraining on large base example dataset and simple finetuning on the scarce novel examples performs surprisingly methods. As an example, RFS [31 outperforms all the
better than the complex mneaalearming-bon meta learning baselines by learning self- supesvipeeii representation followed by simple logistic classifier. Following similar trend, SKD [18] uses rotational self- sefe-sueeeeieed distillation further improve the performance. Rizve et al. [22] explores the complementary strengths of invariant and equivariant representations and self-distillation for better pretraining and performs significantly better than previous methods. Partner-assisted learning [5 which also uses feature level knowledge distillation has proven to be effective.
Base examples for few-shot learning. AssoAlign [ makes use of large base dataset by search- ing nearest neighbors of the novel examples in the pool of base examples along with adversarial associative alignment. pseudo-labeling is a well-known technique for semi-supervised learning and have been successfully used several vision tasks, e.g., image and video classification [2]] Jian et al. [II] used pscudo-labeling of the entire base examples and obtained sigmificant performance boost, Afrasiyabi et al. also used mixture- feature space learning and matching feature sets to further improve the performance.
Mixup Approaches. These are simple yet effective regularization techniques for training deep networks, Several variants of mixup have been proposed in the literature, viz., input mixup [39]. cutmix 37), manifold-mixup (32), etc. Manifold mixup uses feature level mixup, which provides smoother decision boundary and flattened class representations. Mangla et al. [16] use manifold mixup for few-shot learning from a regularization perspective. In contrast to the above methods, we are using manifold mixup to tackle the data scarcity problem in few-shot learning by generating more samples.
3 Problem Statement
We first define the few-shot classification problem addressed in this work. Typically, there exists large-scale labeled base dataset and novel classes with few samples per class. The task is to learn a feature representation on the base class with abundant samples and then discriminatively learn a representation recognize subsequent novel classes. Let the base dataset be denoted as Dbase (= , _ye A where x is the base class sample and the corresponding label ylase E Crase Similarly, the novel dataset is denoted by Duovel = {ahovel povell
} Nowe:
Jt=l where ot novvell is the novel class sample and the corresponding label is yoovel E Chovei The base and novel classes are disjoint, i.e., Chaas n C = 0, and [ > [
The training and testing is performed in episodes on the novel class samples. In each episode i of N-way K-shot problem, the few-shot learner is trained on a support set, i.e., Deupport supporr support INK {x, ,Wi,t Ji=l for N novel classes containing K samples per class. Then, the few-shot learner is evaluated on the query set D.uery = {0 Vitt Jt=1
quary query 1 N. on the same N classes of D cupport
4 Proposed Methodology
Inspired by the efficacy of pseudo-labeling, we further increase sample size by mixup and choose more informative samples by hard mixup-basee based sample selection. Our proposed method (FeLMi), outlined in Fig. [ consists of the following six stages: (i) Learning an embedding on the base dataset (Dbase), ), (ii) Pseudo labeling of the base dataset, (iii) Entropy-based filtering of the pseudo-labeled base dataset, (iv) Mixup sample generation using both base and novel examples, (v) Uncertainty- hard-mixup sample selection, and (vi) Finetune on the entire dataset using filtered base examples, novel examples and hard-mixee mixed examples. We next describe each of these steps in detail.
3

---

base
fe
g
y base
Stage
Stage
Stage
B-N
B'
fo
PBN
x
filt
sel
P Pmix
N-N
Xemix
fe
P PNN
hard
N
xooeel
Stage
Stage
Fewshot Learning with Hard Mixup
base L LD
CE
Novel
CE
mixed
Stage
Figure I: Our proposed method, FeLMi consists of six stages. Stage (I): backbone (fe) is pretrained on the base dataset B. Stage (2): pseudo-labels (ghase) of the entire base dataset generated using classifier Ie. Stage (3): base samples are filtered based on entropy to obtain a reduced dataset B Stage (4): novel-novel (I:Wi= and base-novel (1= mixup samples are generated. Stage (5): hard examples ( (zhard) are selected based on margin. Stage (6): The model is trained on the combination of losses.
4.1 Learning embedding on the base dataset
We first learn a discriminative representation from the base dataset Daase The convolutional neural network consists of a backbone fo and a final classification layer hg. The parameters (0, o) are jointly learned using the cross-entropy loss on the base dataset Dbase We use the self-supervised Invariant and Equivariant Represenatation learning (IER) framework by Rizve et al. [22] which makes use of self-supervised losses addition to standard cross-entropy loss to learn good representations.
(abaze (base) =
(1)
Large base datasets ensure that we have good pretrained model to use for meta-testing stage.
4.2 pseudo-labeling of base dataset using classifier trained on novel dataset
In this step, we train a logistic regression classifier using the novel examples. Then, using the trained classifier, we generate the pseudo-labels on the entire base dataset following Jian er al. [II. This pseudo-labeling step partially mitigates the data scarcity problem prevalent in few-shot training, For each episode we learn a linear classifier o using the support set of novel examples D support such that
Φi argmin E (2. Lev (qune (x)),
Then, the classifier 96. is used to generate pseudo-labels for the entire base dataset, i.c., ghase Io. (form (xx)) for t base
4.3 Entropy based pseudo-label filtering
Given that base classes are disjoint from novel classes, not all classes will provide useful information for the task hand. Specifically, base samples with high entropy pseudo-label, might lead to confusing samples. Therefore, we perform a simple pseudo-label filtering based on entropy.
filtt [ybase H (ybaae) < where t 1,..., N base
(3)

---

We empirically select the entropy threshold т. After refinement we have (Dbase, rile base examples. H (.) here denotes the entropy.
4.4 Mixup sample generation
Handling the data scarcity problem using pseudo-labels motivates us generate even more samples using a simple yet an effective mixup strategy. Particularly, we use a feature level manifold mixup step generate more synthetic samples. Since we have both the base and novel examples, we mix up both base-novel and novel-novel pairs. Note that, since the mixed samples should be close to novel examples, are not performing base-base mixup. A straightforward way of generating synthetic examples through mixup is to mix two pure novel examples. We call this strategy 'Novel-Novel' mixup.
4.4.1 Novel-Novel Mixup
We perform manifold mixup on novel-novel samples to create a pool of mixed up samples PN.N. We select {(22ove1, ) (, (xnovel, gnoval)] and mixup with X, ~ Beta(a,, a) to generate (Emins Umiz) where:
E Daupport, pass them through the feature extractor fo
Xn- (z (1 An) forw (znovel)
Tmiz
y Umix
N-N Xn novel + (1 - An)grovel
(4)
The pool of novel-novel mixup samples of size are generated through randomly sampling Ans i... ):)
PN.N = ((rii M
4.4.2 Base-Novel Mixup
Access the base examples during training and their pseudo-labels allows make of these for mixup. The motivation behind using mixup to generate and augment the small novel set. Since the base classes are disjoint from novel classes and we only have weak mapping through the pseudo-labels, we employ the following steps while using base examples with mixup. 1) We refrain from mixing two base examples, instead always mixes base example with novel example. 2) We only mix with base examples that are close to novel examples. We ensure this by choosing base examples which have a low pseudo-label entropy. 3) We choose small mixup lambda corresponding to the base examples when mixing them ensuring they remain proximal to the distribution of novel samples. Specifically, we sample ÅB ~ Uniform(O, 0.2). From (a base 1) base E Dbase_filt we select k-lowest entropy base examples, i.e., (2.aae jbaae) such that,
(xbare (baae)) ((,,, M bottom_k(H()))
(5)
Novel examples are norovl govvll) E Doupport mixed with these selected base examples to generate (x-B-N .BBNN) as follows.
B-N T min
Nb forne ( (1: + (1 Ao) (2 novel)
B B-N N Umic
Ab-
cbase
+ (1 At)g
(6)
The pool of base-novel mixup samples size are generated through randomly sampling λo i.c., )()
PB.N = [( B-N BBN
4.5 Hard mixup sample generation
In the section above, we have generated pool of base-novel and novel-novel mixed up samples denoted by PB.N and PN.N respectively. Now, we concatenate the generated mixed-up samples, i.e., Pmic PB,N U PN.N and then choose hardest N samples based on uncertainty measure.
More specifically, to get an estimate of hardness of a mixed example chosen from Pmins we first obtain a (potentially noisy) estimate of the class probability (= 9a, (z:) using the classifier trained in stage 2. We then estimate margin (difference in top-2 probabilities). measure of uncertainty that is commonly used in active learning (24) and pick mixed samples which have low margin.
5

---

Intuitively, samples with lower margin lie close to the class boundary thus providing a stronger learning signal.
We choose k-smallest margin values as hard examples (denoted as Phard mix) chosen from both the base and novel mixed up samples follows.
Phard mán poottom_k(marginng (Joowa (x)) (x, E Pminn
4.6 Finetune on the entire dataset
In the last stage, the model is finetuned on a combined loss computed using the filtered base examples (Dbase_tslgg the novel examples (Dnovel), and the hard mixup samples (Phard_ma miz). For the base examples, KI. divergence loss is used, whereas cross-entropy loss has been used for novel and mixup samples. The final loss is denoted as,
E .... Lwwlo
(8)
+ (x.)) (x.)) D Lcc( y) (otea))))).).........../e....... Poansse
and 7 are scaling parameters for losses corresponding to novel and mixed up samples, respectively,
Finally, the trained model is evaluated on the query set of the episode i (D.uary) and averaged accuracy over all the episodes are reported.
Experimental Results
5.1 Experimental Setup
Datasets: We experiment on four widely used few-shot benchmarks: FC-100 17. CIFAR-FS [4]. minilmageNet (3 and magred [20). FC-100 is subset of CIFAR-100, containing 60 classes meta-training. 20 classes for meta-validation and 20 classes for meta-testing. CIFAR-FS also obtained from CIFAR- 100, containing 64 classes for meta-traine meta-training, 16 classes for meta-validatiin and 20 classes for meta-testing. minilmageNet is derived from ImageNet with images downsampled to resolution of 84x 84 pixels. It has 64 classes for meta-training, 16 classes for meta vola-validotion and 20 classes for meta-testing. tieredImageNet (20) is another subset of ImageNet, containing total of 608 classes, from which 351 classes used for meta-training, 97 classes for meta-validation and 160 classes for meta-testing.
Implementation details: For fair comparison with prior works, we use the ResNet-12 12 architecture as the backbone network. attach two-layer MLP on top of the feature extractor which outputs N-way logits. We use the SGD optimizer with momentum 0.9. Learning rate of backbone and classifier are set to 0.025 and 0.05 respectively with weight decay of Se-4. Following [Л we use the temperature coefficient of 4.0 for our KD loss. We use minibatch size of 250 for training. Using standard setting, we also perform data augmentation using color jittering, random crop and horizontal flip. Further details of the hyperparameters and training are provided in the supplementary material.
5.2 Results on benchmark datasets
We have compared our proposed method (FeLMi) with state-of-the-art techniques the benchmark datasets: FC-100, CIFAR-FS, minilmageNet and ticredImageNet. We obtain consistent improve- ments over state-of-the-arr showing the efficacy of our approach on all the aforementioned few-shot benchmarks for both I-shot and 5-shot settings in Table Table 2 and Table
5.3 Results cross-domain benchmark
We test our approach for more challenging cross cross-domain settings, where the base-pretraining is done on minilmageNet and the novel classes stem from samples from another domain, e.g., CUB (34]. Our approach achieves significant performance improvements for both the 5-shot and 1-shot setting in this setting as shown Table 4 Since CUB performs fine-grained classification, we hypothesize that hard mixup samples are even more useful in this case.

---

Table 1: Comparison FeLMi (ours) to prior works on CIFAR-FS. Following prior work, we report our results with 95% confidence intervals on meta-testing split of the dataset. 9 denotes our reproduced numbers using publicly available implementations.
CIFAR-FS R-RFFFFER 5-way
Model
Backbone
1-shot
5-shot
ProtoNet
28
(NIPS' 17)
ResNet-12
72.2 0.7
83.5 + 0.5
MetaOptNet
(CVPR'1) 19)
ResNet-12
72.6 + 0.7
84.3 + 0.5
Shot-Free 19 (ICCV'19)
ResNet-12
69.2 n/a
84.7 + n/a
DSN-MR 27 (CVPR'20)
ResNet-12
7.9
86,2
RFS-simple [31] (ECCV'20)
ResNet-12
7.8
86.0 0.5
RFS-distill [3]] (ECCV'20)
ResNet-12
7.89
86,9 0.5
SKD-GEN1 [8 (Arxiv'20)
ResNet-12
76.6
88.6
IER-distill 22 (2)
ResNet-12
1.0
PAL 15
(ICCV'21)
ResNet-12
77.1.00.7
8.5
Label-Halluc
1
(AAAT'22)
ResNet-12
78.0 1.0
$9.37
FeLMi
ResNet-12
78.22 1 0.7
89.47 + 0.5
Table 2: Comparison of FeLMi (ours) to prior works on FC-100. Following prior work, we report our results with 95% confidence intervals on meta-testing split of the dataset. denotes our reproduced numbers using publicly available implementations.
FC-100 5-way
Model
Backbone
1-shot
5-shot
ProtoNet 22 (NIPS' 17)
ResNet- 12
37.5 + 0.6
52.5 + 0.6
TADAM 77 (NIPS'18)
ResNet-12
40.1 + 0.4
56.1 + 0.4
MetaOptNet [33 (CVPR'19) 19)
ResNet-12
41.1 + 0.6
55.5 0.6
MTI. 20 (CVPR' 19)
ResNet-12
45.1
0.9
DeepEMD 38 (CVPR'20)
ResNet-12
46.5 0.8
63.2:0.7
RFS-simple [31] (ECCV'20)
ResNet-12
42.6
59.1 0.6
RFS-distill [3] (ECCV'20)
ResNet-12
44.6
6..9
AssoAlign 1 (ECCV'20)
ResNet-18
0.5
5.6
SKD-GENI 18 (Arxiv 200)
RcsNet-12
4.8
6..2
InfoPatch 10 (AAAI'21)
ResNet-12
43.8
58.0
IER-distill 22 (CVPR'21)
ResNet-12
48.1
6.70
PAL 115
(ICCV'21)
ResNet-12
47.2
0.6
1 (AAAI'22)
ResNet-12
47.37 + 0.7
67.92 = 0.7
FeLMi
ResNet-12
49.02 + 0.7
68.68 1 0.7
FC-100
CIFAR-FS
minilmageNet
5-way 5-shot
5-way 5-shot
5-way 5-shot
0.69
0.862
Accuratty
0.085
Accuraty
0.895
Accuraty
0.86
0.858
0.856
0.89
0.854
0.675
0.852
Num, mixup samples (N)
Num. mixup samples (N)
Num. mixup samples (N)
(a) 100
(b) CIFAR-FS
(c) mini-mmmaa mini-ImagoNNet
Figure 2: Effect of changing number of mixup samples (N). In this experiment investigate the effect of N on the final cumulative accuracy for the 5-way 1-shot on the three datasets. We notice that N=1000 shows consistent improvements across all datasets.

---

Table 3: Comparison of our method (FeLMi) against the state-of-the-an on minilmageNet and tiered- ImageNet. Following prior work, we report our results with 95% confidence intervals on meta-testing split of the dataset. denotes our reproduced numbers using publicly available implementations.
minilmageNef 5-way
tiered 5-way
model
backbone
1-shot
5-shot
1-shot
5-shot
ProtoNet
28 (NIPS'17) 17)
ResNet-12
60.37 = 0.83
78.02 0.57
65.65 + 0.92
83,40 = 0.65
TADAM
(NIPS' 18)
ResNet-12
58.50 0.30
76.70
0.30
TapNet
16
(ICML'19)
ResNet-12
61.65 0.15
76.36
63.08 0.15
80.26 0.12
MetaOptNet
13
(CVPR'19)
ResNet-12
62.64 0.61
78.63
65.99 0.72
81.56 = 0.53
MTL 29
(CVPR' 19)
ResNet-12
61.20 1.80
75.50
65.62 + 1.80
80.61 = 0,90
Shot-Firee
19
(ICCV'19)
ResNet-12
59.04
77.64
66.87 +: 0.43
82.64 = 0,43
DSN-MR
27
(CVPR'20) 20)
ResNet-12
64.60 0.72
7
67.39 + 0.83
82.85 = 0.56
DcepEMD
38 (C)
ResNet-12
65.91 = 0.82
82.41
71.16
86.03 = 0.58
FEAT [35
(CVPR"20)
ResNet-12
66.78 0.20
82.05
70.80 70.80.1..20 0.23
84,79 1 0.16
Neg-Cosine
14] (ECCV'22)
ResNet-12
63.85 0.81
81.57
RI'S-simple
31
(ECCV'20)
ResNet-12
62.02 0.63
7.444
69.74 0.72
84,41 = 0.55
RFS-distill
(ECCV'20)
ResNet-12
64.82 1
1.41
71.52 1: 0.69
86.03 = 0.49
AssoAlign
(ECCV*20)
RcsNct-18
59.88 0.67
80.73
6..29
85.97 = 0.49
AssoAlign [ (BCCV'220
WRN-28-10
65.92 1:92:00 0.60
0.55
SKD-GENI [8] (Arxiv'20)
R
66.54 0.97
8..18
72.35 + 1.23
85.97 = 0.63
P P-Transfer 25 (AAAI'21)
R2
0.77
1
MELR
g (ICLR'21)
ResNet-12
67.40 +: 0.43
83.40
72.14 0.51
87.01 = 0.35
|EPT 40 (ICLR'21)
ResNet-12
67.05 + 0.44
8.30
72.24 0,50
86.73 = 0.34
IER-distill
[2 (CVPR'21)
ResNet-11
66.85 1: 0.76
84.50
72.71 + 0.89
86.57 = 0.81
Label-Halluc (ITAAAAT22)
ResNet-12
67.04 = 0.75
85.87 1 0.48
71.97 + 0.89
86.80 0.58
FeLMi
ResNet-12
67.47 1 0.78
86.08 1 0.44
71.63 1 0.89
87.07 0.55
Table 4: Comparison of FeLMi (ours) in cross-domain setting (minilmageNett + CUB). We obtain significant boost compared to prior approaches showing the efficacy of our approach.
minilmageNet - CUB 5-way
model
backbone
1-shot
5-shot
Baseline++ 6 (ICLR' 19)
ResNet-18
40.44 + 0.75
56.64 0.72
MetaOptNet 13 (CVPR' 19)
ResNet-18
44.79 + 0.75
64.98 + 0.68
S2M2R [16 (WACV'20)
ResNet-18
48.24 0.84
70.44 = 0.75
AssoAlign
[ (ECCV'20)
ResNet-18
47.25
72.37 = 0.89
MixFSL 2 (ICCV'21)
ResNet-18
68.77 + 0.9
MT-ConFT 7 (ICCV'21)
ResNet-10
49.25 + 0.83
74.45 + 0.71
FelMi
ResNet-12
51.66 + 0.82
77.61 + 0.69
FC-100
CIHAR-FS
minilmageNet
5-way 5-shot
5-way 5-shot
5-way 5-shot
0.59
0.8615
Accuracy
0.38
Accuracy
Accuracy
0.884
0.861
0.57
0.652
0.8605
0.55
0.86
0.55
(a) FC-100
(b) CIFAR-FS
(c) mini-mmaageee
Figure 3: Effect of changing number of a parameter of the Beta distribution. In this experiment we investigate the effect of a on the final cumulative accuracy for the 5-way 1-shot on the three datasets. Note that effectively controls the À values that we sample for N-N mixup. A value of 1 implies sampling from a uniform distribution whereas 0.5 samples À closer to 0 1. We notice that a = shows consistent improvements across all datasets.
8

---

Table 5: Contribution our proposed tech- niques to the final performance on FC-100 for 5-way 5-shot classification. We see that each of our proposed techniques leads to an improvement.
Table 6: Evaluating different mixup strate- gies for 5-way 5-shot classification. Our pro- posed approach of using N-N mixup improves performance. Additionally using B-N mixup leads to further gains.
Approach
Accuracy
Mixup Approach
IER 22
65.00
A
An
FC-100
CIFAR-FS
pseudo-labe
M
67.92
B-N+ N-N
U(0.0.2)
B(1.1)
68.68
89.47
entropy filtering
67.96
B-N+N-M
B[L,
B(1
68.49
89.26
Mixup
68.49
N-N
B(L
68.57
89.4
hard selection
68.68
None
67.92
89.37
Modt
Iabel
3-
0,0
1.0
-5-
4.0
-10
-55 -10
-10
(a) First training step
(b) Last training step
Figure 4: tSNE visualizations. We visualize the tSNE plots of the learned representations at the start of training and at the end for one random episode. We see that as training progresses, the data gets more clustered and query labels (denoted by y_Q) get close to the support set clusters. We also overlay the generated mixup samples. These samples offer good training signal to learn better class boundaries.
5.4 Ablation Studies
In this section, we investigate the effect of each stage of our method on the overall performance.
Ablation of each individual contribution: We start with the base pretraining similar to IER 22 followed by pscudo-labeling of the entire base dataset akin to Jian et al. [I]. Entropy-based filtering of the pseudo-labels provides an improvement of 0.04%. Using both base and novel sample mixup, we get an improvement of 0.53%. Margin-base hard mixup sample selection additionally provide a performance boost of 0.19% on FC- 100 dataset 5-way 5-shot setting as shown Table.
Effect of mixup strategies: Next, we evaluate different mixup strategies, results are provided in Table. on both FC-100 and CIFAR-FS in 5-way 5-shot settings. In both the cases, observe that best results are obtained when incorporating base-novel mixup with A ~ Uniform(0,0.2) in addition to novel-novel mixup with A, Beta(1,1). This aforementioned setting ensures the mixed up base-novel samples (denoted by yellow points in Fig. 4 (a)) lie close to the novel examples (labeled 0-4 in Fig. (a)).
Effect of mixup sample size: We observe the variation of 5-way 5-shot classification accuracy vs. number of mixup sample size for FC-100, CIFAR-FS and minilmageNet in Fig 2a Fig. 25 and Fig. 2c respectively. We notice that all the settings, mixup sample size of 1000 performs the best. Increasing the number of mixed up samples seems to lower the performance bit. We hypothesize this might be due to the increasing imbalance between 'real' and 'generated' points, thus hurting the learning.
Effeet of mixup parameters: The variation of 5-way 5-shot classification accuracy with respect to mixup parameter for FC-100, CIFAR-FS and minilmageNet used for performing novel-novel

---

sample mixup is shown in Fig. 3 In CIFAR-FS and minilmageNet, a 0.5 seems to perform better. However, FC-100, we observe the opposite trend.
Next, we analyse the tSNE visualizations the representations learned using our approach. The visualizations (Figure 4 show these for representations at the first step of training (Figure 4)) and after the training is completed for one episode (Figure (b)). We observe that the representations for each class get more clustered as the training progresses. Further, representations of the query samples get closer to the corresponding class support samples. Our generated hard mixup samples are close to the query samples as seen in the figure, which provide the necessary training signal to improve the model.
tSNE visualizations:
6 Limitations
Like most prior approaches, we make use of the base examples when adapting the classifier for novel examples. This requires access to base examples during meta-testing which can be difficult for very large scale datasets. We would like to fix this requirement in future work.
7 Conclusions
Few-shot learning is an important computer vision problem. The inherent data scarcity issue makes this problem more challenging. To address this issue, we propose to use hard mixup to synthetically generate mixup samples based on lowest margin based uncertainty measure. To the best of our knowledge, we are the first to propose hard mixup for few-shot learning. Moreover, we also perform hard mixup the novel-novel and base- novel samples and further improve performance benchmark datasets, e.g., FC-100, CIFAR-FS, minilmageNet and ticredImageNet in both the 1-shot and 5-shot settings. Our approach also significantly outperforms the state-of-the --tth-rrt in cross-domain few-shot (minilmageNNet >> CUB) settings.
the
8 Acknowledgements
The authors AR, AS, and RC were supported by an ONR MURI grant N00014-20-1-2787.
References
[1] Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné. Associative alignment for few-shot image classification. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[2] Arman Afrasiyabi, Françorançois Lalonde, and Christian Gagné. Mixture-based feature space learning for few-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9041-9051, 2021.
[3] Arman Afrasiyabi, Hugo Larochelle, Jean- François Lalonde, and Christian Gagné. Matching feature sets for few-shot image classification. arXiv preprint arXiv:2204.0000 00949, 2022.
[4] Luca Bertinetto, Joao Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2019.
[5] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Representations, 2021.
[6] Wei- Yu Chen, Yen- Yen-Cheng Liu, Zsolt Kira, Yu- Cl-Chiing Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv: 0000044422, 2019.
[7] Rajshekhar Das, Yu-Xiong Wang, and José MF Moura. On the importance of distractors for few-shot classification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9030-9040, 2021.
10

---

[8] Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songtang Huang. (MELR): Meta-leaming via modeling episode-level relationships for few-shot learning. In International Conference on Learning Representations, 2021.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1 1126-1135, 2017.
[10] Yizhao Gao, Nanyi Fei, Guangzhen Liu, Zhiwu Lu, Tao Xiang, and Songfang Huang. Con- trastive prototype learning with augmented embeddings for few-shot learning. arXiv preprint arXiv:2101.09453 2021.
[11] Yiren Jian and Lorenzo Torresani. Label hallucination for few-shot classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[12] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33:21798-221
2020.
[13] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[14] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- aanMMi Frahm, editors, Computer Vision - ECCV 2020, 2020.
[15] Jiawei Ma, Hanchen Xie, Guangxing Han, Shih-Fu Chang, Aram Galstyan, and Wael Abd- Almageed. Pastner-assisted learning for few-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10573-10582, 2021.
[16] Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vineeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2218-2227, 2020.
[17] Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
[18] Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. self-supervised knowledge distillation for few-shot learning. arXiv preprint arXiv:2006.0973. 09785, 2020.
[19] Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2019.
[20] Mengye Ren, Sachin Ravi, Eleni Triantafillou, Jake Snell, Kevin Swersky, Josh B. Tenen- baum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for supervisd few-shot classification. In International Conference on Learning Representations, 2018
[21] Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-awarc pseudo-label selection framework for semi-supervised learning. arXiv preprini arXiv:2 2021.
[22] Mamshad Nayeem Rizve, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Exploring complementary strengths of invariant and equivariant representations for few-shot learning. arXiv preprint arXiv:2103.01315, 2021.
[23] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals. Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization, In International Conference on Learning Representations, 2019.
[24] Burr Settles. Active learning literature survey. 2009.

---

[25] Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang- Cheng, Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. CoRR, abs/2102.03983, 2021.
[26] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761-769, 2016.
[27] Christian Simon, Piotr Koniusz, Richard Nock. and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
(28) Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
[29] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few- shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[30] Flood Sung, Yongxin Yang, Li Zhang. Tao Xiang, Philipp H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[31] Yonglong Tian, Yue Wang, Dilip Krishman, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: A good embedding is all you need? In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, editors, Computer Vision - ECCV 2020, 2020.
[32] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
[33] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances Neural Information Processing Systems, 2016.
[34] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 2irds-200-2011 dataset. 2011.
[35] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[36] Sung Whan Yoon, Jun Seo, and Jaekyun Moon. TapNet: Neural network augmented with task- adaptive projection for few-shot learning. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 7115-7123. PMLR, 09-15 Jun 2019.
[37] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6033 2019.
[38] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image clas- sification with differentiable earth mover's distance and structured classifiers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[39] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1710:4171. 2017.
[40] Manli Zhang, Jianhong Zhang, Zhiwu Lu, Tao Xiang, Mingyu Ding, and Songfang Huang. (IEPT): Instance-level and episode-level pretext tasks for few-shot learning. In International Conference Learning Representations, 2021.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Code will be released later. To ease reproducibility, we include pseudocode in Appendix.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See 5 and Supplementary material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We report error bars with respect to episodes.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Supplementary material
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [No] We will mention along with code
release.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "_5-EPmcSl2r", "reviewer": "Reviewer_3CtS", "summary": "The paper presents a hard manifold mix up process to augment the few-shot samples during fine-tuning for improving accuracy performance. The mix-up is carried out in different settings: novel-novel, novel-base and the hard samples from the mix up based on margin are utilized during fine-tuning.", "questions": "1.  The paper says that the pretraining is done with  self supervised loss in addition to cross entropy, but the eq. 1 just shows cross entropy where is the full loss according to the description?\n2. How exactly the entropy threshold for filtering pseudolabels for base classes is selected?\n3. How this method is compared with the \"Charting the Right Manifold: Manifold Mixup for Few-shot Learning\" paper by Mangla et al. in WACV 2020?\n4. Why the other standard benchmarks such as tiered-Imagenet or CUB is missing also different architechtures such as WRN? Also, are the methods compared are evaluated based on the same pretrained model? Over, how many episodes the average accuracies are reported?\n5. What about domain-shift results for example mini to CUB setting as in previous papers such as \"A closer look at few-shot learning\" by Chen et al. 2020.\n", "limitations": "I do not see much improvement from this method in comparison to previous methods. In fact in many settings the improvement is negligible such as in Table 1. Moreover, sometimes it is not practical to have access to the base class samples during fine-tuning stages and the assumption. Therefore source free fine-tuning is not practical with this method.\n\nThe pseudolabel creation for base class samples with a classifier initially built on novel target samples is cumbersome as the label set do not match in most cases. Therefore, the filtering step on top of this assumption with entropy criterion does not make sense with a reasonably big domain between base classes and novel classes. Also, many standard benchmark datasets are missing in the evaluation such as tieredImagenet, CUB, domain shift experiments such as mini-imagenet to CUB. ", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The papers' idea of using manifold mix up is complemented by the use of hard sampling for further use during fine-tuning, which is simple and interesting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "X_r2wgmRnVV", "reviewer": "Reviewer_2Qz6", "summary": "This paper presents FeLMi, a type of mixup method for few-shot learning, with margin-based uncertainty criteria. The work aims to augment new data in a mixup form to tackle the overfitting problem in a few-shot classification setting. With this, the presented method consists of six phases. Overall, after pretraining on the base classes and obtaining the pseudo labels of the base examples, both novel-novel and base-novel mixup samples are generated for data augmentation. The pretraining method employs self-supervised-based Invariant and Equivariant Representation learning (IER) [19]. Finally, the evaluation is presented using three few-shot learning benchmarks.   ", "questions": "- While understanding the use of standard transfer learning instead of an episodic training approach, I would like to see more discussion on why fine-tuning and why not episodic pre-training? \n- What about standard pretraining without using a self-supervised [19] training method?\n- What about the time complexity of the method?\n- Could you please elaborate more on “k-smallest margin values as hard examples” (line 194)?\n- What about the fine-tuning implementation details? Also, do the literature uses MLP on top of ResNet12? (“we train a two layer MLP” ll. 214). If so, please cite. \n- Why “temperature coefficient” is set to 4.0 (line 217)? Is this obtained through cross-validation?", "limitations": "Though the presented method is proposed for few-shot learning, it is only evaluated with image classification. Additionally, evaluation with large datasets (such as Tieredimagenet) is missing. While the presentation of the paper is good, I think the author(s) can improve the related work by clearly discussing the difference between the approaches (such as Base examples for few-shot learning) with the proposed method. I think the conclusion can include some limitations and future works too.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, the idea of the paper is interesting. Specifically, I like how the presented method is inspired by several recent works of base examples and mixtup method for few-shot image classification to present novel data-augmentation methods. Additionally, the paper is well-written and easy to follow. However, I think the method contains the following weaknesses:\n- The overall model is complex (having several stages with different criteria).  \n- The method uses many tricks, such as self-supervised learning for pretraining and active learning for the hard mixtup. Therefore, the method's evaluation and justification became more complicated compared to standard methods such as ProtoNet. \n- The proposed method is evaluated on three small datasets, but I think some large datasets are required.   \n- Having a 5-way classification problem, the FeLMi can not gain significant accuracy in CIFAR-FS dataset. I think this is ok, but some extra evaluation might help us understand the proposed method's classification gain/loss. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NG6hSuBVbj", "reviewer": "Reviewer_WHpk", "summary": "In this paper, Few shot Learning with hard Mixup (FeLMi) is developed to mitigate the issue of data scarcity. The proposed method is composed of 6 steps. (1) Pretrain the model on the base dataset using the cross-entropy loss with auxiliary self-supervised loss. (2) Train a linear logistic regression model on the top of the learned feature extractor using the novel data to generate pseudolabels for the base dataset. (3) Filter the pseudolabels based on entropy. (4) Do novel-novel and base-novel mixup to generate more data. (5) Choose hard mixup samples based on the margins in classification probability. (6) Finetune the model using the filtered base data, novel data, and hard mixup data. Experiments indicate that the proposed method leads to improved performance in few-shot learning.", "questions": "According to Table 5, base-novel mixup leads to negligible performance improvement. I wonder if this step is necessary for your proposed method.", "limitations": "Compared with the simplest fine-tuning methods in few-shot learning (e.g. RFS-simple, ECCV 2020), the proposed method is more computationally expensive. Specifically, the proposed method must assign pseudo labels for the base data. It becomes an issue when the base dataset is huge. It’s better to provide an empirical or theoretical analysis of the time and space complexity.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well organized and easy to follow.\n\nThe proposed method is straightforward and easy to understand. \n\nSufficient ablation studies in the experiments.\n\n\nWeaknesses:\n\nAmong the 6 steps in FeLMi, only Step 5 is original (choosing hard mixup samples). However, Table 4 indicates that Step 5 only leads to marginal improvement in few-shot classification accuracy.\n\nThe model is pretrained by cross-entropy loss and self-supervised loss. It is known that pretraining with auxiliary self-supervised loss leads to improved performance in transfer learning for downstream tasks. Although FeLMi achieves the best performance in Tables 1, 2, and 3, it relies on a stronger pretraining method. It does not demonstrate the effectiveness of hard mixup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5EiiKQZ3QNY", "reviewer": "Reviewer_7gPu", "summary": "This work presents a study on applying 1) combining pseudo-labeling with entropy-based label filtering for representation learning and 2) novel-novel and base-novel manifold mixup with entropy-based filtering for adapting base representation to novel classes for improving few-shot image recognition tasks. Under appropriate hyperparameter settings, the proposed approach achieves competitive performance on standard few-shot image recognition benchmarks. Ablation studies are conducted to investigate the gains brought by individual techniques.", "questions": "See weaknesses.", "limitations": "Limitations are addressed adequately.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The paper is well-written and easy to follow. Approaches, experiment settings and implementation details are clearly described, in a way that helps reproducibility of the proposed work. Experiment results are well organized.\n2) Experiments and ablation studies seem thorough. Standard benchmarks are used. Latest works that follow the same experimental settings are included as baselines. Necessary baselines are included. Ablation studies included all components of the system.\n3) Experiment results seem to show highly competitive performance (without transductive learning).\n\nWeaknesses:\n\n1) While it is clear that the proposed approach worked, it is not very clear how it worked. Here are my recommendations\n- For entropy-based filtering, try to fit the qualitative examples in the main paper\n- Show some hard examples selected by mixup. Maybe something like (0.9x image A + 0.1x image B)\n- Which classes often benefit from pseudo-labeing, mixup and entropy filtering?\n\n2) While not necessary, it would be interesting to learn about the sensitivity of the proposed approach to hyperparameters. The use of different hyperparameters for 1- and 5-shot might be a deviation from classical setups? Although it's not entirely surprising that different hyperparameters might be necessary as the dataset size changes. It would help if measures are taken to control overfitting to specific benchmarks.\n\n\n=============================\n\nAuthor response adequately addresses my concerns.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept