POODLE: Improving Few-shot Learning via Penalizing Out-of-Distribution Samples
Authors: Duong Hoang Le
Khoi Duc Nguyen
Khoi Nguyen
Quoc-Huy Tran
Rang Nguyen
Binh-Son Hua
Venue: NeurIPS 2021 Poster (2021)
Keywords: few-shot learning

Abstract: In this work, we propose to use out-of-distribution samples, i.e., unlabeled samples coming from outside the target classes, to improve few-shot learning. Specifically, we exploit the easily available out-of-distribution samples to drive the classifier to avoid irrelevant features by maximizing the distance from prototypes to out-of-distribution samples while minimizing that of in-distribution samples (i.e., support, query data). Our approach is simple to implement, agnostic to feature extractors, lightweight without any additional cost for pre-training, and applicable to both inductive and transductive settings. Extensive experiments on various standard benchmarks demonstrate that the proposed method consistently improves the performance of pretrained networks with different architectures.

POODLE: Improving Few-shot Learning via
Penalizing Out-of-Distribution Samples
Duong H. VinAI Research v. wwuunngnh@@vinai..il 11
Khoi D. Nguyen VinAI Research
Khoi Nguyen VinAI Research ducminhkhoi1gmmm wathhho@@mmmii... com
choinggyenucd@gmal..comm
Quoc-Huy Tran Retrocausal, Inc.
Rang Nguyen
Binh-Son Hua VinAI Research & VinUniversity
VinAI Research
huyGretrocausal.cc
rangnhn@gmail.com com
Abstract
In this work, we propose to leverage out-of-distribution samples, i.e., unla- beled samples coming from outside target classes, for improving few-shot learn- ing. Specifically, we exploit the casily available out-of-distribution samples (e.g., from base classes) drive the classifier avoid irrelevant features by maximizing the distance from prototypes to out-of-distribution samples while minimizing that in-distribution samples (i.e., support, query data). Our ap- proach is simple to implement, agnostic to feature extractors, lightweight with- out any additional cost for pre-training, and applicable to both inductive and transductive settings. Extensive experiments on various standard benchmarks demonstrate that proposed method consistently improves the performance of pretrained networks with different architectures. Our code is available at /etpps:/gitthn www./VanATReeearcc/ppontoo
Introduction
Learning with limited supervision a key challenge to translate the research efforts of deep neural networks to real-world applications where large-scale annotated datasets are prohibitively costly to acquire. This issue has motivated the recent topic of few-shot learning (FSL), which aims to build a system that can quickly lear new tasks from small number of labeled data.
popular group of methods in FSL focus on strengthening the backbone network by various techniques, from increasing model capacity [6, 11], self-supervised learning (SSL. [18, 51, 65], to knowledge distillation (KI) [53]. With these techniques, few-shot methods are expected learn better representations that are more robust and generalized. However, even if the network can discover visual features and semantic cues, few-shot learners have to deal with key challenge the ambiguity: as we have only small amount of support evidence, there multiple plausible hypotheses at inference stage. Existing works, therefore, rely on developed inductive bias of the network (during the pretraining stage), such as shape bias [47, 14], to reduce the hypothesis space.
In this work, we view the classification problem conditional reasoning, i.e., "if X has then is Q". Human beings are good at learning such inferences, thus quickly grasping new concepts with minimal supervision. More importantly, humans learn new concepts context where have already had prior knowledge about other entities. According to mental models in cognitive science, when assessing the validity an inference, one would retrieve counter-examples, iee. which do not
First two authors contribute equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Class #1
Class #2
Query
Distractor
adult
child
child
dog
suit
casual
suit
suit
Figure I: Illustration of advantages of counter-example data. The query image has features from both support classes (i.e., child and suit), which makes classification ambiguous. The prediction result would depend on the inductive bias prior knowledge of the network. By using an out-of-distribution sample, it becomes clear that the query result should favor Class 2.
lead to the conclusion despite satisfying the premise [9, 13, 29, 30, 55]. Thus, if there exists least one of such counter-examples, the inference is known to be erroneous.
Hence, we attempt equip few-shot learning with the above ability so that eliminate incorrect hypotheses when learning novel tasks in a data-driven manner. Specifically, we leverage out-of- distribution data, i.e., samples belonging classes separated from novel tasks as counter-examples for preventing the learned prototypes from overfitting to their noisy features. To that end, when learning novel tasks, we adopt the large margin principle metric learning [59] to encourage the leamned prototypes to be close to support data while being distant from out-of iittof-diittriutionn samples.
Our approach is complementary to existing works in FSL and could be combined to advance the state of the art. Moreover, our method is agnostic to the backbone network; thus, it does not have the need of training phase adopt in SSL and KD, while incurring just little overhead at inference (for fine-tuning prototypes with approximately 200 gradient updating steps).
In summary, contributions are follows:
We propose novel yet simple approach learn the inductive bias deep neural networks for FSL by leveraging out-of-distributio data. We empirically show that out-of-distributin data only require weak labels (i.e., in the form of whether a sample is in- or -itto---iitribution) even in challenging problems such as cross- FSL.
We introduce a new loss function to implement the above idea, which is applicable for both inductive and transductive inference. Our extensive experiments diffferent standard benchmarks show that the proposed approach consistently improves the performance of various network architectures.
We validate the effectiveness of our method in various FSL settings, including cross-domain FSL.
2 Related work
Over the past few years, considerable amount of research efforts [15, 16, 28, 34, 44, 56, 50, 64) have been invested in FSL. They can roughly be classified into two main categories: optimization-bused and metric-based approaches. Optimization simiizzttioo-bssedd methods [15, 16, 28, 34, 44] seek a meta-learner that quickly adjust the parameters of another learner new task, given only few support images. In particular, [35, 15] propose learning to initialize a classifier whose parameters can be obtained with small number of gradient updates on the novel classes. Metric-baser approaches [56, 50, 64] learn task task-agnostic embedding space for measuring the similarity between images. For example, Matching Networks [56] utilizes weighted nearest neighbor classifier, while Prototypical Networks [50] uses the mean features of support images as the prototype for each class. Recently, DeepEMD [64] adopts the Earth Mover's distance to compute the distance between image patches as their distance.
We use OOD samples and distractor samples interchangeably

---

To further boost the performance, recent works have incorporated additional techniques such as self-supervised learning and knowledge distillation during training, and transductive inference during testing, which we will review in the following:
Self-supervised learning. The goal of self-supervised learning is to learn representations from unlabeled data. For FSL, this could be achieved by combining the supervised main task with a self-supervised pretext task which includes predicting image rotations [19], predicting relative patch locations [12], or solving jigsaw puzzles [40]. A few works incorporating self-supervised learning for FSL have been developed recently, e.g., Gidaris et al. [18] suggest pre- vre-training the embedding network by using a combination of supervised and self-supervised loss, i.e., predicting image rotations.
Knowledge distillation. Knowledge distillation is machine learning technique that seeks to compress the knowledge contained in a larger model (teacher) into a smaller one (student). It was introduced by Bucilua et al. [4] and later adopted deep learning by Hinton al. [23]. Recently, Tian et al. [53] show that incorporating self-distillation into FSL can boost the performance by 1 2%.
Transductive inference. Transductive inference aims to leverage the query set (which can be seen as unlabeled data during inference) in addition to the labeled support set. Transductive approaches [39, 24, 43, 66, 2, 10] significantly outperform their inductive counterparts, which do not exploit the query set. For example, TPN [39] constructs a graph whose nodes are support and query images and propagates labels from the support query images, while CAN [24] utilizes confidently classified query images as part of the support set. Recently. TIM [2] assumes a uniform distribution of novel classes, which is often the case for the current FSL benchmarks, and exploits that assumption for boosting the performance. Furthermore, [15, 52] can be considered as transductive methods since the information from query data is used for batch normalization.
In this paper, we improve the performance of the classifier on novel classes by introducing a novel loss function, which penalizes out-of-distribution samples. Our method is complementary to the above approaches and can be combined to establish a new state-of-the-art for FSL. As the time of camera-ready: we aware of concurrent work [8] that also leverages the distractor samples to refine the classifier in FSL. Contrast to that work, our loss function is inherently applicable to both also discover the effectiveness of uniform
inductive and transductive inference. Furthermore, random features, which obviates the need for accessible OOD samples for in-domain FSL..
3 Preliminary
We first define some notations used in the following sections. Let (x. y) consist of sample image x and its corresponding class label y. Let D₁ = {(x,, V: No denote the labeled base samples used for feature pre-training. Next, we denote D. = {(x,, y1) } and D, {(x:, m
Nr as the labeled support samples and query samples respectively. Note that the labels for the query samples are only used for evaluation purposes. The support samples and query samples belong to the novel classes Cn- which are separated from the base classes Co. i.e., C C = Ø. In few-shot learning, we aim to learn classifier that exploits support data to predict labels for query samples. We use pre-trained feature extractor, usually kept fixed, to produce input to the classifier.
In this work, we consider fine-tuning the classifier on the support set only (inductive learning) and on both the support and query set (transductive learning). We also consider both in-domain and cross-domate FSL. In the former, the novel classes Cn and the base classes C₁ are from the same domain (e.g. images from Image-Net), while for the latter, C, and C. are from different domains and the domains of C₁ and Cn are referred to as the source and target domains, respectively.
Pretrained feature extractor. Let fo be the feature extractor trained the base data with the standard cross- cross-entropy loss, which we also refer as the "simple baseline". We also seek to strengthen the baseline with orthogonal techniques such as self-supervised learning (SSL) [18,5] and knowledge distillation (KD) [53]. With SSL, we employ the "rot baseline", which is trained with the standard cross-entropy loss and an auxiliary loss to predict the rotation angles of the perturbed images. We further apply the born-again strategy [17] for the rot baseline in two generations to construct the "rot + KD baseline". More details of our baselines can be found in Section A.
Novel tasks inference. In the few-shot scenario, we freeze the feature extractor fo and train a classifier for each task. Let W [Wo. "Wo.""..wa E KXD be the weight matrix of the classifier,
3

---

where D is the dimension of the encoded vector (output by the feature extractor) and K' is the number of classes (i.e., number of ways) for each task. The predictive distribution over classes p(k] W) is given by:
exp(-7 d(z,, Wk))
fo(x.)
pk]],,, W)
Σj exp( d(z;; w;)) where
2,
(I)
[.fo(x)
where y is the learnable scaling factor [42] and -(..-) denotes the distance function. We use the squared Euclidean distance in our experiments unless otherwise mentioned. In our implementation, we initialize each weight vector Wa to the mean of the sample features in the support set W& =
[8k] Exes, [x) with Sy being the support set of the tth class similar to Prototypical Network [38].
Inductive bias. Inductive learning a process of learning a general principle by observing specific examples. Given limited support data, it is possible to have multiple explanations on the query data, with each corresponding to a different prediction. Inductive bias allows the learner to systematically favor one explanation over another, rather than favoring a model that overfits to the limited support data, Figure/ shows an ambiguous classification example that can be explained by multiple hypothe- ses. Two decision rules possibly learnt from the support data are: 1) People in suit belong to class I; and 2) Boys belong to class 2.
Both rules are simple (satisfying Occam's razor), and can be used to classify the query sample. However, it is unclear which rule the learner would favor; it is only possible to know after training. Hence, solely learning with support data can be obscure. One way to narrow down the hypothesis space is using counter-examples to assess the inductive validity [29, 9]. In Figure the out-of- distribution (OOD) sample hints that the suit should be considered irrelevant, and hence the rule 1 should be rejected.
4 Proposed approach
We introduce our novel technique for few-shot learning namely Penalizing Out-O--Distribttion sampLEs (POODLE). Specifically, attempt to regularize few-shot learning and improve the general- ization of the learned prototypes by leveraging prior knowledge of in- and out-of-distribution samples. Our definition is as follows. Positive samples are in-distribution samples provided in the context of the current task that includes both support and query samples. Negative samples, in contrast, do not belong the context of the current task, and hence are outtof-distribution. Negative samples can either provide additional cues that reduce ambiguity, or act as distractors to prevent the learner from overfitting. Note that negative samples should have the same domain positive samples so that their cues are insightful to the learner, but positive and negative samples are not required to have the same domain as the base data.
To effectively use positive and negative samples in few-shot learning, the following conditions must be met: 1) The regularization guided by nul-of-distribution samples can be combined with traditional loss functions for classification; 2) The regularization should be applicable for both inductive and transductive inference; and 3) The requirement on negative data should be minimal ie., does not need any sort of labels except for aforementioned conditions.
4.1 Refining prototype with distractor samples
We formulate the regularization as new objective function for training. To capitalize negative samples to reduce the ambiguity, we propose leveraging the large-margin principle as in Large Margin Nearest Neighbors (LMNN) [59, 59]. In these works, Weinberger et al. propose loss function with two competing terms to Iearn a distance metric for nearest neighbor classification: "pull" term to penalize the large distance between the embeddings of two nearby neighbors, which likely belong to the same class, and a "push" term to penalize the small distance between the embeddings of samples of difference classes.
In this work, we seek learn prototypes for all categories instead of a distance metric. We do not have labels (i.e., categories) of all samples (e.g., transductive inference) but only the prior knowledge about whether a sample is in- or ouf-o--distribution. Thus, we adapt the above objective of margin maximization for these two groups, namely in- and out-of-distributtinn, with the distance function being the sum of distances from sample to all prototypes. The goal minimizing distances from positive samples to prototypes, while maximizing distances from negative samples to prototypes.

---

In our objective function, we keep the original "pull" term while introducing our new "push" term: we do not explicitly enforce large distances between positive samples and negative samples, but only attempt to maximize distances between prototypes and negative samples:
pos
K
Nueg
Lnaive
Y d( x.)
Σ j=1 A=1
d(Wk:
(2)
i=1
Note that is scaling factor of distance- classifier as in Equation 1. However, this objectíve does not take into account class assignments for positive samples. To tackle this problem, we use weighted distances between prototypes and samples to simultaneously optimize both objectives:
N
Npas
Nneg K
Emargin
T. d(we, (.)Solpp(k)p
W)]
>- (.),,x)))llp(xx,, W)] (3)
1=1
k=
Cpuit
Spich
where (t](]] is the stop-gradient operator. Intuitively, the positive sample will "pull" the prototypes to its location proportional to its distance the prototypes. Subsequently, the prototype of each class will move closer to the positive samples of that class. At the same time, the prototype is enforced to move away from the negative samples, thus discarding features that might lead to high similarity to out-of-distribution data.
We note that removing Sal.] in Equation 3 would result in different underlying objective, which empirically leads to a decrease in performance. Particularly, the objective without stop-grand gradient will also be compounded of auxiliary term for entropy maximization (see Section B). Thus, devoid of meticulous regularization would deteriorate the performance. The above observation is in line with Boudiaf el al. [2]. In summary, POODLE optimizes the classifier on novel tasks with the following objectives:
LPOSOLE Lee + a Cpull 3 passh
where and control the "push" and "'uull" coefficients respectively and E denotes the standard cross-entropy loss. To the best of our knowledge, the proposed objective is the first loss function (in test phase) that be effective for both inductive and transductive inference. Please the supplemental document for the pseudo-code (Section C).
4.2 On negative samples
Choice of negative samples. For in-domain FSL, we simply leverage the base data as negative samples. For cross-domain FSL, one approach would be using a set of samples drawn from classes that are disjoint to novel classes C, negative examples. However, in some cases such negative examples might not be available. Thus, we consider another approach where we have a set of unlabeled data of set of classes C, from the target domain (C,, might overlap with C,)) negative examples, similar to [41]. We name the above two approaches as disjoint and noisy negative sampling, respectively in the context of cross-domain FSL.. Intuitively, when the number of categories of unlabeled data (i.e., |Cu) grow larger, the noise of mixing positive and negative samples in noisy negative samples pool will be reduced.
The burden of additional data. It is worth pointing out that POODLE does not break the setup of FSL as it does not require any additional data of novel classes, thus, still has a few samples from novel tasks. As POODLE obligates additional data in the form of OOD samples, one might ask how practical the algorithm is. For in-domain FSL, the negative samples can be casily obtained by adopting the training data as we already know C. n C. = 2. Eyen in extreme case where we only have access to empirically find that (fz-normalized) uniformly random noise can work
the pretrained models, surprisingly well (Section 4.3). For cross-domain FSL, even though POODLE requires additional OOD data, we find that these samples can be: noisy with both positive and negative samples; and 2) fairly efficient- in our experiments, we can "reuse" 400 OOD samples for all tasks, which is very efficient compared other methods that might use up to 20% unlabeled data of training set [41].
4.3 Intriguing effectiveness of uniform random features
One caveat of POODLE is the obligation of accessing to OOD samples. Fortunately, we find that POODLE can use the uniform random features on the hypersphere as negative samples for in-domain
5

---

FSL. Particularly, we can uniformly sample latent vectors from S as an alternative to the distractor samples from base training data, thus, eliminating the need for accessing to training data.
Our use of uniform distribution as negative examples is inspired by that feature uniformity is a desirable property for contrastive loss [57, 5], and so good representation prefers such uniformity. The problem of uniformly distributing points on the unit hypersphere is related to minimizing pairwise loss [31, IJ. which links well to the theory of supervised classification with softmax and cross-entropy (equivalent to minimizing pairwise loss or maximizing mutual information) [3, 45].
Therefore, using uniform distribution negative samples works for in-domain FSL because they will approximate the features of samples from base training data. For cross-domain FSL, this approach will fail because random features (which are similar to samples from the training domain) have large discrepancy with the target domain, thus, not inducing meaningful cues. Intuitively, the more similar between domains of OOD and test samples, the higher performance gain POODLE can achieve. Our experiments in Section 5.4 empirically prove aforementioned postulation.
5 Experiments
In this section, conduct extensive experiments to demonstrate the performance gain of our method on standard inductive, transductive, cross-domain FSL. To demonstrate the robustness of our method across datasets/networkk architectures, we keep the hyperparameters fixed for all experiments.
5.1 Experimental setup
Datasets. We evaluate our approach on three common FSL datasets. The mini- miii-emmgeeet dataset [56] consists of 100 classes chosen from the ImageNet dataset [48] including 64 training, 16 validation, and 20 test classes with 60,000 images of size 84 84. The tiered-lmagene [46] is another FSL dataset which is also derived from the ImageNet dataset with 351 base, 97 validation, and 160 test classes with 779,165 images of size 84 x 84. Caltech-UCSD Birds (CUB) has 200 classes split into 100, 50, 50 classes for train, validation and test following [6]. Furthermore, we also carry out experiments on iNaturalist 2017 (iNat) [54], ET [22], and ISIC-2018 (ISIC) [7] for the (extreme) cross-domain FSL. The description for these datasets can be found in the supplemental document (Section D.I).
Implementation details. We use ResNet12 as our feature extractor. It is a residual network [61] with 12 layers split into 4 residual blocks. For pre-training on the base classes, we train our backbones with the standard cross-entropy loss for 100 epochs. The optimizer has a weight decay of 54- and the initial learning rate of 0.05 is decreased by factor of 10 after 60, 80 epochs in mini-ImageNNtt and 60, 80, 90 epochs in pered-ImageNet We use the batch size of 64 for all the networks. For fine-tuning on the novel classes, we utilize Adam optimizer [32] with fixed learning rate of 0.001, 31 = 0.9, 82 = 0.999, and do not use weight decay. The classifier is trained with 250 iterations. The coefficients of "push/pull" loss are a and 3 = 0.5 respectively. For negative samples, we randomly select K = 400 samples from the out-of -itto--diitribution samples pool for each novel task. We evaluate the performance of POODLE in 5-way- --shot and 5-way-5-shn settings on 2000 random tasks with 15 queries each.
5.2 Standard FSL
We first experiment with the standard FSL setup (also known as in-domain FSL.) in which the number of query samples is uniformly distributed among classes.
Evaluation with various baselines. Table shows the results of our approach with various baselines, which described in Section 3, with the inductive inference (positive samples are support images). As can be seen, our approach consistently boosts the performance of all baselines by a large margin (1- 3%). Interestingly, the POODLE-R R variant outperforms the CE loss variant significantly and performs comparatively with the POODLE-B variant. We hypothesize that the normalized representations the samples extracted from the base classes are uniformly distributed.
Table demonstrates the efficacy of each loss term of POODLE in transductive inference (positive samples are support and query images). We can that using the "pull" loss with query samples

---

Table 1: Comparison to different baselines for the standard FSL (a.k.a in-domain FSL) mini- ImageNet, sered-Imagent and CUB in the inductive setting with Resnet-12 as backbone. Here, POODLE-B and POODLE-R indicate that the negative samples are sampled from the base classes and the random uniform distribution respectively.
mini-ImageNet 5-shot
fiered-ImageNet
CUB
Baseline
Variant
1-shot
1-shot
5-shot
1-shot
5-shot
CE Loss
61.33
80.76
67.42
84.44
75.60
89.88
Simple
POODLE-B
63.79
81.41
69.26
84.97
75.96
90.06
POODLE-R
64.38
81.35
69.34
85.03
76.22
89.99
CE Loss 64.57
82.89
68.26
85.09
76.39
91.10
Rot
POODLE-B
66.78
83.65
69.74
85.45
77.26
91.30
POODLE-R
67.20
83.54
69.86
85.56
77.35
91.43
CE Loss 65.91
82.95
69.43
84.93
79.70
92.28
Rot KD
POODLE-B 67.50
83.71
70.42
85.26
80.23
92.36
POODLE-R 67.80
83.50
70.47
85.24
80.05
92.37
Table 2: Comparison to different baselines mini-ImageNet, fiered-Imagenet and CUB with and
without the proposed loss the transductive settings. Here, Loaut and L indicate that the negative samples are sampled from the base classes and the random uniform distribution respectively.
mini-ImageNNe 5-shot
tiered-ImageNet
CUB
Baseline
Loss
1-shot
1-shot
5-shot
1-shot
5-shot
CE
61.33
80.76
67.42
84.44
75.60
89.88
CE + Lpail
70.91
83.04
75.58
86.10
84.91
91.60
Simple
CE Lpull Cyush
74.21
83.71
78.72
86.57
87.04
91.84
CF pplli pussh
74.82
83.67
79.24
86.64
87.13
91.60
CE CE Lppll
64.57
82.89
68.26
85.09
76.39
91.10
74.42
85.20
76.59
86.75
85.93
93.02
Rot
CF Lpplt C.
Coushh
77.30
85.91
79.74
87.25
88.68
93.23
CE + Lplli C push
77.69
85,86
80.31
87.33
89.07
93.35
CE
65.91
82.95
69.43
84.93
79.70
92.28
CE + Lpuil
74.90
85.09
77.07
86.50
88.22
93.68
Rot + KD
CF + Lplll Lb
Lpush
77.56
85.81
79.67
86.96
89.88
93.80
CE + Lpull puush
77.24
85,53
79.97
86.91
90.02
93.68
improve the inductive baseline significantly, being as effective as other transductive algorithms. Combining with "push" term, the classifier further enhanced.
We also conduct experiments with additional loss functions including self-supervised loss (SSL) and knowledge distillation (KD) in Figure 2a. Our approach consistently improves the generalization of all baselines without additional computation cost (in the training phase) as justified in Figure 2b. The improvement of POODLE other backbones and datasets is reported in the supplemental document (Section D.2).
Comparison to the state-of-the-arr approaches. We report the performance of our network in comparison with sfate-ofthhe-an art methods in both transductive and inductive settings (with and without information from the query images) in Table 3. We can see that our approach remarkably improves performance the baseline and achieves a comparable performance with the state-of-the-art approaches in the heree--IaaeeNe prior work in both inductive and transductive settings. Experimental results on other backbones are reported in the supplementary document (Section D.2).
In mini-ImageNNe and CUB we significantly outperform the

---

Traiing hime
GPU Menory
Accurcc Gain
Denscnutt WINN-12-8
Resmet-12
MabileNos
Comr-44512
Baseline
Our
KD
SS1.
SSL
KD
Our
Training time (hours)
GPU Memory (Gb)
(b)
Figure 2: (a) Effectiveness of our approach applied to standard FSL. (a) Our approach consistently yields accuracy gains on different backbones, in comparison with those obtained by self-supervised loss (SSL) and knowledge distillation (KD) in 5- 5-way- I-sho protocol setting. (b) Total time and memory cost (in training stage) when adopting SSL, KD, and our method. Note that apply KD for T 2 generation in [53], No large overhead is incurred in method.
Imagenee with inductive
Table 3: Comparison to the state-of the-art art methods on mini- Mini-ImmeeNet, tiered Iraee-etmeeeetttt and CUB using inductive and transductive settings. The results obtained by our models (blue pearl-shaded) are averaged over 2,000 episodes.
mini-ImageNet 5-shot
tiered-ImageNer
CUB
Method
Transd.
Backbone
I-shot
I-shot 5-shot
I-shot
5-shot
MAML [15]
ResNet-18
49.6
65.7
68.4
83.5
RelatNet [24]
ResNet-18
52.5
69.8
68.6
84.0
MatchNet [56]
ResNet-18
52.9
68.9
73.5
84.5
ProtoNet [50]
ResNet-18
54.2
73.4
73.0
86.6
Neg-cosine [37]
X
R
62.3
80.9
72.7
89.4
MetaOpt [33]
ResNet-12
62.6
78.6
66.0
81.6
SimpleShot [58]
ResNet-188
62.9
80.0
68.9
84.6
68.9
84.0
Distill [53]
ResNet- 12
64.8
82.1
71.5
86.0
Rot + KD + POODLE
ResNet-12 12
67.80
83.72
70.42
85.26
80.23
92.36
RelatNet [24]
ResNet- 12
52.4
65.4
TPN [39]
ResNet-12 12
59.5
75.7
TEAM [43]
ResNet-18
60.1
Ent-min [10]
ResNet- 12
62.4
74.5
68.4
83.4
CAN+T [24]
ResNet- 12
67.2
80.6
73.2
84.9
LaplacianShot [66]
ResNet-
72.1
82.3
79.0
86.4
81.0
88.7
TIM-GD [2]
ResNet- 18
73.9
85.0
79.9
88.5
82.2
90.8
Simple POODLE
ResNet-12 12
74.21
83.71
78.72
86.57
87.04
91.84
KD + POODLE
ResNet-12
77.56
85.81
79.67
86.96
89.93
93.78
5.3 Cross-domain FSL
In this section, we conduct experiments to demonstrate the efficacy of our approach even with challenging task: extreme cross-domain FSL first introduced [21, 41]. Many FSL algorithms are known to fail in such a challenging scenario [6].
Recall that have two setups for cross-domain FSL (Section 4.2), in which cases, the negative samples are drawn from disjoint and noisy negative samples pool. Here, we provide the detailed setting of each scheme when transferring network trained on mini- Imagentt to target domains. It is worth mentioning that we only consider inductive inference for comparison with prior work.
Disjoint negative samples. As mentioned before, we evaluate the results of cross-domain FSL on test split of these datasets while employing the train split to draw negative samples. Since the number of categories in EuToSAT and ISIC is relatively small (10 and respectively) compare to the number

---

Table 4: The results of the domain-shift setting from mini-Imagent Imagenet to CUB, iNat, ISIC, and EuroSAT with Rot + KD baseline. The results obtained by our models (blue pearl-shaded) are averaged over 2,000 episodes. The baselines with notation show the results using the setup of disjoint negative samples, and the baselines without the show the results using the setup of noisy negative samples.
CUB
iNat
ISIC
EuroSAT
Baseline
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Transfer [41]
30.71
43.08
60.73
80.30
SimCLR [41]
26.25
36.09
43.52
59.05
Transfer SimCLR [41]
32.63
45.96
57.18
77.61
STARTUP (no SS) [41]
32,24
46,48
62.90
81.81
STARTUP [41]
32.66
47.22
63.88
82.29
Rot + KD + CE
49.98
69.44
48.90
66.58
32.35
43.81
65.00
80.01
Rot + KD + POODLE-R
50.04
69.78
48.94
66.87
32.21
43.87
63.97
79.85
Rot KD + POODLE-B
52.61
70.78
50.62
67.31
33.56
44.17
66.21
80.51
Rot KD + CE
50.28
69.64
48.80
66.72
Rot KD + POODLE-R
50.25
69.97
48.84
67.03
Rot KD + POODLE-B" 53.11
70.96
50.46
67.45
of "way" in novel task (5), we only concern with iNat and CUB. Precisely, we follow [6] and [58] to split CUB and iNat respectively.
Noisy negative samples. We assume that have 20% unlabeled data from the target domain and the rest 80% of data is used for testing. Although large number of classes are beneficial to POODLE diseussed above, also carry out experiments with ISIC and EuroSAT to evaluate the performance of extreme cross-domain FSL with our approach.
From the Table 4 we can see that the improvement of POODLE with the disjoint negative samples (the bottom rows) and the noisy negative samples (the middle rows) when transferring knowledge from mini-Imagelle
to iNat, CUB, ISIC, and EuroSAT. We can observe that despite the extreme gap between target/source domains and a very noisy mix of in- and nnt-of-distribution samples in negative samples pool (in case of EuroSAT and ISIC), POODLE can successfully boost the performance of baselines in all experiments. We also report the performance of other approaches, which have same setup noisy negative samples, in Table 4.
5.4
Ablation study
In this section, we present results of some presentative ablation study to get more insight of POODLE behaviors. For more in-depth experiments, we refer reader supplementary document.
5.4.1 The effect of domain diserepancy between positive-negative samples
Intuitively, the more similar between domains of OOD and test samples, the higher performance gain POODLE can achieve. To understand how our method works when OOD data is from various domains, we train our classifier and compare the performance when using random uniform distribution and other datasets as negative examples. The result of this experiment reported in Table 5. We can observer that the more similar of OOD domain to test domain, the higher the performance gain we can achieve. Thus, we should always aim to leverage the OOD samples test domain.
On the performance of random features: For in do-domain FSL i.e., test domains mini- Imagenet and tiered- dered-Imagenne Imagenet, using uniform examples has the best and second-best accuracy when tested on Mini-ImageNer and MrreedIImgeNee, respectively. As aforesaid, the uniform random features will reflect the distribution of mini- mini-Imagenet samples (which the network is pre-trained on). Thus, random features are effective for in-domain FSL.
For cross cross-domain FSL (i.e., test domains are CUB and EuroSAT), using random uniform features which is approximately equal to using samples from discrepancy between the source and target domain is extremely large, e.g., animal (mini- Imagie)magennt) vs satellite images (EuroSAT).
does not work because the

---

Table 5: Evaluating (simple) Resnet- 12 trained mini-Imagenet different test/OOD domains in the 1-shot inductive protocol (10,000 episodes). random, mini, tiered, CUB, EuroSAT denote fiered- fiered-Imagenet, CUB, EuroSAT dataset respectively. The OOD/test samples drawn from the standard train/test split of each dataset, respectively. The 95% confidence interval is roughly 0.20 for all results.
OOD domain mini
w/o OOD
random
liered
CUB
EuroSAT
mini
61.63
64.30
64.08
63.72
62.71
61.76
liered
63.04
64.28
64.01
64.50
63.85
62.78
Test domain
CUB 48.55
48.88
49.01
49.10
51.40
49.18
EuroSAT 65.18
63.85
64.58
64.25
64.70
66.04
Table The results the various approaches to learn a better classifier with pretrained Rot + KD buseline mini-lmagenet. The results are obtained by averaging over 10,000 episodes.
Method
1-shot
5-shot
baseline
66.32 0.20
82.99 + 0.13
w/ learning cosine classifier
66.66 = 0.20
82.84 + 0.13
2) w/ naive OOD
66.36 = 0.20
83.08 + 0.13
3) w/ large-margin w/o negative samples
66.32 = 0.20
83,01 + 0.13
4) w/ label smoothing
66.13 = 0.20
80.81 + 0.13
POODLE-B POODLE-R
67.84 0.20
83.72 + 0.13
68. + 0.20
83.60 = 0.13
5.4.2 Comparison against simple approaches for enhance pretrained models
To better understand the effectiveness of our method, we also compare to four methods that learn better classifiers on novel classes with pre-trained features. The results are reported in Table 6.
Learning cosine classifier: The classifier in inference phase fine-tuned using CE loss with the support samples of each task. In our experiments, this approach does not bring any meaningful improvement similar [58].
2. Naive OOD: we fine-tune (k + 1)-way classifier with additional class for OOD samples using CE loss. Naive OOD performs worse because the OOD samples are drawn from several classes of the training set, which are well-clustered and separated, cannot find "prototype" with linear classifier to match all of them.
Large-margin w/o negative samples: only use push term in POODLE. This approach is not better than cosine-distange because the initialized prototype (mean of all samples from specific class) already well-clustered and optimal i.e., close the ground-truth class prototype and far away from others for the observed samples.
Label smoothing: we use label smoothing for CE loss (ground- -rround-truth class has the probability of 0.9 and uniformly distribute 0.1 to the rest. This method does not work well because it only makes learned prototypes not be far away from samples from other classes
Conclusions and future work
In this work, have proposed the concept of leveraging -itto---iitribution samples set to improve the generalization of few-shot learners and realize by a simple yet effective objective function. Our approach consistently boosts the performance of FSL across different backbone networks, inference types (inductive/trassductive)) and the challenging cross-domain FSL.
Future work might seek to exploit different sampling strategies (i.e., how to select negative samples) to further boost the performance and reduce time/memory complexity; another interesting direction is enhancing the robustness of the classifier when we have both positive and negative samples in the same sampling pool; leveraging domain adaptation to reduce the need of in-domain negative samples is also promising research direction.
10

---

References
[I] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
[2] Malik Boudiaf, Ziko Imtiaz Masud, Jérôme Rony, José Dolz, Pablo Piantanida, and Is- mail Ben Ayed. Transductive information maximization for few-shot learning. arXiv preprint arXiv:2008. / /297, 2020.
[3] Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European Conference on Computer Vision, pages 548-564. Springer, 2020.
[4] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535-541, 2006.
[5] Ting Chen and Lala Li. Intriguing properties of contrastive losses. arXiv:2011. 1.02001,.0003,,. 2020.
arXiv preprini
[6] Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Frank Wang, and Jia Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904, 04232, 2019.
[7] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helha, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint ar0iv: 2019.
[8] Rajshekhar Das, Yu-Xiong Wang, and Jose MF Moura. On the importance of distractors for few-shot classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9030-9040, 2021.
[9] Wim De Neys, Walter Schaeken, and Géry d' d'Ydeweee. Working memory and everyday condi- tional reasoning: Retrieval and inhibition of stored counterexamples. Thinking & Reasoning, 1144)3399-31, 2005.
[10] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint wwxXiv:1999.027220 2019.
[11] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-award few-shot transfer. arXiv preprint arXiv:2007.11499 1498, 2020.
[12] Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2051 2051-2060, 2017.
[13] Dorothy Edgington. On conditionals. Mind, 104(414):235-329, 1995.
[14] Reuben Feinman and Brenden M. Lake. Learning inductive biases with simple neural networks. Proceedings of the 40th Annual Meeting of the Cognitive Science Society, 2018.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1133 135. PMLR, 2017.
[16] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pages 1568-1577 PMLR, 2018.
[17] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607- 1616. PMLR, 2018.

---

[18] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with supervipevii... International Conference on Computer Vision (ICCV), 2019.
[19] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803.0778, 2018.
[20] Yiluan Guo and Ngai-Man Cheung. Attentive weights generation for few shot learning via information maximization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[21] Yunhui Guo, Noel Codella, Leonid Karlinsky, James Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In European Conference on Computer Vixion, pages 124-141. Springer, 2020.
[22] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226 2019.
[23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini
003.02531,,1222222 2015.
[24] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. arXiv preprint arXiv:1910.07677, 2019.
[25] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint IrXiv::704.08800 2017.
[26] Shell Xu Hu, Pablo G Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. In International Conference on Learning Representations (ICLR), 2020.
[27] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700 7700-4708, 2017.
[28] Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11719-11727, 2019.
[29] Philip N Johnson-Laird. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43): 18:433):8823111 001118822818558 18250, 2010.
[30] Philip Nicholas Johnson-Laird. Mental models: Towards cognitive science of language, inference, and conscionsness. Number 6. Harvard University Press, 1983.
[31] Olav Kallenberg and Olav Kallenberg. Foundations of modern probability, volume 2. Springer, 1997.
[32] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2014.
[33] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[34] Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pages 2927-2936. PMLR, 2018.
[35] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for seo-shot learning, arXiv preprint
2017.
12

---

[36] Moshe Lichtenstein, Prasanna Sattigeri, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. Tafssl: Task-adaptive feature sub-space learning for few-shot classification. In European Conference Computer Vision, pages 522-532 Springer, 2020.
[37] Bin Liu, Yue Cao. Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In European Conference on Computer Vision (ECCV), 2020.
[38] Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In European Conference on Computer Vision (ECCV), 2020.
[39] Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning, arXiv preprint arXiv: 10002, 2018.
[40] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pages 69-84. Springer, 2016.
[41] Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. In International Conference on Learning Representations, 2021.
[42] Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5822-5830, 2018
[43] Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang. Tiejun Huang, and Yonghong Tian. Trans- ductive episodic-wing wise adaptive metric for few-shot learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3603-3612, 2019.
[44] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7229-7238 2018.
[45] Zhenyue Qin, Dongwoo Kim, and Tom Gedeon. Rethinking softmax with cross-entropy: Neural network classifier as mutual information estimator. arXiv preprint arXiv:1911.1066 2019.
[46] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference Learning Representations (ICLR), 2018.
[47] Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning, pages 2940-29209090 2949. PMLR, 2017.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. In International booktitle of Computer Vision (IJCV), 2015.
[49] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero. and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations (ICLR), 2019.
[50] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems (NeurIPS), 2017.
[51] Long-Chyi Su. Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-shot learning? In European Conference Computer Vision, pages 645-666. Springer, 2020.
[52] Flood Sung. Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1199-1208, 2018.
13

---

[53] Yonglong Tian, Yue Wang, Dilip Krishman, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding all you need? In European Conference on Computer Vision (ECCV), 2020.
[54] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8769-8778, 2018.
[55] Niki Verschueren, Walter Schaeken, and Gery d'Ydewalle. Everyday conditional reasoning: A working memory-dependdee tradeoff between counterexample and likelihood use. Memory & Cognition, 33(1):107-119) 2005.
[56] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[57] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pages 9929-9939. PMLR, 2020.
[58] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Sim- pleshot: Revisiting nearest-neighhor classification for few-shot learning. In arXiv preprint arXiv:/9/1.0er 1.0623, 2019.
[59] Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in neural information processing systems, pages 1473-1480, 2006.
[60] Davis Wertheimer and Bharath Hariharan. Few-shot learning with localization in realistic set- tings. Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 3558-6567, 2019.
[61] Han-Jit Jia Ye, Hexiang Hu, De-Chusa Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8805 1805-8814. 14. IEEE, 2020.
[62] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Learning embedding adaptation for few-shot learning. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[63] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[64] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifi- cation with diffcrentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12203-12213, 2020.
[65] Manli Zhang, Jianhong Zhang, Zhiwu Lu, Tao Xiang, Mingyu Ding, and Songfang Huang. lept: Instance-level and episode-leve pretext tasks for few-shot learning. In Proc. Int. Conf. Learn. Represent. pages 1-16, 2021.
[66] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In International Conference on Machine Learning, pages 11660-11670. PMLR, 2020.
14


Reviews:
[{"review_id": "v_bB9cr2xpI", "reviewer": "Reviewer_FMo7", "summary": "The proposed approach leverage the out-of-distribution (OOD) samples to improve the few-shot learning (FSL). The OOD samples were exploited from easily available base classes, although random uniform sampling shows better results. Maximizing the distance between the OOD and positive samples helps to mitigate the ambiguity between the classes. The proposed approach is applicable for the inductive and transductive settings.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has the following pros and cons:\nPros:\n1: The proposed model is simple. Negative samples are easy to obtain.\n2: The paper is written well and easy to understand.\n3: Good result in both (inductive and transductive) settings.\n\nCons:\n1: The paper's main contribution is to use the negative samples to design a margin-based loss that minimizes the confusion between similar contexts. The margin-based loss is not novel. Many previous supervised learning approaches use this type of loss frequently. In the FSL setting, it may be novel, but this contribution is not significant.\n\n2: The result in Table-3 and Table-4 are confusing. Paper reported result by a combination of Rot+KD, which is not a fair comparison with the previous approach. The earlier method can be significantly improved using the self-supervised learning model. The paper combines the earlier model self-supervision, KD and max-margin loss to improve the result. There is no significant improvement in terms of methodology for the FSL. The results reported in the Table-1 and 2 with \n\n3: Few popular are ignored and not compared in the paper, e.g. [1] [2] [3]. I request to the author please compared these approaches and also include the recent baseline. \n\n\n[1] Adaptive Subspaces for Few-Shot Learning, CVPR-20\n[2] META-LEARNING WITH LATENT EMBEDDING OPTIMIZATION, ICLR-19\n[3] Attentive Weights Generation for Few Shot Learning via Information Maximization, CVPR-20\n[4] ECKPN: Explicit Class Knowledge Propagation Network for Transductive Few-shot Learning, CVPR-21\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5"}, "extra_text": {}}, {"review_id": "fA-r_6n63Y", "reviewer": "Reviewer_C2g5", "summary": "The paper considers the problem of few-shot learning, and proposes to leverage out-of-distribution data as negative examples to regularize the few-shot model. The intuition is that in cases where there is ambiguity because of lack of training samples, the out-of-distribution data as counter examples, would help the model eliminate the less generalizable solutions. \n\nThe out-of-distribution data here refers to samples from classes other than classes of the target task.There is an emphasis that these counter examples should be from the same domain as the target task. \n\nThe few-shot learning setup in this paper is as follows:\n1. Use a frozen pre-trained model to get the representation of given data points from the target task.\n2. Learn a prototype representation for each class, `W = {w1,...wk}` (this is initialized to the mean of the representations of the examples from the training set for each class).\n3. Given a query example, compute the distance of its representation with the prototype representation of each class, and apply softmax on the negative scaled distance scores to get predictive probability distribution over all classes.\n\nIn the standard case, the few-shot classifier (with W, and a scalar that is applied to the distance scores as parameters) is simply trained with cross entropy loss. This paper suggests adding an auxiliary loss inspired by Large Margin Nearest Neighbors (Equation 3 and 4 in the paper).\n\nThe paper considers both in-domain and cross domain FSL. For in-domain FSL, they sample negative examples from pre-training data. For cross-domain FSL, since pre-training data has a different domain than the target task, they propose to use a set of samples drawn from classes that are disjoint to the target task. In cases where such negative examples might not be available they propose using a set of unlabeled data of a set of classes from the target domain that overlap with target classes.\n\nAdditionally, the authors discuss the case where there is class imbalance in the few-shot training samples. They propose having higher weights for the push terms of the regularizer loss to avoid the prototype representations to collapse.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** The main idea in the paper is based on contrasting between representations of in and out of distribution samples from the target domain (where the out of distribution-ness is rooted in the samples belonging to a separate set of classes). Finding a way to realize this idea inspired by LMNN, and evaluating its effectiveness empirically is the main contribution of this paper. \n\n**Quality and Clarity**: I find some of the arguments and claims  in the paper a bit vague.\n\nIn the result provided in Table 1 in the paper, POODLE-R performs comparatively with the POODLE-B. First of all I am not sure I correctly understand what happens in POODLE-R, does that mean you simply sample from a uniform d-dimensional distribution? \nI think there should be a more elaborate explanation for this than hypothesizing that the normalized representations of the samples extracted from the base classes are uniformly distributed. If simply samples from a uniform distribution is enough to provide “negative examples” to the auxiliary loss function, how does this fit into the intuitions provided in the paper behind why this method would work to begin with? and why does this not work for all cases, e.g., result in Table 4?\n\nIn line 42 as one the contributions of the paper the authors mentions:\n`We propose a novel yet simple approach to learn the inductive bias of deep neural networks for FSL by leveraging out-of-distribution data.`\nI can not figure out how to interpret this sentence. As the inductive bias is not something that the model learns from data and is not clear to me what the inductive bias refers to here.\n\nIn line 167 the authors claim \n`We note that removing SG[·] in Eq. 3 would result in a different underlying objective, which empirically leads to a decrease in performance.`\nI understand how the stop gradient operation in  Eq. 3 could lead to a difference, but appreciate it if the authors could explain how removing it from the equation leads to a different objective?\n\n**Significance** :The method seems to be applicable on top of other techniques to improve FSL performance, and the results show significant improvements over baselines based on the numbers reported in the paper. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the potential problem if the `number of few shot samples is not balanced` over classes. What happens in this case is that all prototypes are learned to have high similarity to the samples of dominated classes and the predicted classes. Hence, the prototype features would be similar across classes. The authors argue that in this case, increasing the coefficient of the push term of the loss helps.\nAnother point that is implicitly mentioned is the `sensitivity of the proposed technique the negative sample selection strategy`. \nFinally, I think I would add `lack of theoretical support for the proposed method` to this list.\n\nSocietal impact: The authors have generally explained the importance of having few shot learning models. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "XxDR9iJtwRX", "reviewer": "Reviewer_5hh7", "summary": "This paper aims to address the problem of few-shot learning by exploiting unlabeled data from outside target novel classes. The main contribution is that it uses these out-of-distribution data as negative examples and learns a target classifier that simultaneously discriminates among the target classes and the negative class. To this end, the negative class is constructed by either sampling samples from base classes or from relevant classes of the target domain. A large-margin based prototype classifier is trained accordingly, which pulls together positive samples and pushes apart negative samples. A modified version of the classifier using stop-gradient operator is further introduced. The approach is tested in a variety of few-shot classification settings, including in-domain, cross-domain, and imbalanced distribution, and for both inductive and transductive scenarios, and compared with several baseline methods.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n1) The authors tackle an important and challenging problem of few-shot classification with out-of-distribution data. The proposed approach is simple and interesting.\n2) The proposed approach can be viewed as learning a large-margin classifier on novel classes by using additional “negative” examples and with pre-trained feature extractor on base classes. There is a lack of discussion on relevant approaches that also focus on learning better classifiers on novel classes with pre-trained features. See the first point in the “quality” comments.\n\nQuality: \n1) While a series of experiments were conducted, the comparisons did not fully validate the proposed approach. In particular, the main comparisons focused on baselines that train the feature extractors in different ways (e.g., with standard CE loss, with additional rotation self-supervision and with knowledge distillation). While these comparisons did show that the proposed approach works with different types of pre-trained features, to me these are just one aspect of the comparison. What were supposed to be the main comparisons would be comparing against approaches that also focus on learning better classifiers on novel classes with pre-trained features. There are different strategies: for example, (a) learning cosine classifiers as in [3]; (b) learning large-margin classifiers without out additional negative examples; (c) refining few-shot classifiers as in [Learning to learn: Model regression networks for easy small sample learning, ECCV, 2016]; (d) performing data augmentation at test time as in [Free lunch for few-shot learning: Distribution calibration, ICLR, 2021]; etc. Are the proposed approach better than these existing works?\n2) The proposed approach uses large-margin classifiers to pull positive samples and push negative samples. An alternative is contrastive learning, which is widely used recently. How is the connection, difference, and comparison between these two?\n3) How does the size of negative examples K affect the performance? In Table 10 of the supplementary material, it shows that K is helpful when K=500. Does the performance further improve, saturate, or drop with larger K?\n4) Table 3 and Table 4 miss state-of-the-art approaches. Also, from https://paperswithcode.com/dataset/miniimagenet-1 etc., the proposed approach is worse than state of the art.\n5) The authors mentioned that the proposed approach leads to a little overhead at inference. It would be useful to provide more quantitative analysis, like comparing the classifier training time.\n\nClarity:\n1) The paper generally reads well.\n2) Some important technical details are not clear. For example, the stop gradient operator is critical in this paper (Line 162) but without enough details; what is the difference between POODLE-B and POODLE -R (Table 1 caption). While some of the explanations are provided in the supplementary material, I feel they should be discussed in the main paper.\n\nSignificance:\n1) Experimental evaluations demonstrate the effect by introducing the out-of-distribution data as negative examples.\n2) The proposed approach assumes available out-of-distribution examples. This is sometime unrealistic in practice for cross-domain settings, as mentioned by the authors as well (Line 175-181). The authors considered disjoint and noisy negatives for cross-domain settings. A more realistic scenario is in fact to directly use examples from base classes as negative. Does this work?\n3) If we consider the performance in joint base and novel label space which is important in some real-world applications, is the proposed approach still effective?\n4) Following the standard practice in few-shot evaluation, it would be helpful to include the 95% confidence intervals in the tables.\n\nMy initial rating is based on the comments above. I think the paper can be substantially improved by a addressing these comments. \n\nPost Rebuttal:\n\nI do appreciate the efforts and additional experiments that the authors made in the rebuttal. While this paper proposed an interesting setting/approach to few-shot classification, the technical novelty is somehow limited regarding the use of large margin classifiers for few-shot classification, as mentioned by other reviewers as well. From the experimental comparisons in the original submission and in the rebuttal, the proposed approach is comparable to and does not show significant improvements over the state-of-the-art methods, such as the distribution calibration paper. But this approach requires additional examples from related domains, while the distribution calibration paper does not. Also, such reliance on additional examples from related domains might make the approach inapplicable in practice, where this kind of data is difficult to acquire. In addition, some important details are vague and unclear, which is not fully clarified in the rebuttal. Some of these concerns are addressed in the rebuttal, but not fully clarified. This makes the paper not ready for this NeurIPS. The presentation and the writing flow can be improved as well. I encourage the authors to continue this line of work for future submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It looks reasonable to me.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GzfN-JkfY_Z", "reviewer": "Reviewer_FGzB", "summary": "In this paper, authors have leveraged out-of-distribution samples to improve the generalization of few-shot learners. The main idea is to use the large margin principle in metric learning and add terms to the loss function to encourage the learned prototypes to be far from the out-of-distribution samples and to be close to support data. In summary, in addition to standard cross-entropy loss, authors have used to extra loss functions for penalizing  the large distance between the embeddings of two nearby neighbors, and the small distance between the embeddings of samples of difference classes.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and presents an interesting idea and direction. However, there are few issues that make the paper not ready for publication in NeurIPS.\n\n- The core idea of using large-margin principle is not novel and as authors have mentioned, it was first introduced in \"large margin nearest neighbors\" papers.\n- The regularization terms (i.e., the push and pull loss functions) have introduced to new hyper-parameters (\\alpha and \\beta) but there is no discussion on how these two could be chosen effectively.\n- In Figure 2, authors have mentioned that their method \"consistently yields accuracy gain on different backbones\" compared to other approaches. However, there is some issue with the MobileNet case. \n- Comparing to other SOTA methods, authors have mentioned: \"We can see that our approach remarkably improves the performance of the baseline and achieves a comparable performance with the state-of-the-art approaches in the tiered-ImageNet.\" However, this is not true. the difference between other SOTA methods and the proposed approach on the tiered-Imagenet is basically the same as the difference between other approaches and SOTA methods on mini-Imagenet and CUB. Therefore, the performance of the proposed approach is on par with other SOTA methods, roughly speaking.\n- Re table 5, authors have mentioned if we increase the coefficient of the push, we discourage the prototypes from learning the aforesaid features and this will help the POODLE method to gain better performance. However, in most of the cases \\beta=0.75 is giving better results compared to \\beta=1 and this is not explained.\n---------------------------------\nPost-Rebuttal:\n- Thanks to authors for their comments regarding my concerns. I still believe that the novelty of this paper is limited and the presentation has issues which make the paper not ready for NeurIPS. I highly recommend that authors address the concerns that reviewers brought up and submit to the next ML venue.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors have not talked about potential societal impacts such as fairness of their method. It would be great if a short discussion is added on the limitations of the proposed approach.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using unlabeled samples outside of target classes to improve few-shot learning. The main idea is that discriminating these samples from in-distribution samples would allow the model to improve feature learning.

Reviewers recognize that the proposed method improves over SOTA but they point to limited technical novelty regarding the use of large margin, marginal improvements over SOTA methods, the reliance of methods on extra data as the shortcomings of this work. However, all reviewers believe that after improving the presentation of the paper and the empirical results, this paper would make a great publication.

Given the above concerns, I recommend rejecting the paper and resubmitting it after taking reviewers' suggestions into account.
Decision: Accept (Poster)