POODLE: Improving Few-shot Learning via Penalizing Out-of-Distribution Samples
Authors: Duong Hoang Le
Khoi Duc Nguyen
Khoi Nguyen
Quoc-Huy Tran
Rang Nguyen
Binh-Son Hua
Venue: NeurIPS 2021 Poster (2021)
Keywords: few-shot learning

Abstract: In this work, we propose to use out-of-distribution samples, i.e., unlabeled samples coming from outside the target classes, to improve few-shot learning. Specifically, we exploit the easily available out-of-distribution samples to drive the classifier to avoid irrelevant features by maximizing the distance from prototypes to out-of-distribution samples while minimizing that of in-distribution samples (i.e., support, query data). Our approach is simple to implement, agnostic to feature extractors, lightweight without any additional cost for pre-training, and applicable to both inductive and transductive settings. Extensive experiments on various standard benchmarks demonstrate that the proposed method consistently improves the performance of pretrained networks with different architectures.

POODLE: Improving Few-shot Learning via
Penalizing Out-of-Distribution Samples
Duong H. VinAI Research v. wwuunngnh@@vinai..il 11
Khoi D. Nguyen VinAI Research
Khoi Nguyen VinAI Research ducminhkhoi1gmmm wathhho@@mmmii... com
choinggyenucd@gmal..comm
Quoc-Huy Tran Retrocausal, Inc.
Rang Nguyen
Binh-Son Hua VinAI Research & VinUniversity
VinAI Research
huyGretrocausal.cc
rangnhn@gmail.com com
Abstract
In this work, we propose to leverage out-of-distribution samples, i.e., unla- beled samples coming from outside target classes, for improving few-shot learn- ing. Specifically, we exploit the casily available out-of-distribution samples (e.g., from base classes) drive the classifier avoid irrelevant features by maximizing the distance from prototypes to out-of-distribution samples while minimizing that in-distribution samples (i.e., support, query data). Our ap- proach is simple to implement, agnostic to feature extractors, lightweight with- out any additional cost for pre-training, and applicable to both inductive and transductive settings. Extensive experiments on various standard benchmarks demonstrate that proposed method consistently improves the performance of pretrained networks with different architectures. Our code is available at /etpps:/gitthn www./VanATReeearcc/ppontoo
Introduction
Learning with limited supervision a key challenge to translate the research efforts of deep neural networks to real-world applications where large-scale annotated datasets are prohibitively costly to acquire. This issue has motivated the recent topic of few-shot learning (FSL), which aims to build a system that can quickly lear new tasks from small number of labeled data.
popular group of methods in FSL focus on strengthening the backbone network by various techniques, from increasing model capacity [6, 11], self-supervised learning (SSL. [18, 51, 65], to knowledge distillation (KI) [53]. With these techniques, few-shot methods are expected learn better representations that are more robust and generalized. However, even if the network can discover visual features and semantic cues, few-shot learners have to deal with key challenge the ambiguity: as we have only small amount of support evidence, there multiple plausible hypotheses at inference stage. Existing works, therefore, rely on developed inductive bias of the network (during the pretraining stage), such as shape bias [47, 14], to reduce the hypothesis space.
In this work, we view the classification problem conditional reasoning, i.e., "if X has then is Q". Human beings are good at learning such inferences, thus quickly grasping new concepts with minimal supervision. More importantly, humans learn new concepts context where have already had prior knowledge about other entities. According to mental models in cognitive science, when assessing the validity an inference, one would retrieve counter-examples, iee. which do not
First two authors contribute equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Class #1
Class #2
Query
Distractor
adult
child
child
dog
suit
casual
suit
suit
Figure 1: Illustration of advantages of counter-example data. The query image has features from both support classes (i.e., child and suit), which makes classification ambiguous. The prediction result would depend on the inductive bias prior knowledge of the network. By using an out-of-distribution sample, it becomes clear that the query result should favor Class 2.
lead to the conclusion despite satisfying the premise [9, 13, 29, 30, 55]. Thus, if there exists least one of such counter-examples, the inference is known to be erroneous.
Hence, we attempt equip few-shot learning with the above ability so that eliminate incorrect hypotheses when learning novel tasks in a data-driven manner. Specifically, we leverage out-of- distribution data, i.e., samples belonging classes separated from novel tasks as counter-examples for preventing the learned prototypes from overfitting to their noisy features. To that end, when learning novel tasks, we adopt the large margin principle metric learning [59] to encourage the leamned prototypes to be close to support data while being distant from out-of fittof-diittriutionn samples.
Our approach is complementary to existing works in FSL and could be combined to advance the state of the art. Moreover, our method is agnostic to the backbone network; thus, it does not have the need of training phase adopt in SSL and KD, while incurring just little overhead at inference (for fine-tuning prototypes with approximately 200 gradient updating steps).
In summary, contributions are follows:
We propose novel yet simple approach learn the inductive bias deep neural networks for FSL by leveraging out-of-distributio data. We empirically show that out-of-distributin data only require weak labels (i.e., in the form of whether a sample is in- or -itto---iitribution) even in challenging problems such as cross- FSL.
We introduce a new loss function to implement the above idea, which is applicable for both inductive and transductive inference. Our extensive experiments diffferent standard benchmarks show that the proposed approach consistently improves the performance of various network architectures.
We validate the effectiveness of our method in various FSL settings, including cross-domain FSL.
2 Related work
Over the past few years, considerable amount of research efforts [15, 16, 28, 34, 44, 56, 50, 64) have been invested in FSL. They can roughly be classified into two main categories: optimization-bused and metric-based approaches. Optimization cimiizzttioo-basedd methods [15, 16, 28, 34, 44] seek a meta-learner that quickly adjust the parameters of another learner new task, given only few support images. In particular, [35, 15] propose learning to initialize a classifier whose parameters can be obtained with small number of gradient updates on the novel classes. Metric-baser approaches [56, 50, 64] learn task task-agnostic embedding space for measuring the similarity between images. For example, Matching Networks [56] utilizes weighted nearest neighbor classifier, while Prototypical Networks [50] uses the mean features of support images as the prototype for each class. Recently, DeepEMD [64] adopts the Earth Mover's distance to compute the distance between image patches as their distance.
We use OOD samples and distractor samples interchangeably

---

To further boost the performance, recent works have incorporated additional techniques such as self-supervised learning and knowledge distillation during training, and transductive inference during testing, which we will review in the following:
Self-supervised learning. The goal of self-supervised learning is to learn representations from unlabeled data. For FSL, this could be achieved by combining the supervised main task with a self-supervised pretext task which includes predicting image rotations [19], predicting relative patch locations [12], or solving jigsaw puzzles [40]. A few works incorporating self-supervised learning for FSL have been developed recently, e.g., Gidaris et al. [18] suggest pre- vre-training the embedding network by using a combination of supervised and self-supervised loss, i.e., predicting image rotations.
Knowledge distillation. Knowledge distillation is machine learning technique that seeks to compress the knowledge contained in a larger model (teacher) into a smaller one (student). It was introduced by Bucilua et al. [4] and later adopted deep learning by Hinton al. [23]. Recently, Tian et al. [53] show that incorporating self-distillation into FSL can boost the performance by 1 2%.
Transductive inference. Transductive inference aims to leverage the query set (which can be seen as unlabeled data during inference) in addition to the labeled support set. Transductive approaches [39, 24, 43, 66, 2, 10] significantly outperform their inductive counterparts, which do not exploit the query set. For example, TPN [39] constructs a graph whose nodes are support and query images and propagates labels from the support query images, while CAN [24] utilizes confidently classified query images as part of the support set. Recently. TIM [2] assumes a uniform distribution of novel classes, which is often the case for the current FSL benchmarks, and exploits that assumption for boosting the performance. Furthermore, [15, 52] can be considered as transductive methods since the information from query data is used for batch normalization.
In this paper, we improve the performance of the classifier on novel classes by introducing a novel loss function, which penalizes out-of-distribution samples. Our method is complementary to the above approaches and can be combined to establish a new state-of-the-art for FSL. As the time of camera-ready: we aware of concurrent work [8] that also leverages the distractor samples to refine the classifier in FSL. Contrast to that work, our loss function is inherently applicable to both also discover the effectiveness of uniform
inductive and transductive inference. Furthermore, random features, which obviates the need for accessible OOD samples for in-domain FSL..
3 Preliminary
We first define some notations used in the following sections. Let (x. y) consist of sample image x and its corresponding class label y. Let D₁ = {(x,, V: No denote the labeled base samples used for feature pre-training. Next, we denote D. = {(x,, y1) } and D, {(x:, m
Nr as the labeled support samples and query samples respectively. Note that the labels for the query samples are only used for evaluation purposes. The support samples and query samples belong to the novel classes Cn- which are separated from the base classes Co. i.e., C C = Ø. In few-shot learning, we aim to learn classifier that exploits support data to predict labels for query samples. We use pre-trained feature extractor, usually kept fixed, to produce input to the classifier.
In this work, we consider fine-tuning the classifier on the support set only (inductive learning) and on both the support and query set (transductive learning). We also consider both in-domain and cross-domate FSL. In the former, the novel classes Cn and the base classes C₁ are from the same domain (e.g. images from Image-Net), while for the latter, C, and C. are from different domains and the domains of C₁ and Cn are referred to as the source and target domains, respectively.
Pretrained feature extractor. Let fo be the feature extractor trained the base data with the standard cross- cross-entropy loss, which we also refer as the "simple baseline". We also seek to strengthen the baseline with orthogonal techniques such as self-supervised learning (SSL) [18,5] and knowledge distillation (KD) [53]. With SSL, we employ the "rot baseline", which is trained with the standard cross-entropy loss and an auxiliary loss to predict the rotation angles of the perturbed images. We further apply the born-again strategy [17] for the rot baseline in two generations to construct the "rot + KD baseline". More details of our baselines can be found in Section A.
Novel tasks inference. In the few-shot scenario, we freeze the feature extractor fo and train a classifier for each task. Let W [Wo. "Wo.""..wa E KXD be the weight matrix of the classifier,
3

---

where D is the dimension of the encoded vector (output by the feature extractor) and K' is the number of classes (i.e., number of ways) for each task. The predictive distribution over classes p(k] W) is given by:
exp(-7 d(z,, Wk))
fo(x.)
pk]],,, W)
Σj exp( d(z;; w;)) where
2,
(I)
[.fo(x))l2
where y is the learnable scaling factor [42] and -(..-) denotes the distance function. We use the squared Euclidean distance in our experiments unless otherwise mentioned. In our implementation, we initialize each weight vector Wa to the mean of the sample features in the support set W& =
[8k] Exes, [x) with Sy being the support set of the tth class similar to Prototypical Network [38].
Inductive bias. Inductive learning a process of learning a general principle by observing specific examples. Given limited support data, it is possible to have multiple explanations on the query data, with each corresponding to a different prediction. Inductive bias allows the learner to systematically favor one explanation over another, rather than favoring a model that overfits to the limited support data, Figure/ shows an ambiguous classification example that can be explained by multiple hypothe- ses. Two decision rules possibly learnt from the support data are: 1) People in suit belong to class I; and 2) Boys belong to class 2.
Both rules are simple (satisfying Occam's razor), and can be used to classify the query sample. However, it is unclear which rule the learner would favor; it is only possible to know after training. Hence, solely learning with support data can be obscure. One way to narrow down the hypothesis space is using counter-examples to assess the inductive validity [29, 9]. In Figure the out-of- distribution (OOD) sample hints that the suit should be considered irrelevant, and hence the rule 1 should be rejected.
4 Proposed approach
We introduce our novel technique for few-shot learning namely Penalizing Out-O--Distribttion sampLEs (POODLE). Specifically, attempt to regularize few-shot learning and improve the general- ization of the learned prototypes by leveraging prior knowledge of in- and out-of-distribution samples. Our definition is as follows. Positive samples are in-distribution samples provided in the context of the current task that includes both support and query samples. Negative samples, in contrast, do not belong the context of the current task, and hence are outtof-distribution. Negative samples can either provide additional cues that reduce ambiguity, or act as distractors to prevent the learner from overfitting. Note that negative samples should have the same domain positive samples so that their cues are insightful to the learner, but positive and negative samples are not required to have the same domain as the base data.
To effectively use positive and negative samples in few-shot learning, the following conditions must be met: 1) The regularization guided by nul-of-distribution samples can be combined with traditional loss functions for classification; 2) The regularization should be applicable for both inductive and transductive inference; and 3) The requirement on negative data should be minimal ie., does not need any sort of labels except for aforementioned conditions.
4.1 Refining prototype with distractor samples
We formulate the regularization as new objective function for training. To capitalize negative samples to reduce the ambiguity, we propose leveraging the large-margin principle as in Large Margin Nearest Neighbors (LMNN) [59, 59]. In these works, Weinberger et al. propose loss function with two competing terms to Iearn a distance metric for nearest neighbor classification: "pull" term to penalize the large distance between the embeddings of two nearby neighbors, which likely belong to the same class, and a "push" term to penalize the small distance between the embeddings of samples of difference classes.
In this work, we seek learn prototypes for all categories instead of a distance metric. We do not have labels (i.e,, categories) of all samples (e.g., transductive inference) but only the prior knowledge about whether a sample is in- or ouf-o--distribution. Thus, we adapt the above objective of margin maximization for these two groups, namely in- and out-of-distributtinn, with the distance function being the sum of distances from sample to all prototypes. The goal minimizing distances from positive samples to prototypes, while maximizing distances from negative samples to prototypes.

---

In our objective function, we keep the original "pull" term while introducing our new "push" term: we do not explicitly enforce large distances between positive samples and negative samples, but only attempt to maximize distances between prototypes and negative samples:
pos
K
Nueg
Lnaive
Y d( x.)
Σ j=1 A=1
d(Wk:
(2)
i=1
Note that is scaling factor of distance- based classifier as in Equation 1. However, this objectíve does not take into account class assignments for positive samples. To tackle this problem, we use weighted distances between prototypes and samples to simultaneously optimize both objectives:
N
Npas
Nneg K
Emargin
T. d(we, (x)Solpp(//p
W)]
>- (.),,x)))llp(xx,, W)] (3)
1=1
k=
Cpuit
Spich
where (t](]] is the stop-gradient operator. Intuitively, the positive sample will "pull" the prototypes to its location proportional to its distance the prototypes. Subsequently, the prototype of each class will move closer to the positive samples of that class. At the same time, the prototype is enforced to move away from the negative samples, thus discarding features that might lead to high similarity to out-of-distribution data.
We note that removing Sal.] in Equation 3 would result in different underlying objective, which empirically leads to a decrease in performance. Particularly, the objective without stop-grand gradient will also be compounded of auxiliary term for entropy maximization (see Section B). Thus, devoid of meticulous regularization would deteriorate the performance. The above observation is in line with Boudiaf el al. [2]. In summary, POODLE optimizes the classifier on novel tasks with the following objectives:
LPOSOLE Lee + a Cpull 3 passh
where and control the "push" and "'uull" coefficients respectively and E denotes the standard cross-entropy loss. To the best of our knowledge, the proposed objective is the first loss function (in test phase) that be effective for both inductive and transductive inference. Please the supplemental document for the pseudo-code (Section C).
4.2 On negative samples
Choice of negative samples. For in-domain FSL, we simply leverage the base data as negative samples. For cross-domain FSL, one approach would be using a set of samples drawn from classes that are disjoint to novel classes C, negative examples. However, in some cases such negative examples might not be available. Thus, we consider another approach where we have a set of unlabeled data of set of classes C, from the target domain (C,, might overlap with C,)) negative examples, similar to [41]. We name the above two approaches as disjoint and noisy negative sampling, respectively in the context of cross-domain FSL.. Intuitively, when the number of categories of unlabeled data (i.e., |Cu) grow larger, the noise of mixing positive and negative samples in noisy negative samples pool will be reduced.
The burden of additional data. It is worth pointing out that POODLE does not break the setup of FSL as it does not require any additional data of novel classes, thus, still has a few samples from novel tasks. As POODLE obligates additional data in the form of OOD samples, one might ask how practical the algorithm is. For in-domain FSL, the negative samples can be casily obtained by adopting the training data as we already know C. n C. = 2. Eyen in extreme case where we only have access to empirically find that (fz-normalized) uniformly random noise can work
the pretrained models, surprisingly well (Section 4.3). For cross-domain FSL, even though POODLE requires additional OOD data, we find that these samples can be: noisy with both positive and negative samples; and 2) fairly efficient- in our experiments, we can "reuse" 400 OOD samples for all tasks, which is very efficient compared other methods that might use up to 20% unlabeled data of training set [41].
4.3 Intriguing effectiveness of uniform random features
One caveat of POODLE is the obligation of accessing to OOD samples. Fortunately, we find that POODLE can use the uniform random features on the hypersphere as negative samples for in-domain
5

---

FSL. Particularly, we can uniformly sample latent vectors from S as an alternative to the distractor samples from base training data, thus, eliminating the need for accessing to training data.
Our use of uniform distribution as negative examples is inspired by that feature uniformity is a desirable property for contrastive loss [57, 5], and so good representation prefers such uniformity. The problem of uniformly distributing points on the unit hypersphere is related to minimizing pairwise loss [31, IJ. which links well to the theory of supervised classification with softmax and cross-entropy (equivalent to minimizing pairwise loss or maximizing mutual information) [3, 45].
Therefore, using uniform distribution negative samples works for in-domain FSL because they will approximate the features of samples from base training data. For cross-domain FSL, this approach will fail because random features (which are similar to samples from the training domain) have large discrepancy with the target domain, thus, not inducing meaningful cues. Intuitively, the more similar between domains of OOD and test samples, the higher performance gain POODLE can achieve. Our experiments in Section 5.4 empirically prove aforementioned postulation.
5 Experiments
In this section, conduct extensive experiments to demonstrate the performance gain of our method on standard inductive, transductive, cross-domain FSL. To demonstrate the robustness of our method across datasets/networkk architectures, we keep the hyperparameters fixed for all experiments.
5.1 Experimental setup
Datasets. We evaluate our approach on three common FSL datasets. The mini- miii-emmgeeet dataset [56] consists of 100 classes chosen from the ImageNet dataset [48] including 64 training, 16 validation, and 20 test classes with 60,000 images of size 84 84. The tiered-lmagenes [46] is another FSL dataset which is also derived from the ImageNet dataset with 351 base, 97 validation, and 160 test classes with 779,165 images of size 84 x 84. Caltech-UCSD Birds (CUB) has 200 classes split into 100, 50, 50 classes for train, validation and test following [6]. Furthermore, we also carry out experiments on iNaturalist 2017 (iNat) [54], ET [22], and ISIC-2018 (ISIC) [7] for the (extreme) cross-domain FSL. The description for these datasets can be found in the supplemental document (Section D.I).
Implementation details. We use ResNet12 as our feature extractor. It is a residual network [61] with 12 layers split into 4 residual blocks. For pre-training on the base classes, we train our backbones with the standard cross-entropy loss for 100 epochs. The optimizer has a weight decay of 54- and the initial learning rate of 0.05 is decreased by factor of 10 after 60, 80 epochs in mini-ImageNNtt and 60, 80, 90 epochs in Sered-ImageNet. We use the batch size of 64 for all the networks. For fine-tuning on the novel classes, we utilize Adam optimizer [32] with fixed learning rate of 0.001, 31 = 0.9, 82 = 0.999, and do not use weight decay. The classifier is trained with 250 iterations. The coefficients of "push/pull" loss are a and 3 = 0.5 respectively. For negative samples, we randomly select K = 400 samples from the out-of -itto--diitribttion samples pool for each novel task. We evaluate the performance of POODLE in 5-way- --shot and 5-way-5-shn settings on 2000 random tasks with 15 queries each.
5.2 Standard FSL
We first experiment with the standard FSL setup (also known as in-domain FSL.) in which the number of query samples is uniformly distributed among classes.
Evaluation with various baselines. Table shows the results of our approach with various baselines, which described in Section 3, with the inductive inference (positive samples are support images). As can be seen, our approach consistently boosts the performance of all baselines by a large margin (1- 3%). Interestingly, the POODLE-R R variant outperforms the CE loss variant significantly and performs comparatively with the POODLE-B variant. We hypothesize that the normalized representations the samples extracted from the base classes are uniformly distributed.
Table demonstrates the efficacy of each loss term of POODLE in transductive inference (positive samples are support and query images). We can that using the "pull" loss with query samples

---

Table 1: Comparison to different baselines for the standard FSL (a.k.a in-domain FSL) mini- ImageNet, sered-Imagent and CUB in the inductive setting with Resnet-12 as backbone. Here, POODLE-B and POODLE-R indicate that the negative samples are sampled from the base classes and the random uniform distribution respectively.
mini-ImageNet 5-shot
fiered-ImageNet
CUB
Baseline
Variant
1-shot
1-shot
5-shot
1-shot
5-shot
CE Loss
61.33
80.76
67.42
84.44
75.60
89.88
Simple
POODLE-B
63.79
81.41
69.26
84.97
75.96
90.06
POODLE-R
64.38
81.35
69.34
85.03
76.22
89.99
CE Loss 64.57
82.89
68.26
85.09
76.39
91.10
Rot
POODLE-B
66.78
83.65
69.74
85.45
77.26
91.30
POODLE-R
67.20
83.54
69.86
85.56
77.35
91.43
CE Loss 65.91
82.95
69.43
84.93
79.70
92.28
Rot KD
POODLE-B 67.50
83.71
70.42
85.26
80.23
92.36
POODLE-R 67.80
83.50
70.47
85.24
80.05
92.37
Table 2: Comparison to different baselines mini-ImageNet, fiered-Imagenet and CUB with and
without the proposed loss the transductive settings. Here, Loaut and L indicate that the negative samples are sampled from the base classes and the random uniform distribution respectively.
mini-ImageNNe 5-shot
tiered-ImageNet
CUB
Baseline
Loss
1-shot
1-shot
5-shot
1-shot
5-shot
CE
61.33
80.76
67.42
84.44
75.60
89.88
CE + Lpail
70.91
83.04
75.58
86.10
84.91
91.60
Simple
CE Lpull Cyush
74.21
83.71
78.72
86.57
87.04
91.84
CF pplli pussh
74.82
83.67
79.24
86.64
87.13
91.60
CE CE Lppll
64.57
82.89
68.26
85.09
76.39
91.10
74.42
85.20
76.59
86.75
85.93
93.02
Rot
CF Lpplt C.
Coushh
77.30
85.91
79.74
87.25
88.68
93.23
CE + Lplli C push
77.69
85,86
80.31
87.33
89.07
93.35
CE
65.91
82.95
69.43
84.93
79.70
92.28
CE + Lpuil
74.90
85.09
77.07
86.50
88.22
93.68
Rot + KD
CF + Lplll Lb
Lpush
77.56
85.81
79.67
86.96
89.88
93.80
CE + Lpull puush
77.24
85,53
79.97
86.91
90.02
93.68
improve the inductive baseline significantly, being as effective as other transductive algorithms. Combining with "push" term, the classifier further enhanced.
We also conduct experiments with additional loss functions including self-supervised loss (SSL) and knowledge distillation (KD) in Figure 2a. Our approach consistently improves the generalization of all baselines without additional computation cost (in the training phase) as justified in Figure 2b. The improvement of POODLE other backbones and datasets is reported in the supplemental document (Section D.2).
Comparison to the state-of-the-arr approaches. We report the performance of our network in comparison with sfate-ofthhe-an art methods in both transductive and inductive settings (with and without information from the query images) in Table 3. We can see that our approach remarkably improves performance the baseline and achieves a comparable performance with the state-of-the-art approaches in the heree--IaaeeNe prior work in both inductive and transductive settings. Experimental results on other backbones are reported in the supplementary document (Section D.2).
In mini-ImageNNe and CUB we significantly outperform the

---

Traiing hime
GPU Menory
Accurcc Gain
Denscnutt WINN-12-8
Resmet-12
MabileNos
Comr-44512
Baseline
Our
KD
SS1.
SSL
KD
Our
Training time (hours)
GPU Memory (Gb)
(b)
Figure 2: (a) Effectiveness of our approach applied to standard FSL. (a) Our approach consistently yields accuracy gains on different backbones, in comparison with those obtained by self-supervised loss (SSL) and knowledge distillation (KD) in 5- 5-way- I-sho protocol setting. (b) Total time and memory cost (in training stage) when adopting SSL, KD, and our method. Note that apply KD for T 2 generation in [53], No large overhead is incurred in method.
Imagenee with inductive
Table 3: Comparison to the state-of the-art art methods on mini- Mini-ImmeeNet, tiered Imaee-etmeeeetttt and CUB using inductive and transductive settings. The results obtained by our models (blue pearl-shaded) are averaged over 2,000 episodes.
mini-ImageNet 5-shot
tiered-ImageNer
CUB
Method
Transd.
Backbone
I-shot
I-shot 5-shot
I-shot
5-shot
MAML [15]
ResNet-18
49.6
65.7
68.4
83.5
RelatNet [24]
ResNet-18
52.5
69.8
68.6
84.0
MatchNet [56]
ResNet-18
52.9
68.9
73.5
84.5
ProtoNet [50]
ResNet-18
54.2
73.4
73.0
86.6
Neg-cosine [37]
X
R
62.3
80.9
72.7
89.4
MetaOpt [33]
ResNet-12
62.6
78.6
66.0
81.6
SimpleShot [58]
ResNet-188
62.9
80.0
68.9
84.6
68.9
84.0
Distill [53]
ResNet- 12
64.8
82.1
71.5
86.0
Rot + KD + POODLE
ResNet-12 12
67.80
83.72
70.42
85.26
80.23
92.36
RelatNet [24]
ResNet- 12
52.4
65.4
TPN [39]
ResNet-12 12
59.5
75.7
TEAM [43]
ResNet-18
60.1
Ent-min [10]
ResNet- 12
62.4
74.5
68.4
83.4
CAN+T [24]
ResNet- 12
67.2
80.6
73.2
84.9
LaplacianShot [66]
ResNet-
72.1
82.3
79.0
86.4
81.0
88.7
TIM-GD [2]
ResNet- 18
73.9
85.0
79.9
88.5
82.2
90.8
Simple POODLE
ResNet-12 12
74.21
83.71
78.72
86.57
87.04
91.84
KD + POODLE
ResNet-12
77.56
85.81
79.67
86.96
89.93
93.78
5.3 Cross-domain FSL
In this section, we conduct experiments to demonstrate the efficacy of our approach even with challenging task: extreme cross-domain FSL first introduced [21, 41]. Many FSL algorithms are known to fail in such a challenging scenario [6].
Recall that have two setups for cross-domain FSL (Section 4.2), in which cases, the negative samples are drawn from disjoint and noisy negative samples pool. Here, we provide the detailed setting of each scheme when transferring network trained on mini- Imagentt to target domains. It is worth mentioning that we only consider inductive inference for comparison with prior work.
Disjoint negative samples. As mentioned before, we evaluate the results of cross-domain FSL on test split of these datasets while employing the train split to draw negative samples. Since the number of categories in EuToSAT and ISIC is relatively small (10 and respectively) compare to the number

---

Table 4: The results of the domain-shift setting from mini-Imagent Imagenet to CUB, iNat, ISIC, and EuroSAT with Rot + KD baseline. The results obtained by our models (blue pearl-shaded) are averaged over 2,000 episodes. The baselines with notation show the results using the setup of disjoint negative samples, and the baselines without the show the results using the setup of noisy negative samples.
CUB
iNat
ISIC
EuroSAT
Baseline
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Transfer [41]
30.71
43.08
60.73
80.30
SimCLR [41]
26.25
36.09
43.52
59.05
Transfer SimCLR [41]
32.63
45.96
57.18
77.61
STARTUP (no SS) [41]
32,24
46,48
62.90
81.81
STARTUP [41]
32.66
47.22
63.88
82.29
Rot + KD + CE
49.98
69.44
48.90
66.58
32.35
43.81
65.00
80.01
Rot + KD + POODLE-R
50.04
69.78
48.94
66.87
32.21
43.87
63.97
79.85
Rot KD + POODLE-B
52.61
70.78
50.62
67.31
33.56
44.17
66.21
80.51
Rot KD + CE
50.28
69.64
48.80
66.72
Rot KD + POODLE-R
50.25
69.97
48.84
67.03
Rot KD + POODLE-B" 53.11
70.96
50.46
67.45
of "way" in novel task (5), we only concern with iNat and CUB. Precisely, we follow [6] and [58] to split CUB and iNat respectively.
Noisy negative samples. We assume that have 20% unlabeled data from the target domain and the rest 80% of data is used for testing. Although large number of classes are beneficial to POODLE diseussed above, also carry out experiments with ISIC and EuroSAT to evaluate the performance of extreme cross-domain FSL with our approach.
From the Table 4 we can see that the improvement of POODLE with the disjoint negative samples (the bottom rows) and the noisy negative samples (the middle rows) when transferring knowledge from mini-Imagelle
to iNat, CUB, ISIC, and EuroSAT. We can observe that despite the extreme gap between target/source domains and a very noisy mix of in- and nnt-of-distribution samples in negative samples pool (in case of EuroSAT and ISIC), POODLE can successfully boost the performance of baselines in all experiments. We also report the performance of other approaches, which have same setup noisy negative samples, in Table 4.
5.4
Ablation study
In this section, we present results of some presentative ablation study to get more insight of POODLE behaviors. For more in-depth experiments, we refer reader supplementary document.
5.4.1 The effect of domain diserepancy between positive-negative samples
Intuitively, the more similar between domains of OOD and test samples, the higher performance gain POODLE can achieve. To understand how our method works when OOD data is from various domains, we train our classifier and compare the performance when using random uniform distribution and other datasets as negative examples. The result of this experiment reported in Table 5. We can observer that the more similar of OOD domain to test domain, the higher the performance gain we can achieve. Thus, we should always aim to leverage the OOD samples test domain.
On the performance of random features: For in do-domain FSL i.e., test domains mini- Imagenet and tiered- dered-Imagennd Imagenet, using uniform examples has the best and second-best accuracy when tested on Mini-ImageNer and MrreedIImgeNee, respectively. As aforesaid, the uniform random features will reflect the distribution of mini- mini-Imagenet samples (which the network is pre-trained on). Thus, random features are effective for in-domain FSL.
For cross cross-domain FSL (i.e., test domains are CUB and EuroSAT), using random uniform features which is approximately equal to using samples from discrepancy between the source and target domain is extremely large, e.g., animal (mini- Imagi-)magennt) vs satellite images (EuroSAT).
does not work because the

---

Table 5: Evaluating (simple) Resnet- 12 trained mini-Imagenet different test/OOD domains in the 1-shot inductive protocol (10,000 episodes). random, mini, tiered, CUB, EuroSAT denote fiered- fiered-Imagenet, CUB, EuroSAT dataset respectively. The OOD/test samples drawn from the standard train/test split of each dataset, respectively. The 95% confidence interval is roughly 0.20 for all results.
OOD domain mini
w/o OOD
random
liered
CUB
EuroSAT
mini
61.63
64.30
64.08
63.72
62.71
61.76
liered
63.04
64.28
64.01
64.50
63.85
62.78
Test domain
CUB 48.55
48.88
49.01
49.10
51.40
49.18
EuroSAT 65.18
63.85
64.58
64.25
64.70
66.04
Table The results the various approaches to learn a better classifier with pretrained Rot + KD buseline mini-lmagenet. The results are obtained by averaging over 10,000 episodes.
Method
1-shot
5-shot
baseline
66.32 0.20
82.99 + 0.13
w/ learning cosine classifier
66.66 = 0.20
82.84 + 0.13
2) w/ naive OOD
66.36 = 0.20
83.08 + 0.13
3) w/ large-margin w/o negative samples
66.32 = 0.20
83,01 + 0.13
4) w/ label smoothing
66.13 = 0.20
80.81 + 0.13
POODLE-B POODLE-R
67.84 0.20
83.72 + 0.13
68. + 0.20
83.60 = 0.13
5.4.2 Comparison against simple approaches for enhance pretrained models
To better understand the effectiveness of our method, we also compare to four methods that learn better classifiers on novel classes with pre-trained features. The results are reported in Table 6.
Learning cosine classifier: The classifier in inference phase fine-tuned using CE loss with the support samples of each task. In our experiments, this approach does not bring any meaningful improvement similar [58].
2. Naive OOD: we fine-tune (k + 1)-way classifier with additional class for OOD samples using CE loss. Naive OOD performs worse because the OOD samples are drawn from several classes of the training set, which are well-clustered and separated, cannot find "prototype" with linear classifier to match all of them.
Large-margin w/o negative samples: only use push term in POODLE. This approach is not better than cosine-distange because the initialized prototype (mean of all samples from specific class) already well-clustered and optimal i.e., close the ground-truth class prototype and far away from others for the observed samples.
Label smoothing: we use label smoothing for CE loss (ground- -rround-truth class has the probability of 0.9 and uniformly distribute 0.1 to the rest. This method does not work well because it only makes learned prototypes not be far away from samples from other classes
Conclusions and future work
In this work, have proposed the concept of leveraging -itto---iitribution samples set to improve the generalization of few-shot learners and realize by a simple yet effective objective function. Our approach consistently boosts the performance of FSL across different backbone networks, inference types (inductive/trassductive)) and the challenging cross-domain FSL.
Future work might seek to exploit different sampling strategies (i.e., how to select negative samples) to further boost the performance and reduce time/memory complexity; another interesting direction is enhancing the robustness of the classifier when we have both positive and negative samples in the same sampling pool; leveraging domain adaptation to reduce the need of in-domain negative samples is also promising research direction.
10

---

References
[I] Sergiy V Borodachov, Douglas P Hardin, and Edward B Saff. Discrete energy on rectifiable sets. Springer, 2019.
[2] Malik Boudiaf, Ziko Imtiaz Masud, Jérôme Rony, José Dolz, Pablo Piantanida, and Is- mail Ben Ayed. Transductive information maximization for few-shot learning. arXiv preprint arXiv:2008. / /297, 2020.
[3] Malik Boudiaf, Jérôme Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In European Conference on Computer Vision, pages 548-564. Springer, 2020.
[4] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535-541, 2006.
[5] Ting Chen and Lala Li. Intriguing properties of contrastive losses. arXiv:2011. 1.02001,.0003,,. 2020.
arXiv preprini
[6] Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Frank Wang, and Jia Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904, 04232, 2019.
[7] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helha, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint 1r0iv: 2019.
[8] Rajshekhar Das, Yu-Xiong Wang, and Jose MF Moura. On the importance of distractors for few-shot classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9030-9040, 2021.
[9] Wim De Neys, Walter Schaeken, and Géry d' d'Ydeweee. Working memory and everyday condi- tional reasoning: Retrieval and inhibition of stored counterexamples. Thinking & Reasoning, 1144)3399-31, 2005.
[10] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv preprint wwxXiv:1999.027220 2019.
[11] Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-award few-shot transfer. arXiv preprint arXiv:2007.11499 1498, 2020.
[12] Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2051 2051-2060, 2017.
[13] Dorothy Edgington. On conditionals. Mind, 104(414):235-329, 1995.
[14] Reuben Feinman and Brenden M. Lake. Learning inductive biases with simple neural networks. Proceedings of the 40th Annual Meeting of the Cognitive Science Society, 2018.
[15] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1133 135. PMLR, 2017.
[16] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pages 1568-1577 PMLR, 2018.
[17] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pages 1607- 1616. PMLR, 2018.

---

[18] Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with supervipevii... International Conference on Computer Vision (ICCV), 2019.
[19] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803.0778, 2018.
[20] Yiluan Guo and Ngai-Man Cheung. Attentive weights generation for few shot learning via information maximization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[21] Yunhui Guo, Noel Codella, Leonid Karlinsky, James Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In European Conference on Computer Vixion, pages 124-141. Springer, 2020.
[22] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226 2019.
[23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini
003.03533,,1222222 2015.
[24] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. arXiv preprint arXiv:1910.07677, 2019.
[25] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint IrXiv::704.08800 2017.
[26] Shell Xu Hu, Pablo G Moreno, Yang Xiao, Xi Shen, Guillaume Obozinski, Neil D Lawrence, and Andreas Damianou. Empirical bayes transductive meta-learning with synthetic gradients. In International Conference on Learning Representations (ICLR), 2020.
[27] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700 7700-4708, 2017.
[28] Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11719-11727, 2019.
[29] Philip N Johnson-Laird. Mental models and human reasoning. Proceedings of the National Academy of Sciences, 107(43): (000):1822388888 2010.
[30] Philip Nicholas Johnson-Laird. Mental models: Towards cognitive science of language, inference, and conscionsness. Number 6. Harvard University Press, 1983.
[31] Olav Kallenberg and Olav Kallenberg. Foundations of modern probability, volume 2. Springer, 1997.
[32] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2014.
[33] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[34] Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference on Machine Learning, pages 2927-2936. PMLR, 2018.
[35] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for seo-shot learning, arXiv preprint
2017.
12

---

[36] Moshe Lichtenstein, Prasanna Sattigeri, Rogerio Feris, Raja Giryes, and Leonid Karlinsky. Tafssl: Task-adaptive feature sub-space learning for few-shot classification. In European Conference Computer Vision, pages 522-532 Springer, 2020.
[37] Bin Liu, Yue Cao. Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In European Conference on Computer Vision (ECCV), 2020.
[38] Jinlu Liu, Liang Song, and Yongqiang Qin. Prototype rectification for few-shot learning. In European Conference on Computer Vision (ECCV), 2020.
[39] Yanbin Liu, Juho Lee, Minseop Park, Sachoon Kim, Eunho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning, arXiv preprint arXiv: 10002, 2018.
[40] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pages 69-84. Springer, 2016.
[41] Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. In International Conference on Learning Representations, 2021.
[42] Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5822-5830, 2018
[43] Limeng Qiao, Yemin Shi, Jia Li, Yaowei Wang. Tiejun Huang, and Yonghong Tian. Trans- ductive episodic-wing wise adaptive metric for few-shot learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 3603-3612, 2019.
[44] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7229-7238 2018.
[45] Zhenyue Qin, Dongwoo Kim, and Tom Gedeon. Rethinking softmax with cross-entropy: Neural network classifier as mutual information estimator. arXiv preprint arXiv:1911.1066 2019.
[46] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenen- baum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. In International Conference Learning Representations (ICLR), 2018.
[47] Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. In International conference on machine learning, pages 2940-29209090 2949. PMLR, 2017.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. In International booktitle of Computer Vision (IJCV), 2015.
[49] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero. and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations (ICLR), 2019.
[50] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems (NeurIPS), 2017.
[51] Long-Chyi Su. Subhransu Maji, and Bharath Hariharan. When does self-supervision improve few-shot learning? In European Conference Computer Vision, pages 645-666. Springer, 2020.
[52] Flood Sung. Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1199-1208, 2018.
13

---

[53] Yonglong Tian, Yue Wang, Dilip Krishman, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding all you need? In European Conference on Computer Vision (ECCV), 2020.
[54] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8769-8778, 2018.
[55] Niki Verschueren, Walter Schaeken, and Gery d'Ydewalle, Everyday conditional reasoning: A working memory-dependdee tradeoff between counterexample and likelihood use. Memory & Cognition, 33(1):107-119) 2005.
[56] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[57] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pages 9929-9939. PMLR, 2020.
[58] Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Sim- pleshot: Revisiting nearest-neighhor classification for few-shot learning. In arXiv preprint arXiv:/9/1.0er 1.0623, 2019.
[59] Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. In Advances in neural information processing systems, pages 1473-1480, 2006.
[60] Davis Wertheimer and Bharath Hariharan. Few-shot learning with localization in realistic set- tings. Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 3558-6567, 2019.
[61] Han-Jit Jia Ye, Hexiang Hu, De-Chusa Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8805 1805-8814. 14. IEEE, 2020.
[62] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Learning embedding adaptation for few-shot learning. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[63] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[64] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifi- cation with diffcrentiable earth mover's distance and structured classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12203-12213, 2020.
[65] Manli Zhang, Jianhong Zhang, Zhiwu Lu, Tao Xiang, Mingyu Ding, and Songfang Huang. lept: Instance-level and episode-leve pretext tasks for few-shot learning. In Proc. Int. Conf. Learn. Represent. pages 1-16, 2021.
[66] Imtiaz Ziko, Jose Dolz, Eric Granger, and Ismail Ben Ayed. Laplacian regularized few-shot learning. In International Conference on Machine Learning, pages 11660-11670. PMLR, 2020.
14


Reviews:
[{"review_id": "piVQaEfxkrZ", "reviewer": "Reviewer_e7JL", "summary": "The authors improve the performance of metric-based few-shot learners by adding a new term based on out-of-distribution samples' to the loss for ProtoNets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is written well. They describe the contributions clearly. However, the idea of adding out-of-distribution samples requires more ablation study from my point of view.\n\nMinor comments:\nLine 26: If X has P X is Q. I do not see how this relates to this work in particular? I understand that you are talking about using features to classify inputs, but this is just a general intuition about how all models in deep learning work. My point is that this sentence does not have to do anything with network architecture or choices in this paper. So it is better to remove this claim from the introduction or clarify more how this sentence is related to your work.\n\nLine 40: What do you exactly mean by little overhead in inference? Are you talking about finding the distance from each class prototype? What if we have around 10000 classes? Is that overhead still negligible?\nRelated work is also well written and covers important related work. In line 74, I liked how the authors clarify transductive and inductive settings in the paper. \n\nWith regards to Formula 2, d(w_k, x_k). Should not these both be the output of the feature extractor? since in line 112 it is mentioned that d denotes the distance function. This literally means you apply a distance function on raw inputs. Please fix this.\n\nQuestions/Suggestions:\nEquation 1 is the same as in the ProtoNets[1] paper. Also, Line 115 inline formula is exactly how ProtoNets perform this (Formula 1 in ProtoNets). So it should be clarified that this is where it is coming from. Right now it seems like that this is a new idea proposed by this paper. There are two differences with the formula in ProtoNets though: \n1- Using \\gamma\n2- Normalizing the prototype vectors\n\nWhat is the value of \\gamma in the experiments? Why is it useful? How does it impact the results? What is the impact of normalizing the prototype vectors? Furthermore, I believe that increasing the number of shots should eventually resolve the ambiguity. I suggest that authors perform an experiment to find what is the number of shots after which their method does not improve the performance. Right now, it seems that both in 1-shot and 5-shot settings, their method outperforms other baselines.\nI increase my score if I see ablation studies for these since this could be really useful for someone who wants to use metric-based methods.\n\nMy main concern with regards to inductive bias and Figure 1: How can we be sure that out-of-distribution examples are really out-of-distribution? What if it was an image of a child with a suit? Are you making sure you do not pick that one? In line 151 it is mentioned that there is prior knowledge whether the data point is in- or out-of-distribution. Is that possible to do this in the real world? Since we do not have labeled data. Furthermore, what if the task was to classify based on clothes instead of person? It does not really solve the ambiguity. In fact, it can even make the classifier perform worse. Why this does not happen?\n\nI think the claim in Line 173 that this is the first loss function for both transductive and inductive inference is wrong. MAML[2] objective is also effective for both. ProtoNets objective is also effective for both cases. You can just decide to use batch norm information from the whole batch or not. Please clarify this or remove this claim. \n\nLine 159 - 161: The objective does not take into account class assignments for positive instances. But in Formula 3, it seems that you take into account the class assignment for both positive and negative instances. Can you clarify this, please?\n\n[1] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning. In Proc. of the 31st Int'l Conf. on Neural Information Processing Systems 2017.\n\n[2] Finn C, Abbeel P, Levine S. Model-agnostic meta-learning for fast adaptation of deep networks. In Int’l Conf. on Machine Learning 2017.\n\n===============Post Rebuttal ==================\n\nI thank the authors for their quick and accurate responses. I really appreciate their hard work in answering the reviewers' concerns. Unfortunately, I am not going to increase my score and I think this work needs to focus on some substantial improvements that I will explain. I just want to clarify that from my point of view, there is nothing wrong with using well-known techniques such as pull and push terms that are commonly used in deep learning. In fact, it is a good idea to use them to solve new problems and the novelty does not depend on using them, but on how they are assembled to solve a new problem. Thus there is much value in this work that studies adding a pull term for out-of-distribution samples. However, I think to make this work stronger the authors need to study the choice of out-of-distribution samples in much more detail. There are also some claims which I think are not correct. For example, the claim in line 173. The authors argue that *For transductive, the pull loss considers samples of both support and query set (minimizing the weighted distance to support and query features).* This seems not correct to me at all especially for metric-based few-shot learning that is based on minimizing the distance between support and query. It is still not a good practice but acceptable by the few-shot learning community if something like batch normalization data leaks from the query set, but using query set and support set to minimize the weighted distance of features is really bad practice. Regardless of this, I think using 400 out-of-distribution samples actually contradicts few-shot learning. My biggest concern is that in practice, how can we choose out-of-distribution samples? For example, if I want to train a classifier on clothes or different cars. If the authors study this part in their paper thoroughly, their work's impact will be much larger and I strongly suggest they focus on this for future versions of this work.\n\n===============Post Rebuttal After Clarification by Authors==================\n\nThanks to the authors, my concern about transductive settings were resolved. After discussion with other reviewers and carefully reading the authors' responses I decided to increase my score to 7. The authors did a great job in their rebuttal and I strongly suggest including all of these studies and comparisons in the final draft and supplemental material. I think after these discussions this paper is a valuable resource for anyone who wants to explore using out-of-distribution samples for metric-based few-shot learning. The studies can provide insights for researchers and practitioners in the future. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors explained the limitation and broader impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "ZnytbwDwlg", "reviewer": "Reviewer_kwqH", "summary": "This paper proposes a simple large-margin-nearest-neighbors-based penalty for few-shot learning. The proposed method (POODLE) learns the inductive bias of deep neural networks for few-shot learning by leveraging out-of distribution data. Specifically, POODLE penalizes the large distance between the prototypes and positive samples and the small distance between the prototypes and negative samples. Experiments demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths\n1. The paper is well written and easy to follow.\n2. The idea of viewing classification problems as conditional reasoning is interesting. \n3. The authors present adequate experiments on several protocols, including standard inductive, transductive, cross-domain FSL, and imbalanced query set.\n\n*Weaknesses\n1. The contribution of this paper is limited. The main contribution is the newly proposed loss function. But the idea of pushing negative samples and pulling positive samples is not rare in metric learning.\n2. Although the illustration in Figure 1 is logically correct, it is still hard for readers to accept the motivation from the following experiments. Is it possible to present more analysis to the effectiveness of the proposed method using more than just classification accuracy?\n3. The authors should show some analysis on selecting different number of negative samples. It is also not fair to use 400 samples for each novel task, which has only 5 (for 5-way-1-shot) or 25 (for 5-way-5-shot) samples.\n4. The comparison in Figure 2 is disadvantageous. In Figure 2(a), the accuracy gain of the proposed method only outperforms SSL in Conv-4-512 backbone. In Figure 2(b), the backbone information is not provided.\n5. The result of using simple baseline is not well enough to outperform the state-of-the-art methods. For example, the result of Simple + POODLE on mini-ImageNet is 63.79 and 81.41 in inductive setting, which is not presented in Table 3.\n\n==============\n\nI appreciate the authors' efforts in their response, and most of my concerns have been solved.\n\nThe newly added experimental results are also convincing for me. In this sense, I would like to increase my rating to 6 and recommend the authors to include some new experimental results in their revision and make a much more clear clarification.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "NkQc0yrnxIL", "reviewer": "Reviewer_kzjj", "summary": "Authors examine the use of out-of-distribution samples for conditioning few-shot classifiers as they adapt to novel classes. Authors fine-tune a classifier layer at test time, encouraging in-distribution samples to lie close to class representations while out-of-distribution samples lie far away. This simple approach achieves consistent accuracy gains in multiple evaluation settings. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\nThe approach and motivation are interesting. The loss itself is simple, intuitive, and fairly versatile. Experiments are extensive and cover a wide range of settings. Authors do not hesitate to re-implement their and other models over many settings (see also supplementary). Ablation in supplementary is detailed, yet straightforward. Paper is well-organized and clearly presented. \n\nWEAKNESSES:\nUnfortunately, I am not convinced that POODLE works as described, and cannot judge its significance or impact. Three reasons:\n\n- Tables 1 and 2 are interesting, but not convincing. Without confidence intervals over the evaluation trials I cannot tell whether the observed improvements are significant. And more fundamentally, while using the simple CELoss model as the baseline works in demonstrating improvement, it does not indicate improvement that is meaningful. The CE baseline is an inductive model for vanilla FSL, but your evaluation settings in Tables 1 and 2 are semi-supervised SSL and transductive SSL respectively. Thus it is impossible to tell if improvement is coming from your novel loss function, or from the fundamental shift from inductive evaluation to a more friendly evaluation setting. Appropriate baselines would be other semi-supervised learning methods and losses (few-shot or otherwise) such as [1] or [2] for table 1, or other transductive methods for table 2 (as in table 4). As is, the improvement shown is deceptively large. \n\n- No analysis or convincing explanation for why this works. All results appear as accuracy scores; it is impossible to tell if POODLE works for the reasons described (suppressing distractor features). There are even some indicators that it doesn’t: the fact that uniform-POODLE is competitive and sometimes superior to base-class-POODLE indicates that the loss function may not be suppressing distractor features at all, but rather acting as a simple regularizer on labels akin to label smoothing.\n\n- Lack of simple ablations. Some design decisions for the POODLE loss go unexplored, with important baseline ablations absent. 1) What is the performance of a model that simply assigns all negative samples to a distractor class? 2) The pushing loss incentivizes negative samples to be equidistant from all class vectors. What is the performance of a model that does this explicitly, by maximizing entropy of class predictions for negative samples? And as per above: how would this differ from a similar level of label smoothing? 3) As mentioned in sec.5.4, weighting the pull loss by the class predictions can lead to problematic behavior where large classes dominate behavior. Why not use the ground-truth class assignment instead of the predicted one?\n\nIn sum, it is not clear to me why and to what degree POODLE actually works. The second point is the most fundamental issue. Additionally:\n \n- Not actually complementary to many FSL methods. Lines 38-39, 84-85 imply that POODLE can be applied to FSL methods broadly. This is not the case: it only applies to methods that fine-tune a classifier layer at test time, which precludes all metric-based approaches to FSL. Thus POODLE is not nearly as versatile as claimed. \n\n- I hesitate to call POODLE SOTA for inductive settings. In a fair comparison without bells or whistles, [3] outperforms simple-POODLE in all settings, and going strictly by the highest published numbers, [3] outperforms the best POODLE on tiered-ImageNet and CUB. Also, DeepEMD is cited as related work but not included in Table 3, though in this case that doesn’t change the results. \n\nThese last two issues are less important and can probably be fixed with small language changes. \n\nSMALL COMMENTS (feel free to ignore in discussion):\n- Missing citation: [4] is a more recent and SOTA example of employing a self-supervised auxiliary loss for FSL\n- Where does the name POODLE come from? I assume it’s “Penalizing Out Of Distribution sampLEs” but you never actually explain it. \n- In sec4 and later, you use “prototype” to refer to the class representation vector in the learned classification layer. I would consider making explicit the fact that you are referring to this learned vector and not the “class prototype” in the sense of prototypical networks (even if yours are at first initialized to the same thing). \n- Line 220 is misleading, if not outright factually wrong: your improvement in table 1 is lower than 1pp in many settings (eg. Rot+KD improves by only .09pp on 5-shot CUB). \n- I suspect you have mislabeled Table 2: L_push should be L_pull and vice versa. As-is, your notation conflicts with eq.4 and text lines 224-227 (and makes results difficult to interpret). \n\n[1] Ren et al. ICLR 2018: Meta-Learning for Semi-Supervised Few-Shot Classification\n\n[2] Saito et al. ICCV 2019: Semi-Supervised Domain Adaptation via Minimax Entropy\n\n[3] Wertheimer, Tang and Hariharan CVPR 2021: Few-Shot Classification with Feature Map Reconstruction Networks \n\n[4] Doersch, Gupta and Zisserman NEURIPS 2020: CrossTransformers: spatially-aware few-shot transfer\n\nPOST-REBUTTAL:\n\nI'd like to echo the first half of reviewer e7JL's post-rebuttal comment: I really appreciate the clear effort authors put in to address our concerns, there's good value in this work, and in my eyes too novelty is sufficient. Regarding my own review: authors mostly addressed my concerns, with the sole exception that we still have no empirical analysis of POODLE behavior beyond accuracy scores, meaning that we still cannot be entirely sure what POODLE is actually doing in practice. However, after additional clarification the method does make intuitive sense so this is not as troubling to me as it was initially.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No negative impacts are discussed. That said, any risks or dangers inherent to this method are shared with few-shot learning more broadly, so a brief mention of this would likely suffice. Also, if POODLE does in fact work by suppressing distractor features, the ability to target certain distractors (i.e. age, race, gender) could be a very large impact, though of course that is outside the scope of this particular paper to examine or demonstrate. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6.5"}, "extra_text": {}}, {"review_id": "N_rLNOdOQA0", "reviewer": "Reviewer_6ffC", "summary": "The paper proposes to learn from out-of-domain examples in order to help few-shot learning (FSL). The idea is to maximize the distance from prototypes to out-of-distribution samples while minimizing that of in-distribution samples. Experiments are done on standard FSL benchmarks.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Easy to read\n\nCons:\n- It is hard to sense the technical novelty of this method. Using this kind of margin distance to learn in-domain and out-of-domain examples is a very known technique. I did not see the novelty in Section 4.1.\n- The experiments are relatively weak and distracting.\n    - The method only compares with very basic methods on standard FSL benchmarks. Not sure how it compares with sota.\n    -  The experiments also show smaller computation cost and better performance on imbalanced setup. I am confused why these \n        experiments are useful to the main contribution of the paper.\n    -  The setup of the key experiments (cross-domain FSL) is not convincing. \n- The paper even did not follow the submission guideline. The manuscript still has a broader impact section which is not needed \n   this year.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Not sensitive area.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Out of the 3 reviewers who spend the most time on this paper, they all thought that the submission was above the acceptance threshold. Reviewers and I especially appreciated the authors' active engagement and helpful responses. 

The idea of using unlabeled out of domain data to improve few-shot learning should be generally useful, and the authors show good empirical performance. As the reviewers pointed out, the presentation is clear. For these reasons, I recommend the acceptance of this paper. 

Decision: Accept (Poster)