Rectifying the Shortcut Learning of Background for Few-Shot Learning
Authors: Xu Luo
Longhui Wei
Liangjian Wen
Jinrong Yang
Lingxi Xie
Zenglin Xu
Qi Tian
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-Shot Learning
Foreground and Background
Shortcut Learning
Contrastive Learning

Abstract: The category gap between training and evaluation has been characterised as one of the main obstacles to the success of Few-Shot Learning (FSL). In this paper, we for the first time empirically identify image background, common in realistic images, as a shortcut knowledge helpful for in-class classification but ungeneralizable beyond training categories in FSL. A novel framework, COSOC, is designed to tackle this problem by extracting foreground objects in images at both training and evaluation without any extra supervision. Extensive experiments carried on inductive FSL tasks demonstrate the effectiveness of our approaches. 

Rectifying the Shorteut Learning of Background for Few-Shot Learning
Xu Luo', Longhui Wei", Liangjinn Wen', Jinrong Yang', Lingxi Xie
Zenglin Xu , Qi Tian
University of Electronic Science and Technology of China University of Science and Technology of China Tsinghua University "Huzhhong University of Science and Technology SXidian University 6 Harbin Institute of Technology Shenzhen "Pnggcheng Laboratory
Frank. Luozěoutlook com, {wei 1h2568, zenglin manglinn@gmaill com
Abstract
The category gap between training and evaluation has been characterised one of the main obstacles to the success Few-Shot Learning (FSI.). In this paper, we for the first time empirically identify image background, common in realistic images, as a shortcut knowledge helpful for in-class classification but ungeneralizable beyond training categories in FSL. A novel framework, COSOC, is designed to tackle this problem by extracting foreground objects in images at both training and evaluation without any extra supervision. Extensive experiments carried on inductive FSL tasks demonstrate the effectiveness of our approaches.
Introduction
Through observing a few samples at a glance, humans can accurately identify brand-new objects. This advantage comes from years experiences accumulated by the human vision system. Inspired by such learning capabilities, Few-Shot Learning (FSL) is developed to tackle the problem of learning from limited data [24, 53]. At training, FSL models absorb knowledge from a large-scale dataset; later at evaluation, the learned knowledge is leveraged to solve a series of downstream classification tasks, each of which contains very few support (training) images from brand-new categories.
The category gap between training and evaluation has been considered as one of the core issues in FSL [10. Intuitively, the prior knowledge of old categories learned at training may not be applicable novel ones. [62] consider solving this problem from a causal perspective. Their backdoor adjustment method, however, adjusts the prior knowledge black-box manner and cannot tell which specific prior knowledge is harmful and should be suppressed.
In this paper, we identify image background as specific harmful source knowledge for FSL. Empirical studies in [56] suggest that there exists spurious correlations between background and category of images (e.g., birds usually stand on branches, and shells often lie the beaches; see Fig. 1), which serves as a shortcut knowledge for modern CNN-based vision systems to learn. further revealed that background knowledge has positive impact on the performance of in-class classification tasks. As illustrated in the simple example of Fig. 1, images from the same category are more likely to share similar background, making it possible for background knowledge to generalize from training to testing in common classification tasks. For FSL, however, the category gap produces brand-me new foreground, background and their combinations evaluation. The correlations learned at training thus may not be able to generalize and would probably mislead the predictions. We take
*Corresponding author Code: https:// //ggth/btbttt//// httpp///erraaaaao////ssoooooaaaa
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

w/o calegory
Traditional ovaluation task
Support
Query set
birds with
shells
guitor Ci
classified as drum
Training images
w/ category gap
drum
classified as guitar
Pew-Shot.co evaluation task
Figure 1: An illustrative example that demonstrates why background information is useful for regular classification but harmful for few-shot learning.
empirical investigations on the role of image foreground and background in FSL, revealing how image background drastically affects the learning and evaluation of FSL in negative way.
Since the background is harmful, it would be good we could force the model to concentrate on foreground objects at both training and evaluation, but this is not casy since we do not have any prior knowledge of the entity and position of the foreground objects images. When humans are going to recognize foreground objects of images from the same class, they usually look for shared local pattern that appears the majority of images, and recognize patches with this pattern as foreground. design novel framework, COSOC, to extract foreground of images for both training and evaluation of FSL by seeking shared patterns among images. The approach does not depend on any additional fine-grained supervisions such as bounding boxes pixel-level labelings.
This inspires
The procedure of foreground extraction of images the training set is implemented before training. The corresponding algorithm, named Clustering-baseee Object Sceker (COS), first pre-trains a feature extractor on the training set using contrative learning, which has an outstanding performance, shown empirically in later section, on the task of discriminating between ground-truth foreground objects. The feature extractor then maps random crops of images-candiinee candidates of foreground objects- into a well-shaped feature space. This is followed by runing clustering algorithm on all of the features of the same class, imitating the procedure of seeking shared local patterns inspired by human behavior. Each cropped patch is then assigned foreground score according its distance the nearest cluster centroid, for determining a sampling probability of that patch in the later formal training of FSL develop Shared Object Concentrator (SOC), iterative feature matching within the support set, looking for one crop per image at one time that is most likely be foreground. The sorted averaging features of obtained crops are further leveraged match crops of query images so that foreground crops have higher matching scores, A weighted sum of matching scores are finally calculated classification logits of each query sample. Compared to other potential foreground extracting algorithms such as caliency-based methods, our COS and SOC algorithms have additional capability of capturing shared, inter-image information, performing better in complicated, multi mullii-oject scenery. Our methods also have flexibility of dynamically assigning beliefs (probabilities) to all candidate foreground objects, relieving the risk of overconfidence.
models. For evaluation,
algorithm that applies
Our contributions can be summarized as follows. i) By conducting empirical studies on the role of image foreground and background in FSL, we reveal that image background serves as source of shortcut knowledge which harms the evaluation performance. ii) To solve this problem, we propose COSOC, framework combining COS and SOC, which can draw the model's ' attention to image foreground at both training and evaluation. iii) Extensive experiments for non-trone con-trandductve FSL tasks demonstrate the effectiveness our method.
2 Related Works
Few-shot Image Classification. Plenty of previous work tackled few-shot learning in meta-learning framework [18, 50), where model learns experience about how to solve few-shot learning tasks by tackling pseudo few-shot classification tasks constructed from the training set. Existing methods that ultilize meta-learning can be generally divided into three groups: (1) Optimization-basee methods learn the experience of how to optimize the model given few training samples. This kind of methods

---

either meta-learn good model initialization point [12, 45, 39, 70, 20] or the whole optimization
process [40, 58, 34, 27] or both [3, 36]. (2) Hallucination-baat based methods [16, 54, 46, 67, 25, 26, 37] learn to augment similar support samples in few-shot tasks, thus can greatly alleviate the low-shot problem. (3) Metric-based methods [53, 48, 49, 61, 59] learn to map images into metric feature space and classify query images by computing feature distances to support images. Among them, several recent works [19, 63, 57, 10] intended to seek correspondence between images either by attention or meta-filter, in order to obtain more reasonable similarity measure. Our SOC algorithm in one-shot setting is in spirit similar to these methods, in that we both apply pair-wise feature alignment between support and query images, implicitly removing backgrounds that are more likely to be dissimilar across images. SOC differs in multi-shot setting, where potentially useful shared inter-image information in support set exists and can be captured by our SOC algorithm.
The Influence of Background. body of prior work studied the impact of image background on learning- basedd vision systems from different perspectives. [52] showed initial evidence of the existence of background correlations and how influences the predictions of vision models. [64, 43] analyzed background dependence for object detection. Another relevant work [4] utilized camera traps for investigating how performance drops when adapting classifiers to unseen cameras with novel backgrounds. They explore the effect of class-independent background (i.e., background changes from training to testing while categories remain the same) on classification performance. Although the problem is also concerned with image background, shortcut learning of background exists under this setting. This is because under each training camera trap, the classifier must distinguish different categories with background fixed, causing the background knowledge being not useful for predictions of training images. Instead, the learning signal during training pushes the classifier towards ignoring each specific background. The difficulties under this setting lie in the domain shift challenge- the classifier is confident to handle previously existing backgrounds, but lost in novel backgrounds. More recently, [56] systematically explore the role of image background in modern deep-learning-batte vision systems through well-designed experiments. The results give clear evidence on the existence of background correlations and identify it as a positive shortcut knowledge for models to learn. Our results, on the contrary. identify background correlations as a negative knowledge in the context of few-shot learning.
Contrastive Learning. Recent success on contrastive learning of visual representations has greatly promoted the development of unsupervised learning [6, 17, 15, 5]. The promising performance of contrastive learning relies on the instance-level discrimination loss which maximizes agreement between transformed views of the same image and minimizes agreement between transformed views of different images. Recently there have been some attempts [30, 13, 10, 33, 35, 31] at integrating contrastive learning into the framework of FSL. Although achieving good results, these work struggle to have an in-depth understanding of why contrastive learning has positive effects on FSL. Our work takes step forward, revealing the advantages of contrastive learning over supervised FSL models in identifying core objects of images.
3 Empirical Investigation
Few-shot learning consists of a training set Dg and an evaluation set D, which share no overlapping classes. DB contains a large amount of labeled data and is usually used at first to train a backbone network fo(-). After training, a set of N-way K-shot classification tasks T= = {(S,, )r are constructed, each by first sampling N classes in D. and then sampling K and M images from each class to constitute S, and 2,, respectively. In each task т, given the learned backbone f.))) and a small support set S, = {(*%_, )k.)) K.N consisting of K images TT.n and corresponding labels yf.n from each of N classes, few-shot classification algorithm is )
Problem Definition.
designed to classify MN images from the query set Q
Preparation. To investigate the role of background and foreground in FSL, we need ground-truth image foreground for comparison. However, it is time- cone-consuming to label the whole dataset. Thus we select only a subset Dnew = (Dn, D.) of minilmageNee [53] and crop each image manually according to the largest rectangular bounding box that contains the foreground object. We denote the uncropped version of the subset as (Dn-Ori, DriO and the cropped foreground version as D.. Two well-known FSL baselines are selected in our empirical studies: Cosine
3

---

OriOri
85
675
Dg-PG
Exemplar
84
650
Dy-Fuse
Accmmey(5)
62.5
82
825
D,On
D.FFG
D,On
D.FG
D,-Oni
DG-FG
PN
CC
(a)
(b)
Figure 2: 5-way 5-shot FSL performance on different variants of training and evaluation datasets detailed in Sec. 3. (a) Empirical exploration of image foreground and background in FSL using two models: PN and CC. (b) Comparison between CC and Exemplar trained on the full training set of minilmageNer and evaluated Dr-Ori and D.-FG.
Classifier (CC) [14] and Prototypical Networks (PN) [48]. See Appendix A for details of constructing D Dew and formal introdcution of CC and PN.
3.1 The Role of Foreground and Background Few-Shot Image Classification
Fig. 2(a) shows the average of 5-way 5-shot classification accuracy obtained by training CC and PN on Dr-Ori and DFGFG, and evaluating on Ori and D.G respectively. See Appendix F for additional 5-way 1-shot experiments.
Category gap disables generalization of background knowledge. can first noticed that, under any condition, the performance is consistently and significantly improved if background is removed at the evaluation stage (switch from D,-Ori to D,GFG). The result implics that background at the evaluation stage in FSL is harmful. This is the opposite of that reported in [56] which shows background helps improve on the performance traditional classification task, where no category gap exists between training and evaluation. Thus can infer that the class/distribution gap in FSL disables generalization of background knowledge and degrades performance.
Removing background at training prevents shortcut learning. When only foreground is given evaluation (D,,FG),, the models trained with only foreground (Dn-FG) perform much better than those trained with original images (Dn-Ori). This indicates that models trained with original images may not pay enough attention to the foreground object that really matters for classification. Background information at training serves shortcut for models to learn and cannot generalize to brand-new classes. In contrast, models trained with only foreground "learn to compare" different objects-a desirable ability for reliable generalization to downstream few-shot learning tasks with out-of-domain classes.
Training with background helps models to handle complex scenes. When evaluating on Dr-Ori, the models trained with original dataset O-rrri are slightly better than those with foreground dataset DF- We attribute this to sort of domain shift: models trained with DF-FG never meet images with complex background and do not know how handle it. In Appendix D. we further verify the assertion by showing evaluation accuracy of each class under the above two training situations. Note that since we apply random crop augmentation at training, domain shift does not exist the models instead trained Dg-Ori and evaluated on D.FFG.
Simple fusion sampling combines ad advantages of both sides. One may wish to cut off shortcut learning of background while maintaining adaptability of model to complex scenes. A simple solution may be fusion sampling: given an image input, choose its foreground version with probability and its original version with probability 1 p. We simply set p equal to 0.5. We denote the dataset using this sampling strategy as Fuse. As observed in Fig. 2(a), models trained this way indeed combine advantages of both sides: achieving relatively good performance on both Dr-Ori and D.G In Appendix C, we compare the training curves of PN trained on three versions of datasets to further investigate the effectiveness of fusion sampling.

---

The above analysis provides new inspiration for how to improve FSL further: (1) Fusion sampling of foreground and original images could be applied to training. (2) Since background information disturbs evaluation, it is needed to focus on foreground objects assign image patches, that are more likely to be foreground, a larger weight for classification. Therefore, a foreground object identification mechanism is required at both training (for fusion sampling) and evaluation.
3.2 Contrastive Learning is Good at Identifying Objects
In this subsection, we reveal the potential of contrastive learning in identifying foreground objects, which we will use later for foreground extraction. Given one transformed view of one image, contrastive learning tends to distinguish another transformed view of that same image from thousands of views of other images. A more detailed introduction of contrastive learning is given in Appendix B. The two augmented views of the same image always cover the same object, but probably with different parts, sizes and color. To discriminate two augmented patches from thousands of other image patches, the model has to learn to identify the key discriminative information of the object under varying environment. In this manner, semantic relations among crops of images are explicitly modeled, thereby clustering semantically similar contents automatically. The features of different images are pushed away, while those of similar objects in different images are pulled closer. Thus it reasonable speculate that contrastive learning may enable models with better identification of centered foreground object.
To verify this, train CC and contrastive learning models on the whole training set of minilmageNet (Dy-Full) and compare their accuracy on D.,-Ori and D D...FG. The contrastive learning method we use is Exemplar [68], modified version of MoCo [17]. Fig 2(b) shows that, while the evaluation accuracy of Exemplar on Dr-Ori is slightly worse than that of CC, Exemplar performs much better when only foreground of images are given evaluation, affirming that contrastive learning indeed has a better discriminative ability of single centered object. In Appendix D.2, we provide a more in-depth analysis of why contrastive learning has such properties and infer that the shape bias and viewpoint invariance may play an important role.
4 Rectifying the Shorteut Learning of Background
Given the analysis in the previous section, we wish to focus more on image foreground both at training and evaluation. Inspired by how humans recognise foreground objects, we propose COSOC, a framework ultilizing contrastive learning draw the model's attention the foreground objects of images.
4.1 Clustering-based Object Seeker (COS) with Fusion Sampling for Training
Since contrastive learning is good at discriminating foreground objects, we utilize it to extract foreground objects before training. The first step is to pre-train a backbone fo(.) on the training set DB using Exemplar [68]. Then a clustering- based algorithm is used to extract "objects" identified by the pre-trained model. The basic idea is that features of foreground objects in images within one class extracted by contrastive learning models are similar, thereby can be identified via a clustering algorithm; see a simple example in Fig. 3. All images within the i-th class DB form a set (x) 1. For clarity, we omit the class index in the following descriptions. The scheme of seeking foreground objects in one class is detailed as follows:
===
1) For each image X,, we randomly crop it L times to obtain L image patches {pnmm 1 Each image patch Pr,m is then passed through the pre-trained model fo and we get normalized feature fo(pa.m))
vector Vn,m TTpp. E Rd.
2) We run clustering algorithm A on all features vectors of the class and obtain H clusters where z, is the feature centroid of the j-th cluster.
{z, 1 A., ,.,h..))
3) We say an image x, E 2,. if there exists l E [T.] s.t. Vn,k E Zj. where [T.] = (1,2, L). Let
1(2,) = ( xxxx be the proportion of images the class that belong to zj- If (f(()) is small, then
N
the cluster zj is not representative for the whole class and is possibly background. Thus we remove
5

---

3 Forgavead Claner
0.7
Forerenddnpp
Bedkgeraad
CNN
0.04
feature space
01105
Figure 3: Simplified schematic illustration of COS algorithm. We show how we obtain foreground objects from three exemplified images. The value under each crop denotes its foreground score.
all the clusters z with l(z) < y, where y is threshold that controls the generality of clusters. The remaining h clusters {z, )= represent "objects" of the class that we are looking for.
minje[M] ||v,, Zoo 1l2/n. where max min VVa. Za, lla used normalize the score into [0, 1]. Then top-k
4) The foreground score of image patch Pn,m defined as Sn,m
n.m jE(he]
scores of each image x,, are obtained as {5n,m 1 = Topk(sn,m). The corresponding patches mE(L)
1. m=8, are seen as possible crops of the foreground object in image X,. and the foreground (P scores (8 |m= as the confidence. We then use it as prior knowledge to rectify the shortcut learning of background for FSL models.
The training strategy resembles fusion sampling introduced before. For an image x,, the probability that we choose the original version is max Sn,8,, and the probability of choosing Pn.š, from
=ε|K]
top-k patches is (8n,03/ ΣiE[K] Sn.0,) max S. Then adjust the chosen image patch and make *E(k]
sure that the least area proportion to the original image keeps as a constant. We use this strategy to train a backbone f-(-) using FSL algorithm.
4.2 Few-shot Evaluation with Shared Object Concentrator (SOC)
As discussed before, if the foreground crop of the image is used at evaluation, the performance of FSL model will be boosted by large margin, serving To approach this upper bound, we propose SOC algorithm to capture foreground objects by seeking shared contents among support images of the same class and query images.
upper bound of the model performance.
Step Shared Content Searching within Each Class. For each image X within one class c from support set S,, we randomly crop V times and obtain corresponding candidates (Pk.n bxx...=l.... Each patch Pk. is individu- ally sent to the learned backbone fo to obtain normalized feature vector Vk.n- Thus we have totally K x V feature vectors within class Our goal to obtain feature vector W1 that
Figure 4:
The overall pipeline of step in SOC.
Points in one color represent features of crops from image. The red points w1, W2 and W3. contains maximal shared information of all images in class Ideally, w1 represents the centroid of the most similar image patches, each from one image, which can be formulated as
W1 K
Vk.oope(
Aopt arg XE[K][V]
cos(V:,M() Vg.Ag),.
(2)
ISKISKK
where cos(:,) denotes cosine similarity and [K]IM denotes the set of functions that take [K] as domain and [V] range. While Aope be obtained by enumerating all possible combinations of image patches, the computation complexity of this brute-force method is O(VK), which is

---

computation prohibitive when V or K is large. Thus when the computation is not affordable, we turn to use a simplified method that leverages iterative optimization. Instead of seeking for the closest image patches, we directly optimize wi so that the sum of minimum distance to patches of each image is minimized, i.e.,
W1 arg max max(cos(w, Vk,n)];
(3)
WERd
which can be achieved by iterative optimization algorithms. We apply SGD our experiments. After optimization, we remove the patch of each image that is most similar to w1, and obtain K x (V-1) 1) feature vectors, Then we repeatedly implement the above optimization process until no features are left, as shown in Fig. 4. We eventually obtain V sorted feature vectors (w, J which we use to represent the class c. As for the case where shot K 1, there is no shared inter-image information inside class, similar to the handling in PN and DeepEMD [63], we just skip step 1 and use the original V feature vectors.
Step 2: Feature Matching for Concentrating on Foreground Object of Query Images. Once the foreground class representations are identified, the next step is to use them to implicitly concentrate on foreground of query images by feature matching. For each IV randomly crop it for V times and obtain V candidate features {pr In- For each class c, we have V sorted representative feature vectors {wn, J obtained in step 1. We then match the most similar patches between query features and class features, i.e.,
x in the query set Q., we also
$1
max co--ccos(pp: w,))) 15335V
(4)
where a S. is an importance factor. Thus the weight a decreases exponentially index n 1, indicating decreased belief of each vector representing foreground. Similarly, the two matched features are removed and the above process repeats until no features left. Finally, the score of x w.r.t. class c is obtained as a weighted sum of all similarities, i.e., S. = =- gn- where 3 S is another importance factor controlling the belief of each crop being foreground objects. In this way, features matched earlier-thus more likely to be foreground-a will have higher contributions to the score. The predicted class of is the one with the highest score.
Experiments
5.1 Experiment Setup
Dataset. We adopt two benchmark datasets which are the most representative in few-shot learning.
The first is minilmageNet [53], small subset of ILSVRC-12 12 [44] that contains 600 images within each of the 100 categories. The categories are split into 64, 16, 20 classes for training. validation and evaluation, respectively. The second dataset, tieredImageNett [41], is much larger subset of ILSVRC- 12 and is more challenging. It is constructed by choosing 34 super-classes with 608 categorics. The super-classes are split into 20, 6, 8 super-classes which ensures separation between training and evaluation categories. The final dataset contains 351, 97, 160 classes for training, validation and evaluation, respectively. On both datasets, the input image size is 84 x 84 for fair comparison.
Evaluation Protocols. We follow the 5-way 5-shot (1-shot) FSL evaluation setting. Specifically, 2000 tasks, each contains 15 testing images and 5 (1) training images per class, are randomly sampled from the evaluation set D, and the average classification accuracy is computed. This is repeated times and the mean of the average accuracy with 95% confidence intervals is reported.
Implementation Details. The backbone we use throughout the article ResNet-12, which is widely used in few-shot learning. We use Pytorch [38] to implement all our experiments on two NVIDIA 1080Ti GPUs We train the model using SGD with cosine learning rate schedule without restart to reduce the number of byperparameters (Which epochs to decay the learning rate). The initial learning rate for training Exemplar is 0.1, and for CC 0.005. The batch size for Exemplar, CC are 256 and 128, respectively. For minilmageNet, we train Exemplar for 150k iterations, and train CC for 6k iterations. For lieredlmageNet, we train Exemplar for approximately 900k iterations, and train CC for 120k iterations. We choose k-means [32] as the clustering algorithm for COS. The threshold is set to 0.5, and top 3 out of 30 features are chosen per image at the training stage. At the evaluation stage, crop each image 7 times. The importance factors a and 3 are both set to 0.8.
7

---

Table 1: Ablative study on miniImageNet. All models are trained on the full training set of minilmageNNe. Since the aim of SOC algorithm is to find foreground objects, it is unnecessary to evaluate SOC on the foreground dataset D,.FG. FT means finetuning from Exemplar used in COS.
D,-Ori
D.-FG
CC FT COS
SOC
1-shot
5-shot
1-shot
5-shot
62.67 1 0.32
80.22 1 0.24
66.69
0.32
82.86 1 0.19
64.76 0.13
81.18 = 0.21
71.13
0.36
86.21 0.15
65,05 =
81.16 = 0.17
71.36 + 0.30
86.20 # 0.14
64.41 0:22
81.54
= 0.28
69.29 + 0.12
84.94 + 0.28
Table 2: Comparisons with baselines of foreground extractors using saliency detection algo-
rithms on minilmageNet. For fair comparison, all models in the right column at evaluation use multi-cropping. GT means evaluating with ground truth foreground.
Used for training
Used for evaluation
Method
1-shot
5-shot
Method
1-shot
5-shot
CC
62.67 0.32
80.22 0.24
COS
67.23 0.35
82.79 0.31
CC+RBD
63.24 004
80.45 = 0.37
COS+RBD
67.03 + 0.52
82.57 + 0.27
CC+MBD
61.50 0.3
79.12 0.32
COS+MBD
62.98 0.45
79.56 + 0.38
CC+FT
62.71 0.11
80.06 = 0.08
COS+FT
64.74 + 0.28
80.74 + 0.13
CC+COS
64.76 0.13
81.18
COSOC
69.28 0.49
85.16 + 0.42
COS+GT
72.71 + 0.57
87.43 + 0.36
5.2 Model Analysis
In this subsection, we show the effectiveness each component of our method. Tab. 1 shows the ablation study conducted on minilmageNet.
On the effect of finetuning. Since feature extractor is pre-trained using contrastive learning in COS, it may help accelerate convergence if we directly finetune from the pre-trained model instead of training from scratch. As shown in line 2-3 in Tab. 1, fintuning gives no improvement on the performance over training from scratch. Thus we adopt finetuning mainly for speeding up convergence (5 x faster).
Effectiveness of COS Algorithm. As observed in Tab. 1. When COS is applied on CC, the performance is improved on both versions of datasets. In Fig. 5, we show the curves of training and validation error of CC during train- ing with and without COS. Both models are trained from scratch and validated on the full minilmageNet. We observe that CC sinks into overfitting: the training accuracy drops to zero, and validation accuracy stops improving before the end of the training. Meanwhile, the COS algorithm helps slow down convergence and prevent training accuracy from reaching zero. This makes validation accuracy comparable at first but higher at the end. Our COS algorithm weakens the "background shortcut" for learning, draws model's attention on foreground objects, and improves upon generalization.
0
Cr
0L-008
CL-006
spnchs
Figure 5: Comparison of training and validation curves between CC with and without COS.
Effectiveness of SOC Algorithm. The result in Tab. 1 shows that the SOC algorithm is the key to maximally exploit the potential of good bbject-discrimination ability. The performance even approaches the upper bound performance obtained by evaluating the model on the ground-truth foreground D,G. One potential unfairness in our SOC algorithm may lie in the use of multi- cropping, which could possibly lead to performance improvement for other approaches as well. We ablate this concern in Appendix G, as well in the comparisons to other methods the later subsections.

---

Table Comparisons with state-of-the-art models on minilmage and mageed The average inductive 5-way few-shot classification accuracies with 95 confidence interval are reported. * indicates methods evaluated using multi-cropping.
minilmage mageNNt
riered
Model
backbone
1-shot
5-shot
1-shot
5-shot
MetaOptNet [22]
ResNet-112
62.64 1: 0.82
78.63 1: 0.46
65.99 1 0.72
81,56 1: 0.53
DC [28]
ResNet-112
62.53 1: 0.19
79.77
0.19
CTM [25]
ResNet-118
64.12
80.51
0.13
68,41 =
84.28 1.73
CAM [19]
ResNet-11
63.85 + 0:48
79.44 + 0.34
69.89 + 0.51
84.23 0.37
AFHN [26]
ResNet-188
62.38 + 0.72
78.16 + 0.56
DSN [47]
ResNet-12
62.64 = 0.66
78.83 0.45
66.22 + 0.75
82.79 + 0.48
AM3+TRAML [23]
ResNet-112
67.10 : 0.52
79.54 + 0.60
Net-Cosine [29]
ResNett-12
63.85 0.11
81.57 0.56
0.56
CA [2]
WRN-28-10
65,92 1: 0.60
82.85 0.55
74.40 + 0.68
86.61 1: 0.59
MABAS [21]
ResNet-112
65.08 1 0.86
82.70 1: 0.54
ConsNet [59]
Re
64.89 1 0.23
79.95 0.17
IEPT [66]
Re
67.05 0.44
62.90
72.24 = 0.50
86.73 0.34
MELR [11]
1e
67.40 + 0.43
83.40 0.28
72.14 = 0.51
87.01 0.35
IER-Distill [42]
ResNet-11
67.28 # 0.80
84.78 0.52
72.21 = 0.90
87,08 0.58
LDAMF [57]
1esNet-11
67.76 = 0.46
82.71 0.31
71.89 = 0.52
85,96 0.35
FRN [55]
ResNet-112
66.45 +: 0.19
82.83 0.13
72.06 + 0.22
86.89 1 0.14
Baseline* [7]
ResNet-12
63.83 +:
81.38 0.41
DeepEMD* [63]
ResNet-12
67.63 : 0.46
83.47 0.61
74.29 + 0.32
86.98 + 0.60
RFS-Distill [S1]
ResNet-112
65.02 + 0.44
82.04 0.38
71.52 + 0.69
86.03 +: 0.49
FEAT* [60]
ResNet-12
68.03 0.38
82.99 0.31
Meta-baseline [8]
ResNet-12
65.31 0.51
81.26 0.23
68.62
83.74 0.18
COSOC (ours)
12
69.28 0.49
85.16 0.42
73.57 + 0.43
87.57 0.10
Note that if we apply only the SOC algorithm on CC, the performance degrades. This indicates that COS and SOC are both necessary: COS provides the discrimination ability of foreground objects and SOC leverages it to maximally boost the performance.
5.3 Comparison to Saliency-based Foreground Extractors
There could be other possible ways of extracting foreground objects. A simple yet possibly strong baseline could be running saliency detection to extract the most salient region in an image, followed by cropping obtain patches without background. We consider comparing with three classical unsupervised saliency methods- RBD [69], FT [1] and MBD [65]. The cropping threshold is specially tuned. For training, fusion sampling with probability 0.5 is used for unsupervised saliency methods. For evaluation, We replace the original images with crops obtained by unsupervised saliency methods directly for classification. Tab. displays the comparisons of performance using different foreground extraction methods applied at training or evaluation. For fair comparison, all methods are trained from scratch, and all compared baselines are evaluated with coopiig (i.e. using the average of features obtained from multiple crops for classification)and tested on the same backbone (COS trained).
The results show that: (1) Our method performs consistently much better than the listed unsupervised saliency methods. (2) The performance of different unsupervised saliency methods varies. While RBD gives a small improvement, MBI) and FT have negative effect on the performance. The performance severely depends on the efffcttivnness of unsupervised saliency methods, and is very sensitive the cropping threshold. Intuitively speaking, saliency detection methods focus on noticeable objects in the image, and might fail when there is another irrelevant salient object in the image (e.g., man is walking a dog. Dog is the label, but the man is of high saliency). On the contrary, our method focuses on shared objects across images in the same class, thereby avoiding this problem. addition, our COS algorithm has the ability to dynamically assign foreground scores to different patches, which reduces the risk of overconfidence. One of our main contributions is paving new way towards improving FSL by rectifying shorteut learning of background, which can be implemented using any effective methods. Given the upper bound with ground truth foreground, we believe there room to improve and there can be other more effective approaches in the future.

---

Origina
image
Found
abjoct
bird
unicycle
dog.
Saluki
Brying pan
Figure 6: Examples of objects obtained with COS from the training set of manelmageNet. The first row shows the original images;the second row shows the picked patch with the highest foreground score.
Original images
Found
object
dalmatian
guitar
Visualization examples of the SOC algorithm. The first row displays 5 images that belong to dalmatian and guitar classes respectively from evaluation set of minilmageNet. The second row shows image patches that are picked up from the first round of SOC algorithm. Our method succesfully puts focus on the shared contents/foreground.
Figure 7:
5.4 Comparison to State-of-the-Arts
Tab. presents 5-way 1-shot and 5-shot classification results minilmageNNet and lieredImageNet. compare with state-of-the-art few-shot learning methods. fair comparison, we reimplement some methods, and evaluate them with multi-cropping. See Appendix G for a detailed study on the influence of multi-cropping. Our method achieves state-of-the-ar performance under all settings except for 1-shot task tieredImageNet, on which the performance of our method is slightly worse than CA, which uses WRN-28- 10, a deeper backbone, as the feature extractor.
5.5 Visualization
Fig. 6 and 7 display visualization examples of the COS and SOC algorithms. See more examples in Appendix Thanks the well-designed mechanism capturing shared inter-image information, the COS and SOC algorithms are capable of locating foreground patches embodied in complicated, multi-object scenery.
6 Conclusion
Few-shot image classification benefits from increasingly more complex network and algorithm design, but little attention has been focused on image itself. In this paper, reveal that image background serves as a source of harmful knowledge that few-shot learning models easily absorb in. This problem is tackled by our COSOC framework that can draw the model's attention to image foreground at both training and evaluation. Our method is only one possible solution, and future work may include exploring the potential of unsupervised segmentation or detection algorithms which may be more reliable alternative of random cropping, or looking for a completely different but better algorithm customized for foreground extraction.
Acknowledgments and Disclosure of Funding
Special thanks to Qi Yong, who gives indispensable support on the spirit of this paper. We also thank Junran Peng for his help and fruitful discussions. This paper was partially supported by National Key Research and Development Program of China (No. 2018 2018AAA0000200 and key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ202001091134038266,
10

---

References
[1] Radhakrishna Achanta, Sheila S. Hemami, Francisco J. Estrada, and Sahine Süsstrunk. Frequency-tuned salient region detection. In CVPR, 2009.
[2] Arman Afrasiyabi, Jean-Frange Finn-rraco Lalonde, and Christian Gagné. Associative alignment for few-shot image classification. In ECCV, 2020.
[3] Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lec. Meta-lcarning with adaptive hyperparameters. In NIPS, 2020.
[4] Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV, 2018.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NIPS, 2020.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. In ICML,, 2020.
[7] Wei- Yu Chen, Ven-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR, 2019.
[8] Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: exploring simple meta-lcarning for few-shot learning. In ICCV, 2021.
[9] Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation neta-networks for one-shot learning. In CVPR, 2019.
[10] Carl Doersch, Ankush Cupta, and Andrew Zisserman. Crosstransformers: spatially aware few-shot transfer. In NIPS, 2020.
[11] Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang. MELR: meta-learning via modeling episode-level relationships for few-shot learning, In ICLR, 2021.
[12] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In JCML, 2017.
[13] Yizhao Gao, Nanyi Fci, Guangzhen Liu, Zhiwu Lu, and Tao Xiang. Contrastive prototype learning with augmented embeddings for few-shot learning. In UAI, 2021.
[14] Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In CVPR, 2018.
[15] Jean-Baster BastiBBstenn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self- supervised learning. In NIPS, 2020.
[16] Bharath Hariharan and Ross B. Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, 2017.
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, 2020.
[18] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. In TPAMI, 2021.
[19] Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for few-shot classification. In NIPS. 2019.
[20] Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot leaning. In CVPR, 2019.
[21] Jackycom Kim, Hyoungseok Kim, and Gunhee Kim. Model-agnostic adversaral sampling for test-time generalization in few-shot learning. In ECCV, 2020.
[22] K Kwonjoon Lec, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-lcarning with differen- tiable convex optimization. In CVPR, 2019.
[23] Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng. Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In CVPR, 2020.
[24] Fei-Fei Li. Robert Fergus, and Pietro Perona. Oue-shot learning of object categories. In TPAMI, 2006.
[25] Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task-relevant
features for few-shot learning by category traversal. In CVPR, 2019.
[26] Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu. Adversarial feature hallucination networks for few-shot learning. In CVPR, 2020.
[27] Zhenguo Li, Fengwei Zhou, Fci Chen, and Hang Li. Mcta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: 07935, 2017.
[28] Yann I Lifchitz, Yannis Avrithis, Sylvaine Picard, and Andrei Bursuc. Dense classification and implanting for few-shot learning. In CVPR, 2019.
[29] Bin Liu, Yue Cao. Yutong Lin, Qi Li. Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in sho-shot classification. In ECCV, 2020.
11

---

[30]
Chen Liu, Yanwei Fu, Chengming Xu, Siqian Yang, and Jilin Li. Learning few-shot embedding model with contrastive learning. In AAAI, 2021.
[31] Xu Luo, Yuxuan Chen, Liangjian Wen, Lili Pan, and Zenglin Xu. Boosting few-shot classification with siew-learnable contrastive learning. In ICME, 2021.
[32] James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley synposium on mathematical statistics and probability, 1967.
[33] Orchid Majumder, Avinash Ravichandran, Subhransu Maji. Marzia Polito, Rahul Bhotika, and Stefano Soatto. Revisiting contrastive learning for few-shot classification. arXiv preprint arXiv:2101. 2021.
[34] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICLR, 2017.
[35] Yassine Ouali, Céline Hudclot, and Myriam Tami. Spatial contrastive learning for few-shot classification. In ECML/PKDD, 2021.
[36] Funbyung Park and Junier B. Oliva. Meta-curvature. In NIPS, 2019.
[37] Seong-Jin Park, Seungju Han, Ji-Won Back, Insoo Kim, Juhwan Song. Hacbeom Lee, Jac-Joon Han, and Sung Ju Hwang. Meta variance transfer: Learning to augment from the others. In ICML, 2020.
[38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, serf-peforrance deep learning library. In NIPS, 2019.
[39] Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-leaming with implicit gradients. In NIPS, 2019.
[40] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
[41] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell. Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for supervisd few-shot classification. In ICLR. 2018.
[42] Mamshad Nayeem Rizve. Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Exploring complemen- tary strengths of invariant and equivariant representations for few-shot learning. In CVPR, 2021.
[43] Amir Rosenfeld, Richard S. Zemel, and John K. Tsotsos. The elephant in the room. arXiv preprint arXiv:1808.00 03300, 2018.
[44] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large
scale visual recognition challenge. In IJCV, 2015.
[45] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learming with latent embedding optimization. In ICLR, 2019.
[46] Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary. Mattias Marder, Abhishek Kumar, Rogério Schmidt Feris, Raja Giryes. and Alexander M. Bronstein. Delta-encoder: effective sam- plc synthesis method for few-shot object recognition. In NIPS, 2018.
[47] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In CVPR, 2020.
[48] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
[49] Flood Sung. Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In CVPR, 2018.
[50] Sebastian Thrun and Lorien Y. Pratt. Learning to learn: Introduction and overview. In Learning to Learn, 1998.
[Si] Yonglong Tian, Yue Wang. Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: A good embedding all you need? In ECCV, 202V.
[52] Antonio Torralba. Contextual priming for object detection. JJC,, 2003.
[53] Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, 2016.
[54] Yu-Xiong Wang, Ross B. Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018
[55] Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map reconstruction networks. In CVPR, 2021.
[56] Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In ICLR, 2021.
[57] Chengming Xu, Chen Liu, Li Zhang, Chengjie Wang, Jilin Li, Feiyue Huang, Xiangyang Xue, and Yanwei Fu. Learning dynamic alignment via meta-filter for few-shot learning. In CVPR, 2021.
[58] Jin Xu, Jean- Francois Ton, Hyunjik Kim, Adam R. Kosiorek, and Yee Whye Teh. Metafun: Meta-learning with iterative functional updates. In ICML, 2020.
12

---

[59]
Weijian Xu. yifan Huaijin Wang, and Zhuowen Tu. Attentional constellation nets for few-shot learning. In ICLR, 2021.
[60] Han-Jia Ye. Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In CVPR, 2020.
[61] Sung Whan Yoon, Jun Seo, and Jackyun Moon. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. ICML,, 2019.
[62] Zhongqi Yuc, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. Interventional few-shot learning. In NIPS, 2020.
[63] Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, 2020.
[64] Jianguo Zhang. Marcin Marszalek, Svetlana Lazebnik, and Cordelia Schmid. Local features and kernels for classification of texture and object categories: A comprehensive study. 1JCV, 2007.
[65]
Jianming Zhang, Stan Sclaroff, Zhe L. Lin, Xiaohui Shen, Brian L. Price, and Radomír Mech. Minimum barrier salient object detection at 80 FPS. In ICCV, 2015.
[66] Manli Zhang, Jianhong Zhang, Zhiwu Lu. Tao Xiang, Mingyu Ding, and Songfang Huang. IEPT: instance- level and episode-level pretext tasks for few-shot learning. In ICLR, 2021.
[67]
Ruixiang Zhang. Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach few-shot learning. In NIPS, 2018.
[68]
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? ICLR, 2021.
[69] Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust background detection. CVPR, 2014.
[70] Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whitcson, Fast context adaptation via meta-learning, In ICML, 2019.
13


Reviews:
[{"review_id": "umQoLVkIy7J", "reviewer": "Reviewer_ua3N", "summary": "The paper proposed a few-short learning (FSL) method by excluding the negative impact of background noise common in the existing FSL framework. A few separate stages are employed here to achieve the goal: 1) Clustering-based Object Seeker: a pretraining stage to pretrain the network and hence the feature space of the foreground objects; 2) Shared Object Concentrator (SOC): get the representation of the foreground of each class (which is then used in test time to determine the foreground of a test image). The method showed good empirical results on selected datasets. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The originality of the work is moderate - Specific steps of training and extracting foreground features are integrated into the existing FSL framework. The paper is well-written in general except for a few issues mentioned below. The significance of the work though is work debating and a few major questions are listed here. \n\nThe key issue of the work resides in the test time of the method, i.e. using multiple crops to aggregate the scores of each class. I am sure this step can improve the final performance, however it is dangerously close to the common trick of test time augmentation to boost the model performance in practice. Simply put, for other few-shot methods, if we simply do multiple crops and averaging the scores as the final output, it is very likely that most of them, if not all, would see some performance improvement. If we further do a clever weighted sum, e.g. assign a weight based on the similarity of crops to the images of a given class in the training data, the performance may get further improved. In other words, the setup here is not persuasive enough to me that the improvement is mainly caused by rectifying the short learning of background - which is the main conclusion of the paper, although I’d be very much inclined to believe so in theory. \n\nSecondly, the reason why the miniImageNet is not directly used in the comparison is worth a detailed explanation. Most of the SOTA methods report performance on miniImageNet directly, yet here a subset of miniImageNet is deliberately constructed for comparison. Why add this additional step into the experiment preparation? \n\nAdditionally, a large portion of the work tried to construct an effective foreground detector and tried to persuade the audience that the proposed method: random cropping + contrastive learning + clustering solved the problem well. It could easily trigger the curiosity of an audience that why not try some existing saliency detector and combine it with the few-shot framework? Does the proposed foreground detector work better than the existing saliency detectors? If so, is it because of better foreground detection results or because of the inherent pretraining step? A further discussion would be helpful for the audience to understand better the value of this work. \n\n\nA couple of relatively minor issues that worth consideration: \n\n1. In the Step 2 of 4.2, the meaning of \\alpha^{j-1} and \\beta^{n-1} is unclear. Specifically, the meaning of the superscript here could be further clarified especially if alpha and beta are all fixed in the paper? \n\n\n2. Considering the numerous separate steps in the entire workflow, it would be helpful to show some failure cases and try to attribute the cause to the failure of individual steps. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations have been discussed in the work. No negative societal impact as far I can think of. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hrs"}, "extra_text": {}}, {"review_id": "kb_i7figIjz", "reviewer": "Reviewer_Q4tA", "summary": "This paper presents a new few-shot learning framework COSOC that can focus more on foreground objects at both pre-training and evaluation stage. COSOC is a two-stage algorithm. At the pretraining stage, a clustering-based object seeker (COS) module is used to force the pretraining model to focus on found foreground objects by a fusion sampling strategy. At the evaluation stage, a shared object concentrator (SOC) module is used to seek for shared contents and filter out background, and then the foreground of testing images is matched with the recognized foreground objects of each class. Experiments on two benchmarks are conducted to verify the effectiveness of the presented method.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strengths of this work are as follows.\n\n+This paper considers that background information is easy to be used as a shortcut to distinguish images and constructs a small dataset D_new with proper experiments to analyze the adverse effect of background information in few-shot learning.\n\n+This paper presents a new few-shot learning framework COSOC.\n\n+The results seem good.\n\nHowever, several major concerns should be addressed.\n\n-More details should be provided for the training of contrastive learning models (Exampler used in this paper, a modified version of MoCo). It raises a major concern on the baseline. How the model is initialized? How many training epochs are used? The training is commonly expensive, and how much time is used? Does the training follow the original paper?\n\n-As shown by the results, the presented method achieves more performance gain in the 1-shot setting. But it is really in doubt that the presented method can locate the foreground by using V randomly crops (no clustering is used in 1-shot setting, as stated in the paper). This is a major problem. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "There are no obvious ethical concerns on this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "30"}, "extra_text": {}}, {"review_id": "ibjuXY7m5pG", "reviewer": "Reviewer_z1Ft", "summary": "This paper dives into the importance of learned “shortcuts” based on background similarly across the training samples in few-shot learning scenarios, at the detriment of generalization to images with a “category gap” - a background that does not match what was seen in training. They explore the effect of background on recognition and propose two simple algorithms based on contrastively-learned features (one method at train time, one at test) to find foreground vs background and learn more generalizable representations with little data. At training time they use clustering over contrastive-pretrained features extracted from a set of crops from images across the training data for each class to find crops that are highly likely to contain foreground and use those crops instead of the image (with some probability, based on the “foreground score” of the crop). At test time a representative feature per-class is generated using the training data, and simple feature matching is used to determine the class.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper was an enjoyable read - It was well laid out and clearly written. The problem formulation was justified with the analysis in section 3.1, the methods were simple, intuitive, and explained clearly and the results outperformed the methods they compared against. I like that the methods were so simple (feature matching!) and relied heavily on available and powerful contrastively-learned feature representations. There is possibly further room to improve by extending these simple approaches in the future.\n\nHere’s an earlier paper that seems quite relevant, it points out performance drops when moving to novel image backgrounds: https://arxiv.org/abs/1807.04975 \n\nThis method has a strong inter-class visual similarity assumption, are there categories which might break this? Did you see any consistent failure modes of the method? Possibly worth mentioning in discussion.\n\nIn line 109: “crop each image manually” - how exactly? Do you take the largest bounding box inside the object or make a per-pixel segmentation mask?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Beyond the known and problematic biases in the underlying datasets they use, I do not see potential for negative societal impact from this work", "review_text": "", "extra_scores": {"ethical_concerns": "I do not see ethical concerns with this paper", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "4g0o55GCU56", "reviewer": "Reviewer_2di6", "summary": "In this work, the authors conducted an emperical analysis and revealed that image background serves as a source of shortcut knowledge which is harmful for few-shot learning task. To alleviate this, they proposed a two-stage method, COSOC, that employs contrastive learning to draw the model's attention to image foreground during both pre-training and evaluation stage. Given the assumption that foreground objects from different images of the same class share more similar patterns as background, in the pre-training stage, a clustering is performed on randomly cropped patches to identify crops that best represent the foreground objects in the image. Then, fusion sampling is adopted, i.e. these crops are used to replace original image with a certain probability based on its foreground score during pre-training. In the evaluation stage, mean features maximally representing the shared contents among training images from the same class is obtained. Given this, the foregrounds in testing images can be located via matching.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The overall writing is clear and easy to follow. \n- Although the idea of removing background to avoid learning shortcut knowledge is intuitive, its realization is however non-trivial. The authors provide a feasible solution to this, where its effectiveness is well proven by extensive experiments.\n\nQuestions\n- The fusion strategy selects either the original image or randomly cropped patch with a certain probability. Why not using the top K crops discovered in step 4 in L202 to replace the randomly cropped patch? This also better simulates the training setting in Sec 3.1.\n- In the evaluation stage, instead of computing features $\\omega$ that minimizes the sum of pair-wise distances between crops from different images within the same class, why not performing clustering on the training images to extract reliable crops representing the foregrounds similar to what's being done in the pre-training stage for matching?\n- A simple yet possibly strong baseline could be running unsupervised saliency detection to extract the most salient region in an image, followed by cropping to obtain patches without background. This also eliminates the need of having large number of crops to avoid missing the foreground. The authors could consider comparing with this.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to \"Main Review\" section above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors provide a simple but effective technique for reducing bias towards background pixels, in a few-shot setting. They provided extensive data and explanations in their rebuttal. Though two of the reviewers remained unconvinced, the other two reviewers were very supportive. However, I believe that the simplicity of the method and it's clear effectiveness, along with the extensive empirical studies, makes it a good candidate to be accepted to Neurips 2021. The authors should (as they have themselves promised) revise the paper to incorporate the good comments from the reviewers.
Decision: Accept (Poster)