Rectifying the Shortcut Learning of Background for Few-Shot Learning
Year: 2022

Abstract:
The category gap between training and evaluation has been characterised as one of the main obstacles to the success of Few-Shot Learning (FSL). In this paper, we for the first time empirically identify image background, common in realistic images, as a shortcut knowledge helpful for in-class classification but ungeneralizable beyond training categories in FSL. A novel framework, COSOC, is designed to tackle this problem by extracting foreground objects in images at both training and evaluation without any extra supervision. Extensive experiments carried on inductive FSL tasks demonstrate the effectiveness of our approaches.

heading: 1 Introduction
text: Through observing a few samples at a glance, humans can accurately identify brand-new objects. This advantage comes from years of experiences accumulated by the human vision system. Inspired by such learning capabilities, Few-Shot Learning (FSL) is developed to tackle the problem of learning from limited data [24, 53]. At training, FSL models absorb knowledge from a large-scale dataset; later at evaluation, the learned knowledge is leveraged to solve a series of downstream classification tasks, each of which contains very few support (training) images from brand-new categories.
The category gap between training and evaluation has been considered as one of the core issues in FSL [10]. Intuitively, the prior knowledge of old categories learned at training may not be applicable to novel ones. [62] consider solving this problem from a causal perspective. Their backdoor adjustment method, however, adjusts the prior knowledge in a black-box manner and cannot tell which specific prior knowledge is harmful and should be suppressed.
In this paper, we identify image background as one specific harmful source knowledge for FSL. Empirical studies in [56] suggest that there exists spurious correlations between background and category of images (e.g., birds usually stand on branches, and shells often lie on the beaches; see Fig. 1), which serves as a shortcut knowledge for modern CNN-based vision systems to learn. It is further revealed that background knowledge has positive impact on the performance of in-class classification tasks. As illustrated in the simple example of Fig. 1, images from the same category are more likely to share similar background, making it possible for background knowledge to generalize from training to testing in common classification tasks. For FSL, however, the category gap produces brand-new foreground, background and their combinations at evaluation. The correlations learned at training thus may not be able to generalize and would probably mislead the predictions. We take
*Corresponding author Code: https://github.com/Frankluox/FewShotCodeBase
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
empirical investigations on the role of image foreground and background in FSL, revealing how image background drastically affects the learning and evaluation of FSL in a negative way.
Since the background is harmful, it would be good if we could force the model to concentrate on foreground objects at both training and evaluation, but this is not easy since we do not have any prior knowledge of the entity and position of the foreground objects in images. When humans are going to recognize foreground objects of images from the same class, they usually look for a shared local pattern that appears in the majority of images, and recognize patches with this pattern as foreground. This inspires us to design a novel framework, COSOC, to extract foreground of images for both training and evaluation of FSL by seeking shared patterns among images. The approach does not depend on any additional fine-grained supervisions such as bounding boxes or pixel-level labelings.
The procedure of foreground extraction of images in the training set is implemented before training. The corresponding algorithm, named Clustering-based Object Seeker (COS), first pre-trains a feature extractor on the training set using contrative learning, which has an outstanding performance, shown empirically in a later section, on the task of discriminating between ground-truth foreground objects. The feature extractor then maps random crops of images—candidates of foreground objects—into a well-shaped feature space. This is followed by runing a clustering algorithm on all of the features of the same class, imitating the procedure of seeking shared local patterns inspired by human behavior. Each cropped patch is then assigned a foreground score according to its distance to the nearest cluster centroid, for determining a sampling probability of that patch in the later formal training of FSL models. For evaluation, we develop Shared Object Concentrator (SOC), an algorithm that applies iterative feature matching within the support set, looking for one crop per image at one time that is most likely to be foreground. The sorted averaging features of obtained crops are further leveraged to match crops of query images so that foreground crops have higher matching scores. A weighted sum of matching scores are finally calculated as classification logits of each query sample. Compared to other potential foreground extracting algorithms such as saliency-based methods, our COS and SOC algorithms have additional capability of capturing shared, inter-image information, performing better in complicated, multi-object scenery. Our methods also have flexibility of dynamically assigning beliefs (probabilities) to all candidate foreground objects, relieving the risk of overconfidence.
Our contributions can be summarized as follows. i) By conducting empirical studies on the role of image foreground and background in FSL, we reveal that image background serves as a source of shortcut knowledge which harms the evaluation performance. ii) To solve this problem, we propose COSOC, a framework combining COS and SOC, which can draw the model’s attention to image foreground at both training and evaluation. iii) Extensive experiments for non-transductive FSL tasks demonstrate the effectiveness of our method.

heading: 2 Related Works
text: Few-shot Image Classification. Plenty of previous work tackled few-shot learning in meta-learning framework [18, 50], where a model learns experience about how to solve few-shot learning tasks by tackling pseudo few-shot classification tasks constructed from the training set. Existing methods that ultilize meta-learning can be generally divided into three groups: (1) Optimization-based methods learn the experience of how to optimize the model given few training samples. This kind of methods
either meta-learn a good model initialization point [12, 45, 39, 70, 20] or the whole optimization process [40, 58, 34, 27] or both [3, 36]. (2) Hallucination-based methods [16, 54, 46, 67, 25, 9, 26, 37] learn to augment similar support samples in few-shot tasks, thus can greatly alleviate the low-shot problem. (3) Metric-based methods [53, 48, 49, 61, 59] learn to map images into a metric feature space and classify query images by computing feature distances to support images. Among them, several recent works [19, 63, 57, 10] intended to seek correspondence between images either by attention or meta-filter, in order to obtain a more reasonable similarity measure. Our SOC algorithm in one-shot setting is in spirit similar to these methods, in that we both apply pair-wise feature alignment between support and query images, implicitly removing backgrounds that are more likely to be dissimilar across images. SOC differs in multi-shot setting, where potentially useful shared inter-image information in support set exists and can be captured by our SOC algorithm.
The Influence of Background. A body of prior work studied the impact of image background on learning-based vision systems from different perspectives. [52] showed initial evidence of the existence of background correlations and how it influences the predictions of vision models. [64, 43] analyzed background dependence for object detection. Another relevant work [4] utilized camera traps for investigating how performance drops when adapting classifiers to unseen cameras with novel backgrounds. They explore the effect of class-independent background (i.e., background changes from training to testing while categories remain the same) on classification performance. Although the problem is also concerned with image background, no shortcut learning of background exists under this setting. This is because under each training camera trap, the classifier must distinguish different categories with background fixed, causing the background knowledge being not useful for predictions of training images. Instead, the learning signal during training pushes the classifier towards ignoring each specific background. The difficulties under this setting lie in the domain shift challenge—the classifier is confident to handle previously existing backgrounds, but lost in novel backgrounds. More recently, [56] systematically explore the role of image background in modern deep-learning-based vision systems through well-designed experiments. The results give clear evidence on the existence of background correlations and identify it as a positive shortcut knowledge for models to learn. Our results, on the contrary, identify background correlations as a negative knowledge in the context of few-shot learning.
Contrastive Learning. Recent success on contrastive learning of visual representations has greatly promoted the development of unsupervised learning [6, 17, 15, 5]. The promising performance of contrastive learning relies on the instance-level discrimination loss which maximizes agreement between transformed views of the same image and minimizes agreement between transformed views of different images. Recently there have been some attempts [30, 13, 10, 33, 35, 31] at integrating contrastive learning into the framework of FSL. Although achieving good results, these work struggle to have an in-depth understanding of why contrastive learning has positive effects on FSL. Our work takes a step forward, revealing the advantages of contrastive learning over supervised FSL models in identifying core objects of images.

heading: 3 Empirical Investigation
text: Problem Definition. Few-shot learning consists of a training set DB and an evaluation set Dv which share no overlapping classes. DB contains a large amount of labeled data and is usually used at first to train a backbone network fθ(·). After training, a set of N -way K-shot classification tasks T = {(Sτ ,Qτ )}NTτ=1 are constructed, each by first sampling N classes in Dv and then sampling K and M images from each class to constitute Sτ and Qτ , respectively. In each task τ , given the learned backbone fθ(·) and a small support set Sτ = {(xτk,n, yτk,n)} K,N k,n=1 consisting of K images xτk,n and corresponding labels y τ k,n from each of N classes, a few-shot classification algorithm is designed to classify MN images from the query set Qτ = {(xτmn)} M,N m,n=1.
Preparation. To investigate the role of background and foreground in FSL, we need ground-truth image foreground for comparison. However, it is time-consuming to label the whole dataset. Thus we select only a subset Dnew = (DB ,Dv) of miniImageNet [53] and crop each image manually according to the largest rectangular bounding box that contains the foreground object. We denote the uncropped version of the subset as (DB-Ori,Dv-Ori), and the cropped foreground version as (DB-FG,Dv-FG). Two well-known FSL baselines are selected in our empirical studies: Cosine
Classifier (CC) [14] and Prototypical Networks (PN) [48]. See Appendix A for details of constructing Dnew and a formal introdcution of CC and PN.

heading: 3.1 The Role of Foreground and Background in Few-Shot Image Classification
text: Fig. 2(a) shows the average of 5-way 5-shot classification accuracy obtained by training CC and PN on DB-Ori and DB-FG, and evaluating on Dv-Ori and Dv-FG, respectively. See Appendix F for additional 5-way 1-shot experiments.
Category gap disables generalization of background knowledge. It can be first noticed that, under any condition, the performance is consistently and significantly improved if background is removed at the evaluation stage (switch from Dv-Ori to Dv-FG). The result implies that background at the evaluation stage in FSL is harmful. This is the opposite of that reported in [56] which shows background helps improve on the performance of traditional classification task, where no category gap exists between training and evaluation. Thus we can infer that the class/distribution gap in FSL disables generalization of background knowledge and degrades performance.
Removing background at training prevents shortcut learning. When only foreground is given at evaluation (Dv-FG), the models trained with only foreground (DB-FG) perform much better than those trained with original images (DB-Ori). This indicates that models trained with original images may not pay enough attention to the foreground object that really matters for classification. Background information at training serves as a shortcut for models to learn and cannot generalize to brand-new classes. In contrast, models trained with only foreground "learn to compare" different objects—a desirable ability for reliable generalization to downstream few-shot learning tasks with out-of-domain classes.
Training with background helps models to handle complex scenes. When evaluating on Dv-Ori, the models trained with original dataset DB-Ori are slightly better than those with foreground dataset DB-FG. We attribute this to a sort of domain shift: models trained with DB-FG never meet images with complex background and do not know how to handle it. In Appendix D.1 we further verify the assertion by showing evaluation accuracy of each class under the above two training situations. Note that since we apply random crop augmentation at training, domain shift does not exist if the models are instead trained on DB-Ori and evaluated on Dv-FG. Simple fusion sampling combines advantages of both sides. One may wish to cut off shortcut learning of background while maintaining adaptability of model to complex scenes. A simple solution may be fusion sampling: given an image as input, choose its foreground version with probability p, and its original version with probability 1 − p. We simply set p equal to 0.5. We denote the dataset using this sampling strategy as DB-Fuse. As observed in Fig. 2(a), models trained this way indeed combine advantages of both sides: achieving relatively good performance on both Dv-Ori and Dv-FG. In Appendix C, we compare the training curves of PN trained on three versions of datasets to further investigate the effectiveness of fusion sampling.
The above analysis provides new inspiration for how to improve FSL further: (1) Fusion sampling of foreground and original images could be applied to training. (2) Since background information disturbs evaluation, it is needed to focus on foreground objects or assign image patches, that are more likely to be foreground, a larger weight for classification. Therefore, a foreground object identification mechanism is required at both training (for fusion sampling) and evaluation.

heading: 3.2 Contrastive Learning is Good at Identifying Objects
text: In this subsection, we reveal the potential of contrastive learning in identifying foreground objects, which we will use later for foreground extraction. Given one transformed view of one image, contrastive learning tends to distinguish another transformed view of that same image from thousands of views of other images. A more detailed introduction of contrastive learning is given in Appendix B. The two augmented views of the same image always cover the same object, but probably with different parts, sizes and color. To discriminate two augmented patches from thousands of other image patches, the model has to learn to identify the key discriminative information of the object under varying environment. In this manner, semantic relations among crops of images are explicitly modeled, thereby clustering semantically similar contents automatically. The features of different images are pushed away, while those of similar objects in different images are pulled closer. Thus it is reasonable to speculate that contrastive learning may enable models with better identification of centered foreground object.
To verify this, we train CC and contrastive learning models on the whole training set of miniImageNet (DB-Full) and compare their accuracy on Dv-Ori and Dv-FG. The contrastive learning method we use is Exemplar [68], a modified version of MoCo [17]. Fig 2(b) shows that, while the evaluation accuracy of Exemplar on Dv-Ori is slightly worse than that of CC, Exemplar performs much better when only foreground of images are given at evaluation, affirming that contrastive learning indeed has a better discriminative ability of single centered object. In Appendix D.2, we provide a more in-depth analysis of why contrastive learning has such properties and infer that the shape bias and viewpoint invariance may play an important role.

heading: 4 Rectifying the Shortcut Learning of Background
text: Given the analysis in the previous section, we wish to focus more on image foreground both at training and evaluation. Inspired by how humans recognise foreground objects, we propose COSOC, a framework ultilizing contrastive learning to draw the model’s attention to the foreground objects of images.

heading: 4.1 Clustering-based Object Seeker (COS) with Fusion Sampling for Training
text: Since contrastive learning is good at discriminating foreground objects, we utilize it to extract foreground objects before training. The first step is to pre-train a backbone fθ(·) on the training set DB using Exemplar [68]. Then a clustering-based algorithm is used to extract "objects" identified by the pre-trained model. The basic idea is that features of foreground objects in images within one class extracted by contrastive learning models are similar, thereby can be identified via a clustering algorithm; see a simple example in Fig. 3. All images within the i-th class in DB form a set {xin}Nn=1. For clarity, we omit the class index i in the following descriptions. The scheme of seeking foreground objects in one class is detailed as follows:
1) For each image xn, we randomly crop it L times to obtain L image patches {pn,m}Lm=1. Each image patch pn,m is then passed through the pre-trained model fθ and we get a normalized feature vector vn,m =
fθ(pn,m) ||fθ(pn,m)||2 ∈ R d.
2) We run a clustering algorithm A on all features vectors of the class and obtain H clusters {zj}Hj=1 = A({vn,m} N,L n,m=1), where zj is the feature centroid of the j-th cluster.
3) We say an image xn ∈ zj , if there exists k ∈ [L] s.t. vn,k ∈ zj , where [L] = {1, 2, . . . , L}. Let l(zj) = #{x|x∈zj} N be the proportion of images in the class that belong to zj . If l(zj) is small, then the cluster zj is not representative for the whole class and is possibly background. Thus we remove
all the clusters z with l(z) < γ, where γ is a threshold that controls the generality of clusters. The remaining h clusters {zj}αhj=α1 represent “objects” of the class that we are looking for.
4) The foreground score of image patch pn,m is defined as sn,m = 1−minj∈[h] ||vn,m − zαj ||2/η, where η = max
n,m min j∈[hc] ||vn,m − zαj ||2 is used to normalize the score into [0, 1]. Then top-k
scores of each image xn are obtained as {sn,m}βkm=β1 = Topk m∈[L] (sn,m). The corresponding patches {pn,m}βkm=β1 are seen as possible crops of the foreground object in image xn, and the foreground scores {sn,m}βkm=β1 as the confidence. We then use it as prior knowledge to rectify the shortcut learning of background for FSL models.
The training strategy resembles fusion sampling introduced before. For an image xn, the probability that we choose the original version is 1 − max
i∈[k] sn,βi , and the probability of choosing pn,βj from top-k patches is (sn,βj/ ∑
i∈[k] sn,βi) ·max i∈[k] sn,βi . Then we adjust the chosen image patch and make
sure that the least area proportion to the original image keeps as a constant. We use this strategy to train a backbone fθ(·) using a FSL algorithm.

heading: 4.2 Few-shot Evaluation with Shared Object Concentrator (SOC)
text: As discussed before, if the foreground crop of the image is used at evaluation, the performance of FSL model will be boosted by a large margin, serving as an upper bound of the model performance. To approach this upper bound, we propose SOC algorithm to capture foreground objects by seeking shared contents among support images of the same class and query images.
contains maximal shared information of all images in class c. Ideally, ω1 represents the centroid of the most similar K image patches, each from one image, which can be formulated as
ω1 = 1
K K∑ k=1 vk,λopt(k), (1)
λopt = argmax λ∈[K][V ] ∑ 1≤i<j≤K cos(vi,λ(i),vj,λ(j)), (2)
where cos(·, ·) denotes cosine similarity and [K][V ] denotes the set of functions that take [K] as domain and [V ] as range. While λopt can be obtained by enumerating all possible combinations of image patches, the computation complexity of this brute-force method is O(V K), which is
computation prohibitive when V or K is large. Thus when the computation is not affordable, we turn to use a simplified method that leverages iterative optimization. Instead of seeking for the closest image patches, we directly optimize ω1 so that the sum of minimum distance to patches of each image is minimized, i.e.,
ω1 = argmax ω∈Rd K∑ k=1 max n [cos(ω,vk,n)], (3)
which can be achieved by iterative optimization algorithms. We apply SGD in our experiments. After optimization, we remove the patch of each image that is most similar to ω1, and obtain K × (V − 1) feature vectors. Then we repeatedly implement the above optimization process until no features are left, as shown in Fig. 4. We eventually obtain V sorted feature vectors {ωn}Vn=1, which we use to represent the class c. As for the case where shot K = 1, there is no shared inter-image information inside class, so similar to the handling in PN and DeepEMD [63] , we just skip step 1 and use the original V feature vectors.
Step 2: Feature Matching for Concentrating on Foreground Object of Query Images. Once the foreground class representations are identified, the next step is to use them to implicitly concentrate on foreground of query images by feature matching. For each image x in the query set Qτ , we also randomly crop it for V times and obtain V candidate features {µn}Vn=1. For each class c, we have V sorted representative feature vectors {ωn}Vn=1 obtained in step 1. We then match the most similar patches between query features and class features, i.e.,
s1 = max 1≤i,j≤V
[αj−1cos(µi,ωj)], (4)
where α ≤ 1 is an importance factor. Thus the weight αj−1 decreases exponentially in index n− 1, indicating a decreased belief of each vector representing foreground. Similarly, the two matched features are removed and the above process repeats until no features left. Finally, the score of x w.r.t. class c is obtained as a weighted sum of all similarities, i.e., Sc = ∑V n=1 β
n−1sn, where β ≤ 1 is another importance factor controlling the belief of each crop being foreground objects. In this way, features matched earlier—thus more likely to be foreground—will have higher contributions to the score. The predicted class of x is the one with the highest score.

heading: 5 Experiments

heading: 5.1 Experiment Setup
text: Dataset. We adopt two benchmark datasets which are the most representative in few-shot learning. The first is miniImageNet [53], a small subset of ILSVRC-12 [44] that contains 600 images within each of the 100 categories. The categories are split into 64, 16, 20 classes for training, validation and evaluation, respectively. The second dataset, tieredImageNet [41], is a much larger subset of ILSVRC12 and is more challenging. It is constructed by choosing 34 super-classes with 608 categories. The super-classes are split into 20, 6, 8 super-classes which ensures separation between training and evaluation categories. The final dataset contains 351, 97, 160 classes for training, validation and evaluation, respectively. On both datasets, the input image size is 84 × 84 for fair comparison.
Evaluation Protocols. We follow the 5-way 5-shot (1-shot) FSL evaluation setting. Specifically, 2000 tasks, each contains 15 testing images and 5 (1) training images per class, are randomly sampled from the evaluation set Dv and the average classification accuracy is computed. This is repeated 5 times and the mean of the average accuracy with 95% confidence intervals is reported.
Implementation Details. The backbone we use throughout the article is ResNet-12, which is widely used in few-shot learning. We use Pytorch [38] to implement all our experiments on two NVIDIA 1080Ti GPUs. We train the model using SGD with cosine learning rate schedule without restart to reduce the number of hyperparameters (Which epochs to decay the learning rate). The initial learning rate for training Exemplar is 0.1, and for CC is 0.005. The batch size for Exemplar, CC are 256 and 128, respectively. For miniImageNet, we train Exemplar for 150k iterations, and train CC for 6k iterations. For tieredImageNet, we train Exemplar for approximately 900k iterations, and train CC for 120k iterations. We choose k-means [32] as the clustering algorithm for COS. The threshold γ is set to 0.5, and top 3 out of 30 features are chosen per image at the training stage. At the evaluation stage, we crop each image 7 times. The importance factors α and β are both set to 0.8.

heading: 5.2 Model Analysis
text: In this subsection, we show the effectiveness of each component of our method. Tab. 1 shows the ablation study conducted on miniImageNet.
On the effect of finetuning. Since a feature extractor is pre-trained using contrastive learning in COS, it may help accelerate convergence if we directly finetune from the pre-trained model instead of training from scratch. As shown in line 2-3 in Tab. 1, fintuning gives no improvement on the performance over training from scratch. Thus we adopt finetuning mainly for speeding up convergence (5× faster).
Effectiveness of COS Algorithm. As observed in Tab. 1, When COS is applied on CC, the performance is improved on both versions of datasets. In Fig. 5, we show the curves of training and validation error of CC during training with and without COS. Both models are trained from scratch and validated on the full miniImageNet. We observe that CC sinks into overfitting: the training accuracy drops to zero, and validation accuracy stops improving before
the end of the training. Meanwhile, the COS algorithm helps slow down convergence and prevent training accuracy from reaching zero. This makes validation accuracy comparable at first but higher at the end. Our COS algorithm weakens the “background shortcut” for learning, draws model’s attention on foreground objects, and improves upon generalization.
Effectiveness of SOC Algorithm. The result in Tab. 1 shows that the SOC algorithm is the key to maximally exploit the potential of good object-discrimination ability. The performance even approaches the upper bound performance obtained by evaluating the model on the ground-truth foreground Dv-FG. One potential unfairness in our SOC algorithm may lie in the use of multicropping, which could possibly lead to performance improvement for other approaches as well. We ablate this concern in Appendix G, as well as in the comparisons to other methods in the later subsections.
Note that if we apply only the SOC algorithm on CC, the performance degrades. This indicates that COS and SOC are both necessary: COS provides the discrimination ability of foreground objects and SOC leverages it to maximally boost the performance.

heading: 5.3 Comparison to Saliency-based Foreground Extractors
text: There could be other possible ways of extracting foreground objects. A simple yet possibly strong baseline could be running saliency detection to extract the most salient region in an image, followed by cropping to obtain patches without background. We consider comparing with three classical unsupervised saliency methods—RBD [69], FT [1] and MBD [65]. The cropping threshold is specially tuned. For training, fusion sampling with probability 0.5 is used for unsupervised saliency methods. For evaluation, We replace the original images with crops obtained by unsupervised saliency methods directly for classification. Tab. 2 displays the comparisons of performance using different foreground extraction methods applied at training or evaluation. For fair comparison, all methods are trained from scratch, and all compared baselines are evaluated with multi-cropping (i.e. using the average of features obtained from multiple crops for classification)and tested on the same backbone (COS trained).
The results show that: (1) Our method performs consistently much better than the listed unsupervised saliency methods. (2) The performance of different unsupervised saliency methods varies. While RBD gives a small improvement, MBD and FT have negative effect on the performance. The performance severely depends on the effectiveness of unsupervised saliency methods, and is very sensitive to the cropping threshold. Intuitively speaking, saliency detection methods focus on noticeable objects in the image, and might fail when there is another irrelevant salient object in the image (e.g., a man is walking a dog. Dog is the label, but the man is of high saliency). On the contrary, our method focuses on shared objects across images in the same class, thereby avoiding this problem. In addition, our COS algorithm has the ability to dynamically assign foreground scores to different patches, which reduces the risk of overconfidence. One of our main contributions is paving a new way towards improving FSL by rectifying shortcut learning of background, which can be implemented using any effective methods. Given the upper bound with ground truth foreground, we believe there is room to improve and there can be other more effective approaches in the future.

heading: 5.4 Comparison to State-of-the-Arts
text: Tab. 3 presents 5-way 1-shot and 5-shot classification results on miniImageNet and tieredImageNet. We compare with state-of-the-art few-shot learning methods. For fair comparison, we reimplement some methods, and evaluate them with multi-cropping. See Appendix G for a detailed study on the influence of multi-cropping. Our method achieves state-of-the-art performance under all settings except for 1-shot task on tieredImageNet, on which the performance of our method is slightly worse than CA, which uses WRN-28-10, a deeper backbone, as the feature extractor.

heading: 5.5 Visualization
text: Fig. 6 and 7 display visualization examples of the COS and SOC algorithms. See more examples in Appendix H. Thanks to the well-designed mechanism of capturing shared inter-image information, the COS and SOC algorithms are capable of locating foreground patches embodied in complicated, multi-object scenery.

heading: 6 Conclusion
text: Few-shot image classification benefits from increasingly more complex network and algorithm design, but little attention has been focused on image itself. In this paper, we reveal that image background serves as a source of harmful knowledge that few-shot learning models easily absorb in. This problem is tackled by our COSOC framework that can draw the model’s attention to image foreground at both training and evaluation. Our method is only one possible solution, and future work may include exploring the potential of unsupervised segmentation or detection algorithms which may be a more reliable alternative of random cropping, or looking for a completely different but better algorithm customized for foreground extraction.
Acknowledgments and Disclosure of Funding
Special thanks to Qi Yong, who gives indispensable support on the spirit of this paper. We also thank Junran Peng for his help and fruitful discussions. This paper was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), and a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826).

Review summaries:
The paper proposed a few-short learning (FSL) method by excluding the negative impact of background noise common in the existing FSL framework. A few separate stages are employed here to achieve the goal: 1) Clustering-based Object Seeker: a pretraining stage to pretrain the network and hence the feature space of the foreground objects; 2) Shared Object Concentrator (SOC): get the representation of the foreground of each class (which is then used in test time to determine the foreground of a test image). The method showed good empirical results on selected datasets. 

This paper presents a new few-shot learning framework COSOC that can focus more on foreground objects at both pre-training and evaluation stage. COSOC is a two-stage algorithm. At the pretraining stage, a clustering-based object seeker (COS) module is used to force the pretraining model to focus on found foreground objects by a fusion sampling strategy. At the evaluation stage, a shared object concentrator (SOC) module is used to seek for shared contents and filter out background, and then the foreground of testing images is matched with the recognized foreground objects of each class. Experiments on two benchmarks are conducted to verify the effectiveness of the presented method.

In this work, the authors conducted an emperical analysis and revealed that image background serves as a source of shortcut knowledge which is harmful for few-shot learning task. To alleviate this, they proposed a two-stage method, COSOC, that employs contrastive learning to draw the model's attention to image foreground during both pre-training and evaluation stage. Given the assumption that foreground objects from different images of the same class share more similar patterns as background, in the pre-training stage, a clustering is performed on randomly cropped patches to identify crops that best represent the foreground objects in the image. Then, fusion sampling is adopted, i.e. these crops are used to replace original image with a certain probability based on its foreground score during pre-training. In the evaluation stage, mean features maximally representing the shared contents among training images from the same class is obtained. Given this, the foregrounds in testing images can be located via matching.

This paper dives into the importance of learned “shortcuts” based on background similarly across the training samples in few-shot learning scenarios, at the detriment of generalization to images with a “category gap” - a background that does not match what was seen in training. They explore the effect of background on recognition and propose two simple algorithms based on contrastively-learned features (one method at train time, one at test) to find foreground vs background and learn more generalizable representations with little data. At training time they use clustering over contrastive-pretrained features extracted from a set of crops from images across the training data for each class to find crops that are highly likely to contain foreground and use those crops instead of the image (with some probability, based on the “foreground score” of the crop). At test time a representative feature per-class is generated using the training data, and simple feature matching is used to determine the class.