Rectifying the Shortcut Learning of Background for Few-Shot Learning

1 Introduction . Through observing a few samples at a glance , humans can accurately identify brand-new objects . This advantage comes from years of experiences accumulated by the human vision system . Inspired by such learning capabilities , Few-Shot Learning ( FSL ) is developed to tackle the problem of learning from limited data [ 24 , 53 ] . At training , FSL models absorb knowledge from a large-scale dataset ; later at evaluation , the learned knowledge is leveraged to solve a series of downstream classification tasks , each of which contains very few support ( training ) images from brand-new categories . The category gap between training and evaluation has been considered as one of the core issues in FSL [ 10 ] . Intuitively , the prior knowledge of old categories learned at training may not be applicable to novel ones . [ 62 ] consider solving this problem from a causal perspective . Their backdoor adjustment method , however , adjusts the prior knowledge in a black-box manner and can not tell which specific prior knowledge is harmful and should be suppressed . In this paper , we identify image background as one specific harmful source knowledge for FSL . Empirical studies in [ 56 ] suggest that there exists spurious correlations between background and category of images ( e.g. , birds usually stand on branches , and shells often lie on the beaches ; see Fig . 1 ) , which serves as a shortcut knowledge for modern CNN-based vision systems to learn . It is further revealed that background knowledge has positive impact on the performance of in-class classification tasks . As illustrated in the simple example of Fig . 1 , images from the same category are more likely to share similar background , making it possible for background knowledge to generalize from training to testing in common classification tasks . For FSL , however , the category gap produces brand-new foreground , background and their combinations at evaluation . The correlations learned at training thus may not be able to generalize and would probably mislead the predictions . We take * Corresponding author Code : https : //github.com/Frankluox/FewShotCodeBase 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . empirical investigations on the role of image foreground and background in FSL , revealing how image background drastically affects the learning and evaluation of FSL in a negative way . Since the background is harmful , it would be good if we could force the model to concentrate on foreground objects at both training and evaluation , but this is not easy since we do not have any prior knowledge of the entity and position of the foreground objects in images . When humans are going to recognize foreground objects of images from the same class , they usually look for a shared local pattern that appears in the majority of images , and recognize patches with this pattern as foreground . This inspires us to design a novel framework , COSOC , to extract foreground of images for both training and evaluation of FSL by seeking shared patterns among images . The approach does not depend on any additional fine-grained supervisions such as bounding boxes or pixel-level labelings . The procedure of foreground extraction of images in the training set is implemented before training . The corresponding algorithm , named Clustering-based Object Seeker ( COS ) , first pre-trains a feature extractor on the training set using contrative learning , which has an outstanding performance , shown empirically in a later section , on the task of discriminating between ground-truth foreground objects . The feature extractor then maps random crops of images—candidates of foreground objects—into a well-shaped feature space . This is followed by runing a clustering algorithm on all of the features of the same class , imitating the procedure of seeking shared local patterns inspired by human behavior . Each cropped patch is then assigned a foreground score according to its distance to the nearest cluster centroid , for determining a sampling probability of that patch in the later formal training of FSL models . For evaluation , we develop Shared Object Concentrator ( SOC ) , an algorithm that applies iterative feature matching within the support set , looking for one crop per image at one time that is most likely to be foreground . The sorted averaging features of obtained crops are further leveraged to match crops of query images so that foreground crops have higher matching scores . A weighted sum of matching scores are finally calculated as classification logits of each query sample . Compared to other potential foreground extracting algorithms such as saliency-based methods , our COS and SOC algorithms have additional capability of capturing shared , inter-image information , performing better in complicated , multi-object scenery . Our methods also have flexibility of dynamically assigning beliefs ( probabilities ) to all candidate foreground objects , relieving the risk of overconfidence . Our contributions can be summarized as follows . i ) By conducting empirical studies on the role of image foreground and background in FSL , we reveal that image background serves as a source of shortcut knowledge which harms the evaluation performance . ii ) To solve this problem , we propose COSOC , a framework combining COS and SOC , which can draw the model ’ s attention to image foreground at both training and evaluation . iii ) Extensive experiments for non-transductive FSL tasks demonstrate the effectiveness of our method . 2 Related Works . Few-shot Image Classification . Plenty of previous work tackled few-shot learning in meta-learning framework [ 18 , 50 ] , where a model learns experience about how to solve few-shot learning tasks by tackling pseudo few-shot classification tasks constructed from the training set . Existing methods that ultilize meta-learning can be generally divided into three groups : ( 1 ) Optimization-based methods learn the experience of how to optimize the model given few training samples . This kind of methods either meta-learn a good model initialization point [ 12 , 45 , 39 , 70 , 20 ] or the whole optimization process [ 40 , 58 , 34 , 27 ] or both [ 3 , 36 ] . ( 2 ) Hallucination-based methods [ 16 , 54 , 46 , 67 , 25 , 9 , 26 , 37 ] learn to augment similar support samples in few-shot tasks , thus can greatly alleviate the low-shot problem . ( 3 ) Metric-based methods [ 53 , 48 , 49 , 61 , 59 ] learn to map images into a metric feature space and classify query images by computing feature distances to support images . Among them , several recent works [ 19 , 63 , 57 , 10 ] intended to seek correspondence between images either by attention or meta-filter , in order to obtain a more reasonable similarity measure . Our SOC algorithm in one-shot setting is in spirit similar to these methods , in that we both apply pair-wise feature alignment between support and query images , implicitly removing backgrounds that are more likely to be dissimilar across images . SOC differs in multi-shot setting , where potentially useful shared inter-image information in support set exists and can be captured by our SOC algorithm . The Influence of Background . A body of prior work studied the impact of image background on learning-based vision systems from different perspectives . [ 52 ] showed initial evidence of the existence of background correlations and how it influences the predictions of vision models . [ 64 , 43 ] analyzed background dependence for object detection . Another relevant work [ 4 ] utilized camera traps for investigating how performance drops when adapting classifiers to unseen cameras with novel backgrounds . They explore the effect of class-independent background ( i.e. , background changes from training to testing while categories remain the same ) on classification performance . Although the problem is also concerned with image background , no shortcut learning of background exists under this setting . This is because under each training camera trap , the classifier must distinguish different categories with background fixed , causing the background knowledge being not useful for predictions of training images . Instead , the learning signal during training pushes the classifier towards ignoring each specific background . The difficulties under this setting lie in the domain shift challenge—the classifier is confident to handle previously existing backgrounds , but lost in novel backgrounds . More recently , [ 56 ] systematically explore the role of image background in modern deep-learning-based vision systems through well-designed experiments . The results give clear evidence on the existence of background correlations and identify it as a positive shortcut knowledge for models to learn . Our results , on the contrary , identify background correlations as a negative knowledge in the context of few-shot learning . Contrastive Learning . Recent success on contrastive learning of visual representations has greatly promoted the development of unsupervised learning [ 6 , 17 , 15 , 5 ] . The promising performance of contrastive learning relies on the instance-level discrimination loss which maximizes agreement between transformed views of the same image and minimizes agreement between transformed views of different images . Recently there have been some attempts [ 30 , 13 , 10 , 33 , 35 , 31 ] at integrating contrastive learning into the framework of FSL . Although achieving good results , these work struggle to have an in-depth understanding of why contrastive learning has positive effects on FSL . Our work takes a step forward , revealing the advantages of contrastive learning over supervised FSL models in identifying core objects of images . 3 Empirical Investigation . Problem Definition . Few-shot learning consists of a training set DB and an evaluation set Dv which share no overlapping classes . DB contains a large amount of labeled data and is usually used at first to train a backbone network fθ ( · ) . After training , a set of N -way K-shot classification tasks T = { ( Sτ , Qτ ) } NTτ=1 are constructed , each by first sampling N classes in Dv and then sampling K and M images from each class to constitute Sτ and Qτ , respectively . In each task τ , given the learned backbone fθ ( · ) and a small support set Sτ = { ( xτk , n , yτk , n ) } K , N k , n=1 consisting of K images xτk , n and corresponding labels y τ k , n from each of N classes , a few-shot classification algorithm is designed to classify MN images from the query set Qτ = { ( xτmn ) } M , N m , n=1 . Preparation . To investigate the role of background and foreground in FSL , we need ground-truth image foreground for comparison . However , it is time-consuming to label the whole dataset . Thus we select only a subset Dnew = ( DB , Dv ) of miniImageNet [ 53 ] and crop each image manually according to the largest rectangular bounding box that contains the foreground object . We denote the uncropped version of the subset as ( DB-Ori , Dv-Ori ) , and the cropped foreground version as ( DB-FG , Dv-FG ) . Two well-known FSL baselines are selected in our empirical studies : Cosine Classifier ( CC ) [ 14 ] and Prototypical Networks ( PN ) [ 48 ] . See Appendix A for details of constructing Dnew and a formal introdcution of CC and PN . 3.1 The Role of Foreground and Background in Few-Shot Image Classification . Fig . 2 ( a ) shows the average of 5-way 5-shot classification accuracy obtained by training CC and PN on DB-Ori and DB-FG , and evaluating on Dv-Ori and Dv-FG , respectively . See Appendix F for additional 5-way 1-shot experiments . Category gap disables generalization of background knowledge . It can be first noticed that , under any condition , the performance is consistently and significantly improved if background is removed at the evaluation stage ( switch from Dv-Ori to Dv-FG ) . The result implies that background at the evaluation stage in FSL is harmful . This is the opposite of that reported in [ 56 ] which shows background helps improve on the performance of traditional classification task , where no category gap exists between training and evaluation . Thus we can infer that the class/distribution gap in FSL disables generalization of background knowledge and degrades performance . Removing background at training prevents shortcut learning . When only foreground is given at evaluation ( Dv-FG ) , the models trained with only foreground ( DB-FG ) perform much better than those trained with original images ( DB-Ori ) . This indicates that models trained with original images may not pay enough attention to the foreground object that really matters for classification . Background information at training serves as a shortcut for models to learn and can not generalize to brand-new classes . In contrast , models trained with only foreground `` learn to compare '' different objects—a desirable ability for reliable generalization to downstream few-shot learning tasks with out-of-domain classes . Training with background helps models to handle complex scenes . When evaluating on Dv-Ori , the models trained with original dataset DB-Ori are slightly better than those with foreground dataset DB-FG . We attribute this to a sort of domain shift : models trained with DB-FG never meet images with complex background and do not know how to handle it . In Appendix D.1 we further verify the assertion by showing evaluation accuracy of each class under the above two training situations . Note that since we apply random crop augmentation at training , domain shift does not exist if the models are instead trained on DB-Ori and evaluated on Dv-FG . Simple fusion sampling combines advantages of both sides . One may wish to cut off shortcut learning of background while maintaining adaptability of model to complex scenes . A simple solution may be fusion sampling : given an image as input , choose its foreground version with probability p , and its original version with probability 1 − p. We simply set p equal to 0.5 . We denote the dataset using this sampling strategy as DB-Fuse . As observed in Fig . 2 ( a ) , models trained this way indeed combine advantages of both sides : achieving relatively good performance on both Dv-Ori and Dv-FG . In Appendix C , we compare the training curves of PN trained on three versions of datasets to further investigate the effectiveness of fusion sampling . The above analysis provides new inspiration for how to improve FSL further : ( 1 ) Fusion sampling of foreground and original images could be applied to training . ( 2 ) Since background information disturbs evaluation , it is needed to focus on foreground objects or assign image patches , that are more likely to be foreground , a larger weight for classification . Therefore , a foreground object identification mechanism is required at both training ( for fusion sampling ) and evaluation .

Summary: The paper proposed a few-short learning (FSL) method by excluding the negative impact of background noise common in the existing FSL framework. A few separate stages are employed here to achieve the goal: 1) Clustering-based Object Seeker: a pretraining stage to pretrain the network and hence the feature space of the foreground objects; 2) Shared Object Concentrator (SOC): get the representation of the foreground of each class (which is then used in test time to determine the foreground of a test image). The method showed good empirical results on selected datasets.