Rectifying the Shortcut Learning of Background for Few-Shot Learning

1 Introduction . Through observing a few samples at a glance , humans can accurately identify brand-new objects . This advantage comes from years of experiences accumulated by the human vision system . Inspired by such learning capabilities , Few-Shot Learning ( FSL ) is developed to tackle the problem of learning from limited data [ 24 , 53 ] . At training , FSL models absorb knowledge from a large-scale dataset ; later at evaluation , the learned knowledge is leveraged to solve a series of downstream classification tasks , each of which contains very few support ( training ) images from brand-new categories . The category gap between training and evaluation has been considered as one of the core issues in FSL [ 10 ] . Intuitively , the prior knowledge of old categories learned at training may not be applicable to novel ones . [ 62 ] consider solving this problem from a causal perspective . Their backdoor adjustment method , however , adjusts the prior knowledge in a black-box manner and can not tell which specific prior knowledge is harmful and should be suppressed . In this paper , we identify image background as one specific harmful source knowledge for FSL . Empirical studies in [ 56 ] suggest that there exists spurious correlations between background and category of images ( e.g. , birds usually stand on branches , and shells often lie on the beaches ; see Fig . 1 ) , which serves as a shortcut knowledge for modern CNN-based vision systems to learn . It is further revealed that background knowledge has positive impact on the performance of in-class classification tasks . As illustrated in the simple example of Fig . 1 , images from the same category are more likely to share similar background , making it possible for background knowledge to generalize from training to testing in common classification tasks . For FSL , however , the category gap produces brand-new foreground , background and their combinations at evaluation . The correlations learned at training thus may not be able to generalize and would probably mislead the predictions . We take * Corresponding author Code : https : //github.com/Frankluox/FewShotCodeBase 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . empirical investigations on the role of image foreground and background in FSL , revealing how image background drastically affects the learning and evaluation of FSL in a negative way . Since the background is harmful , it would be good if we could force the model to concentrate on foreground objects at both training and evaluation , but this is not easy since we do not have any prior knowledge of the entity and position of the foreground objects in images . When humans are going to recognize foreground objects of images from the same class , they usually look for a shared local pattern that appears in the majority of images , and recognize patches with this pattern as foreground . This inspires us to design a novel framework , COSOC , to extract foreground of images for both training and evaluation of FSL by seeking shared patterns among images . The approach does not depend on any additional fine-grained supervisions such as bounding boxes or pixel-level labelings . The procedure of foreground extraction of images in the training set is implemented before training . The corresponding algorithm , named Clustering-based Object Seeker ( COS ) , first pre-trains a feature extractor on the training set using contrative learning , which has an outstanding performance , shown empirically in a later section , on the task of discriminating between ground-truth foreground objects . The feature extractor then maps random crops of images—candidates of foreground objects—into a well-shaped feature space . This is followed by runing a clustering algorithm on all of the features of the same class , imitating the procedure of seeking shared local patterns inspired by human behavior . Each cropped patch is then assigned a foreground score according to its distance to the nearest cluster centroid , for determining a sampling probability of that patch in the later formal training of FSL models . For evaluation , we develop Shared Object Concentrator ( SOC ) , an algorithm that applies iterative feature matching within the support set , looking for one crop per image at one time that is most likely to be foreground . The sorted averaging features of obtained crops are further leveraged to match crops of query images so that foreground crops have higher matching scores . A weighted sum of matching scores are finally calculated as classification logits of each query sample . Compared to other potential foreground extracting algorithms such as saliency-based methods , our COS and SOC algorithms have additional capability of capturing shared , inter-image information , performing better in complicated , multi-object scenery . Our methods also have flexibility of dynamically assigning beliefs ( probabilities ) to all candidate foreground objects , relieving the risk of overconfidence . Our contributions can be summarized as follows . i ) By conducting empirical studies on the role of image foreground and background in FSL , we reveal that image background serves as a source of shortcut knowledge which harms the evaluation performance . ii ) To solve this problem , we propose COSOC , a framework combining COS and SOC , which can draw the model ’ s attention to image foreground at both training and evaluation . iii ) Extensive experiments for non-transductive FSL tasks demonstrate the effectiveness of our method . 2 Related Works . Few-shot Image Classification . Plenty of previous work tackled few-shot learning in meta-learning framework [ 18 , 50 ] , where a model learns experience about how to solve few-shot learning tasks by tackling pseudo few-shot classification tasks constructed from the training set . Existing methods that ultilize meta-learning can be generally divided into three groups : ( 1 ) Optimization-based methods learn the experience of how to optimize the model given few training samples . This kind of methods either meta-learn a good model initialization point [ 12 , 45 , 39 , 70 , 20 ] or the whole optimization process [ 40 , 58 , 34 , 27 ] or both [ 3 , 36 ] . ( 2 ) Hallucination-based methods [ 16 , 54 , 46 , 67 , 25 , 9 , 26 , 37 ] learn to augment similar support samples in few-shot tasks , thus can greatly alleviate the low-shot problem . ( 3 ) Metric-based methods [ 53 , 48 , 49 , 61 , 59 ] learn to map images into a metric feature space and classify query images by computing feature distances to support images . Among them , several recent works [ 19 , 63 , 57 , 10 ] intended to seek correspondence between images either by attention or meta-filter , in order to obtain a more reasonable similarity measure . Our SOC algorithm in one-shot setting is in spirit similar to these methods , in that we both apply pair-wise feature alignment between support and query images , implicitly removing backgrounds that are more likely to be dissimilar across images . SOC differs in multi-shot setting , where potentially useful shared inter-image information in support set exists and can be captured by our SOC algorithm . The Influence of Background . A body of prior work studied the impact of image background on learning-based vision systems from different perspectives . [ 52 ] showed initial evidence of the existence of background correlations and how it influences the predictions of vision models . [ 64 , 43 ] analyzed background dependence for object detection . Another relevant work [ 4 ] utilized camera traps for investigating how performance drops when adapting classifiers to unseen cameras with novel backgrounds . They explore the effect of class-independent background ( i.e. , background changes from training to testing while categories remain the same ) on classification performance . Although the problem is also concerned with image background , no shortcut learning of background exists under this setting . This is because under each training camera trap , the classifier must distinguish different categories with background fixed , causing the background knowledge being not useful for predictions of training images . Instead , the learning signal during training pushes the classifier towards ignoring each specific background . The difficulties under this setting lie in the domain shift challenge—the classifier is confident to handle previously existing backgrounds , but lost in novel backgrounds . More recently , [ 56 ] systematically explore the role of image background in modern deep-learning-based vision systems through well-designed experiments . The results give clear evidence on the existence of background correlations and identify it as a positive shortcut knowledge for models to learn . Our results , on the contrary , identify background correlations as a negative knowledge in the context of few-shot learning . Contrastive Learning . Recent success on contrastive learning of visual representations has greatly promoted the development of unsupervised learning [ 6 , 17 , 15 , 5 ] . The promising performance of contrastive learning relies on the instance-level discrimination loss which maximizes agreement between transformed views of the same image and minimizes agreement between transformed views of different images . Recently there have been some attempts [ 30 , 13 , 10 , 33 , 35 , 31 ] at integrating contrastive learning into the framework of FSL . Although achieving good results , these work struggle to have an in-depth understanding of why contrastive learning has positive effects on FSL . Our work takes a step forward , revealing the advantages of contrastive learning over supervised FSL models in identifying core objects of images . 3 Empirical Investigation . Problem Definition . Few-shot learning consists of a training set DB and an evaluation set Dv which share no overlapping classes . DB contains a large amount of labeled data and is usually used at first to train a backbone network fθ ( · ) . After training , a set of N -way K-shot classification tasks T = { ( Sτ , Qτ ) } NTτ=1 are constructed , each by first sampling N classes in Dv and then sampling K and M images from each class to constitute Sτ and Qτ , respectively . In each task τ , given the learned backbone fθ ( · ) and a small support set Sτ = { ( xτk , n , yτk , n ) } K , N k , n=1 consisting of K images xτk , n and corresponding labels y τ k , n from each of N classes , a few-shot classification algorithm is designed to classify MN images from the query set Qτ = { ( xτmn ) } M , N m , n=1 . Preparation . To investigate the role of background and foreground in FSL , we need ground-truth image foreground for comparison . However , it is time-consuming to label the whole dataset . Thus we select only a subset Dnew = ( DB , Dv ) of miniImageNet [ 53 ] and crop each image manually according to the largest rectangular bounding box that contains the foreground object . We denote the uncropped version of the subset as ( DB-Ori , Dv-Ori ) , and the cropped foreground version as ( DB-FG , Dv-FG ) . Two well-known FSL baselines are selected in our empirical studies : Cosine Classifier ( CC ) [ 14 ] and Prototypical Networks ( PN ) [ 48 ] . See Appendix A for details of constructing Dnew and a formal introdcution of CC and PN . 3.1 The Role of Foreground and Background in Few-Shot Image Classification . Fig . 2 ( a ) shows the average of 5-way 5-shot classification accuracy obtained by training CC and PN on DB-Ori and DB-FG , and evaluating on Dv-Ori and Dv-FG , respectively . See Appendix F for additional 5-way 1-shot experiments . Category gap disables generalization of background knowledge . It can be first noticed that , under any condition , the performance is consistently and significantly improved if background is removed at the evaluation stage ( switch from Dv-Ori to Dv-FG ) . The result implies that background at the evaluation stage in FSL is harmful . This is the opposite of that reported in [ 56 ] which shows background helps improve on the performance of traditional classification task , where no category gap exists between training and evaluation . Thus we can infer that the class/distribution gap in FSL disables generalization of background knowledge and degrades performance . Removing background at training prevents shortcut learning . When only foreground is given at evaluation ( Dv-FG ) , the models trained with only foreground ( DB-FG ) perform much better than those trained with original images ( DB-Ori ) . This indicates that models trained with original images may not pay enough attention to the foreground object that really matters for classification . Background information at training serves as a shortcut for models to learn and can not generalize to brand-new classes . In contrast , models trained with only foreground `` learn to compare '' different objects—a desirable ability for reliable generalization to downstream few-shot learning tasks with out-of-domain classes . Training with background helps models to handle complex scenes . When evaluating on Dv-Ori , the models trained with original dataset DB-Ori are slightly better than those with foreground dataset DB-FG . We attribute this to a sort of domain shift : models trained with DB-FG never meet images with complex background and do not know how to handle it . In Appendix D.1 we further verify the assertion by showing evaluation accuracy of each class under the above two training situations . Note that since we apply random crop augmentation at training , domain shift does not exist if the models are instead trained on DB-Ori and evaluated on Dv-FG . Simple fusion sampling combines advantages of both sides . One may wish to cut off shortcut learning of background while maintaining adaptability of model to complex scenes . A simple solution may be fusion sampling : given an image as input , choose its foreground version with probability p , and its original version with probability 1 − p. We simply set p equal to 0.5 . We denote the dataset using this sampling strategy as DB-Fuse . As observed in Fig . 2 ( a ) , models trained this way indeed combine advantages of both sides : achieving relatively good performance on both Dv-Ori and Dv-FG . In Appendix C , we compare the training curves of PN trained on three versions of datasets to further investigate the effectiveness of fusion sampling . The above analysis provides new inspiration for how to improve FSL further : ( 1 ) Fusion sampling of foreground and original images could be applied to training . ( 2 ) Since background information disturbs evaluation , it is needed to focus on foreground objects or assign image patches , that are more likely to be foreground , a larger weight for classification . Therefore , a foreground object identification mechanism is required at both training ( for fusion sampling ) and evaluation .

Summary: This paper dives into the importance of learned “shortcuts” based on background similarly across the training samples in few-shot learning scenarios, at the detriment of generalization to images with a “category gap” - a background that does not match what was seen in training. They explore the effect of background on recognition and propose two simple algorithms based on contrastively-learned features (one method at train time, one at test) to find foreground vs background and learn more generalizable representations with little data. At training time they use clustering over contrastive-pretrained features extracted from a set of crops from images across the training data for each class to find crops that are highly likely to contain foreground and use those crops instead of the image (with some probability, based on the “foreground score” of the crop). At test time a representative feature per-class is generated using the training data, and simple feature matching is used to determine the class.