Self-training For Few-shot Transfer Across Extreme Task Differences

1 INTRODUCTION . Despite progress in visual recognition , training recognition systems for new classes in novel domains requires thousands of labeled training images per class . For example , to train a recognition system for identifying crop types in satellite images , one would have to hire someone to go to the different locations on earth to get the labels of thousands of satellite images . The high cost of collecting annotations precludes many downstream applications . This issue has motivated research on few-shot learners : systems that can rapidly learn novel classes from a few examples . However , most few-shot learners are trained on a large base dataset of classes from the same domain . This is a problem in many domains ( such as medical imagery , satellite images ) , where no large labeled dataset of base classes exists . The only alternative is to train the fewshot learner on a different domain ( a common choice is to use ImageNet ) . Unfortunately , few-shot learning techniques often assume that novel and base classes share modes of variation ( Wang et al. , 2018 ) , class-distinctive features ( Snell et al. , 2017 ) , or other inductive biases . These assumptions are broken when the difference between base and novel is as extreme as the difference between object classification in internet photos and pneumonia detection in X-ray images . As such , recent work has found that all few-shot learners fail in the face of such extreme task/domain differences , underperforming even naive transfer learning from ImageNet ( Guo et al. , 2020 ) . Another alternative comes to light when one considers that many of these problem domains have unlabeled data ( e.g. , undiagnosed X-ray images , or unlabeled satellite images ) . This suggests the possibility of using self-supervised techniques on this unlabeled data to produce a good feature representation , which can then be used to train linear classifiers for the target classification task using just a few labeled examples . Indeed , recent work has explored self-supervised learning on a variety of domains ( Wallace & Hariharan , 2020 ) . However , self-supervised learning starts tabula rasa , and as such requires extremely large amounts of unlabeled data ( on the order of millions of images ) . With more practical unlabeled datasets , self-supervised techniques still struggle to outcompete naive ImageNet transfer ( Wallace & Hariharan , 2020 ) . We are thus faced with a conundrum : on the one hand , few-shot learning techniques fail to bridge the extreme differences between ImageNet and domains such as X-rays . On the other hand , self-supervised techniques fail when they ignore inductive biases from ImageNet . A sweet spot in the middle , if it exists , is elusive . In this paper , we solve this conundrum by presenting a strategy that adapts feature representations trained on source tasks to extremely different target domains , so that target task classifiers can then be trained on the adapted representation with very little labeled data . Our key insight is that a pre-trained base classifier from the source domain , when applied to the target domain , induces a grouping of images on the target domain . This grouping captures what the pre-trained classifier thinks are similar or dissimilar in the target domain . Even though the classes of the pre-trained classifier are themselves irrelevant in the target domain , the induced notions of similarity and dissimilarity might still be relevant and informative . This induced notion of similarity is in contrast to current self-supervised techniques which often function by considering each image as its own class and dissimilar from every other image in the dataset ( Wu et al. , 2018 ; Chen et al. , 2020 ) . We propose to train feature representations on the novel target domain to replicate this induced grouping . This approach produces a feature representation that is ( a ) adapted to the target domain , while ( b ) maintaining prior knowledge from the source task to the extent that it is relevant . A discerning reader might observe the similarity of this approach to self-training , except that our goal is to adapt the feature representation to the target domain , rather than improve the base classifier itself . We call our approach “ Self Training to Adapt Representations To Unseen Problems ” , or STARTUP . In a recently released BSCD-FSL benchmark consisting of datasets from extremely different domains ( Guo et al. , 2020 ) , we show that STARTUP provides significant gains ( up to 2.9 points on average ) over few-shot learning , transfer learning and self-supervision state-of-the-art . To the best of our knowledge , ours is the first attempt to bridge such large task/domain gaps and successfully and consistently outperform naive transfer in cross-domain few-shot learning . 2 PROBLEM SETUP . Our goal is to build learners for novel domains that can be quickly trained to recognize new classes when presented with very few labeled data points ( “ few-shot ” ) . Formally , the target domain is defined by a set of data points ( e.g . images ) XN , an unknown set of classes ( or label space ) YN , and a distribution DN over XN × YN . A “ few-shot learning task ” in this domain will consist of a set of classes Y ⊂ YN , a very small training set ( “ support ” ) S = { ( xi , yi ) } ni=1 ∼ DnN , yi ∈ Y and a small test set ( “ query ” ) Q = { xi } mi=1 ∼ DmN When presented with such a few-shot learning task , the learner must rapidly learn the classes presented and accurately classify the query images . As with prior few-shot learning work , we will assume that before being presented with few-shot learning tasks in the target domain , the learner has access to a large annotated dataset DB known as the base dataset . However , crucially unlike prior work on few-shot learning , we assume that this base dataset is drawn from a very different distribution . In fact , we assume that the base dataset is drawn from a completely disjoint image space XB and a disjoint set of classes YB : DB = { ( xi , yi ) } NBi=1 ⊂ XB × YB where XB is the set of data ( or the source domain ) and YB is the set of base classes . Because the base dataset is so different from the target domain , we introduce another difference vis-a-vis the conventional few-shot learning setup : the learner is given access to an additional unlabeled dataset from the target domain : Du = { xi } Nui=1 ∼ DN Nu Put together , the learner will undergo two phases . In the representation learning phase , the learner will pre-train its representation on DB and Du ; then it goes into the evaluation phase where it will be presented few-shot tasks from the target domain where it learns the novel classes ( Figure 1 ) . 3 RELATED WORK . Few-shot Learning ( FSL ) . This paper explores few-shot transfer , and as such the closest related work is on few-shot learning . Few-shot learning techniques are typically predicated on some degree of similarity between classes in the base dataset and novel classes . For example , they may assume that features that are discriminative for the base classes are also discriminative for the novel classes , suggesting a metric learning-based approach ( Gidaris & Komodakis , 2018 ; Qi et al. , 2018 ; Snell et al. , 2017 ; Vinyals et al. , 2016 ; Sung et al. , 2018 ; Hou et al. , 2019 ) or transfer learning-based approach ( Chen et al. , 2019b ; Wang et al. , 2019 ; Kolesnikov et al. , 2020 ; Tian et al. , 2020 ) . Alternatively , they may assume that model initializations that lead to rapid convergence on the base classes are also good initializations for the novel classes ( Finn et al. , 2017 ; 2018 ; Ravi & Larochelle , 2017 ; Nichol & Schulman ; Rusu et al. , 2019 ; Sun et al. , 2019 ; Lee et al. , 2019 ) . Other methods assume that modes of intra-class variation are shared , suggesting the possibility of learned , class-agnostic augmentation policies ( Hariharan & Girshick , 2017 ; Wang et al. , 2018 ; Chen et al. , 2019c ) . Somewhat related is the use of a class-agnostic parametric model that can “ denoise ” few-shot models , be they from the base or novel classes ( Gidaris & Komodakis , 2018 ; 2019 ) . In contrast to such strong assumptions of similarity between base and novel classes , this paper tackles few-shot learning problems where base and novel classes come from very different domains , also called cross-domain few-shot learning . Cross-domain Few-shot Classification ( CD-FSL ) . When the domain gap between the base and novel dataset is large , recent work ( Guo et al. , 2020 ; Chen et al. , 2019b ) has shown that existing stateof-the-art few-shot learners fail to generalize . Tseng et al . ( 2020 ) attempt to address this problem by simulating cross-domain transfer during training . However , their approach assumes access to an equally diverse array of domains during training , and a much smaller domain gap at test time : for example , both base and novel datasets are from internet images . Another relevant work ( Ngiam et al. , 2018 ) seeks to build domain-specific feature extractor by reweighting different classes of examples in the base dataset based on the target novel dataset but their work only investigates transfer between similar domains ( both source and target are internet images ) . Our paper tackles a more extreme domain gap . Another relevant benchmark for this problem is ( Zhai et al. , 2019 ) but they assume access to more annotated examples ( 1k annotations ) during test time than the usual FSL setup . Few-shot learning with unlabeled data . This paper uses unlabeled data from the target domain to bridge the domain gap . Semi-supervised few-shot learning ( SS-FSL ) ( Ren et al. , 2018 ; Li et al. , 2019 ; Yu et al. , 2020 ; Rodrı́guez et al. , 2020 ; Wang et al. , 2020 ) and transductive few-shot learning ( T-FSL ) ( Liu et al. , 2019 ; Dhillon et al. , 2020 ; Hou et al. , 2019 ; Wang et al. , 2020 ; Rodrı́guez et al. , 2020 ) do use such unlabeled data , but only during evaluation , assuming that representations trained on the base dataset are good enough . In contrast our approach leverages the unlabeled data during representation learning . The two are orthogonal innovations and can be combined . Self-Training . Our approach is closely related to self-training , which has been shown to be effective for semi-supervised training and knowledge distillation . In self-training , a teacher model trained on the labeled data is used to label the unlabeled data and another student model is trained on both the original labeled data and the unlabeled data labeled by the teacher . Xie et al . ( 2020 ) and Yalniz et al . ( 2019 ) have shown that using self-training can improve ImageNet classification performance . Knowledge distillation ( Hinton et al. , 2015 ) is similar but aims to compress a large teacher network by training a student network to mimic the prediction of the teacher network . A key difference between these and our work is that self-training / knowledge distillation focus on a single task of interest , i.e , there is no change in label space . Our approach is similar , but we are interested in transferring to novel domains with a wholly different label space : an unexplored scenario . Domain Adaptation . Transfer to new domains is also in the purview of domain adaptation ( Tzeng et al. , 2017 ; Hoffman et al. , 2018 ; Long et al. , 2018 ; Xu et al. , 2019 ; Laradji & Babanezhad , 2020 ; Wang & Deng , 2018 ; Wilson & Cook , 2020 ) where the goal is to transfer knowledge from the labelabundant source domain to a target domain where only unlabeled data is available . In this realm , self-training has been extensively explored ( Zou et al. , 2018 ; Chen et al. , 2019a ; Zou et al. , 2019 ; Zhang et al. , 2019 ; Mei et al. , 2020 ) . However , a key assumption in domain adaptation is that the source domain and target domain share the same label space which does not hold for FSL . Self-supervised Learning . Learning from unlabeled data has seen a resurgence of interest with advances in self-supervised learning . Early self-supervised approaches were based on handcrafted “ pretext tasks ” such as solving jigsaw puzzles ( Noroozi & Favaro , 2016 ) , colorization ( Zhang et al. , 2016 ) or predicting rotation ( Gidaris et al. , 2018 ) . A more recent ( and better performing ) line of self-supervised learning is contrastive learning ( Wu et al. , 2018 ; Misra & Maaten , 2020 ; He et al. , 2020 ; Chen et al. , 2020 ) which aims to learn representations by considering each image together with its augmentations as a separate class . While self supervision has been shown to boost few-shot learning ( Gidaris et al. , 2019 ; Su et al. , 2020 ) , its utility in cases of large domain gaps between base and novel datasets have not been evaluated . Our work focuses on this challenging scenario .

Summary: Problem: The paper introduces the problem of few-shot transfer when there is an extreme difference between the base task and the target task.  The usual few-shot learning setup considers a representation that is trained on a large amount of labeled data. This base representation is then fine-tuned for the target task (that has a few examples, say 1 or 5 labeled examples per class). This strategy works well when the data distribution of the base and target task is similar. However, few-shot learners fail when the data distribution for the two domains are different (e.g., imagenet and crop-diseases) as shown by Guo et al., 2020.