Switch to Generalize: Domain-Switch Learning for Cross-Domain Few-Shot Classification

1 INTRODUCTION . This paper challenges a realistic problem , i.e. , the cross-domain scenario , for few-shot learning . Basically , the few-shot learning task uses a classifier learned on the training set to recognize novel classes with very few support samples . In real-world applications , there is usually a domain gap between the training samples and the support samples Tseng et al . ( 2020 ) ; Guo et al . ( 2020 ) ; Chen et al . ( 2019 ) . The domain gap further imposes a critical challenge : since the support samples are very rare , they do not suffice for mitigating the domain gap between the support set ( for novel classes ) and the training set . Consequentially , the domain gap significantly compromises the recognition accuracy of the novel classes . Therefore , learning a model with strong cross-modality generalization capacity is important for cross-domain few-shot learning Tseng et al . ( 2020 ) . We argue that we may enhance the desired cross-modality generalization by “ learning-togeneralize ” . To be more specific , we hold a hypothesis / intuition that if a deep model learns to fast generalize itself to different domains ( using very few samples ) during training , it will maintain the good domain generalization capacity for testing . We model our intuition into a novel Domain-Switch Learning ( DSL ) framework , as illustrated in Fig . 1 . DSL uses multiple ( M > 1 ) domains to construct cross-domain training scenario in a “ fast switching ” manner . Instead of simply mixing all the domains to construct a mini-batch ( Fig . 1 ( a ) ) , DSL includes only a single domain into every training iteration and switches to another domain for the following iteration ( Fig . 1 ( b ) ) . Moreover , each iteration contains very few samples per category , so as to imitate the few-shot setting . Therefore , after every switch , the deep model crosses into a domain different from the former one , yielding the cross-domain few-shot scenario . An important advantage of this domain-switch learning manner is that it favors domain-general knowledge over domain-specific knowledge , as illustrated in Fig . 1 . Specifically , we consider each domain contains both domain-general and domain-specific knowledge . If we mix all the domains for each training iteration ( Fig . 1 ( a ) ) , the deep model may memorize all the domain-specific knowledge ( apart from the domain-general knowledge ) . Consequentially , during testing , the domain-specific knowledge hinders generalization towards the novel domain and thus compromises few-shot classification accuracy . In contrast , in DSL ( Fig . 1 ( b ) ) , the model only learns from one single domain in an iteration . Since the domain-specific knowledge of the current domain does not fit the following domain , the deep model should favor the domain-general knowledge and ignore the domain-specific knowledge , so as to prepare for the fast domain switching . However , the preference for domain-general knowledge ( over domain-specific knowledge ) does not guarantee that DSL learns more domain general knowledge than the domain-mix learning scheme does . To further promote cross-domain generalization in this domain-switch learning scheme , DSL enforces two constraints as follows : 1 ) The deep model should not over-fit the domain in the current iteration , because over-fitting the current domain makes the model memorize much of the domain-specific knowledge . The first constraint is implemented with a domain-specific prompter module ( consisted of multiple prompters ) . After the model learns from the i-th training domain Di and gets updated , its prediction accuracy on Di increases . We store this edition of model as the prompter for Di . Next time when the learning domain switches to Di again , we average the softmax prediction of the prompter and the learner as the final prediction . Since the prompter fits Di well , it reduces the prediction errors of the final prediction and thus attenuates the penalty on the learner . The details of the domain-specific prompter module are to be accessed in Section 3.3 . 2 ) The deep model should not forget the already-learned knowledge of former domains , so that next time the model crosses into these domains again , it can directly re-use the corresponding knowledge for prediction . The second constraint is implemented with a domain-general teacher module . Specifically , we collect several historical models and average their parameters to get a mean model . During training , we use the softmax prediction of the mean model as the auxiliary supervising signals for the learner ( apart from the ground truth label ) . The mean model serves as a teacher distilling the already-learned knowledge of former domains to the learner . Since this teacher has no obvious bias towards any single domain , we name it as a domain-general teacher . The details of the domaingeneral teacher module are to be accessed in Section 3.4 . These two constraints achieve complementary benefits for DSL , and jointly reinforce the crossdomain generalization . Extensive experiments under four cross-domain scenarios show that DSL consistently improves cross-domain few-shot learning and achieves performance on par with the state-of-the-art methods . Our main contributions are summarized as follows : •We propose a novel Domain-Switch Learning ( DSL ) framework for cross-domain few-shot learning . DSL uses multiple domains for training and switches the domain in consecutive training iterations . It provides a cross-domain learning scenario where the deep model learns to generalize across different domains . • Under the DSL framework , we integrate two modules , i.e. , the domain-specific prompter and the domain-general teacher . These two modules achieve complementary benefits for DSL and jointly reinforce the cross-domain generalization . •We conduct extensive experiments to validate the effectiveness of the proposed DSL . Experimental results confirm that the fast generalization capacity can be inherited from training to testing and thus improves cross-domain few-shot classification . On all the four popular benchmarks , DSL achieves performance on par with the state of the art . 2 RELATED WORKS . Few-Shot Learning . Few shot learning methods could be roughly categorized into two branches : meta-learning and fine-tuning methods . Some meta-learningSatorras & Estrach ( 2018 ) ; Mishra et al . ( 2018 ) ; Ren et al . ( 2018 ) ; Snell et al . ( 2017 ) ; Sung et al . ( 2018 ) ; Vinyals et al . ( 2016 ) methods map the few-shot samples into a non-linear embedding space and evaluate the similarity between support samples and the query samples . Recently , fine-tuning methods Chen et al . ( 2019 ) ; Tian et al . ( 2020 ) ; Afrasiyabi et al . ( 2020 ) propose to pre-train the model on the training set ( typically in a classification learning approach ) , and fine-tune the classifier on the novel classes . The finetuning methods achieve competitive performance compared with the state-of-the-art meta-learning methods . This work considers the few-shot learning task under the cross-domain scenario and adopts the fine-tuning pipeline as the baseline . Domain Generalization and Adaptation may be viewed as two closely related concepts . Domain adaptation uses abundant ( unlabeled ) samples to adapt the already-learned model from the source domain to the target domain Ganin & Lempitsky ( 2015 ) ; Tzeng et al . ( 2017 ) ; Long et al . ( 2017 ; 2015 ; 2016 ) ; Tzeng et al . ( 2014 ) . In contrast , domain generalization aims to generalize the knowledge from multiple source domains to the target domain without learning any samples of the target domain during the training stage Balaji et al . ( 2018 ) ; Li et al . ( 2018 ) . In cross-domain fewshot learning , there is a domain gap between the training set and the testing set . To promote the cross-domain generalization , Tseng et al . ( 2020 ) considers the challenge is more of a domain generalization problem than a domain adaptation problem . This consideration is reasonable because the support samples are very rare and do thus not suffice for a typical domain adaptation . From the viewpoint of domain generalization , FWT Tseng et al . ( 2020 ) proposes to simulate the cross-domain scenario to improve the domain generalization capacity . It indeed inspires us of the multi-domain training strategy . In spite of sharing the multi-domain training strategy , DSL significantly differs from FWT : First , FWT uses changes the training domain epoch-by-epoch . It does not rely on fast switching for better generalization . In contrast , DSL is featured for the novel fast switching manner and finds that this manner promotes learning domain-general knowledge . Second , DSL further considers two constraints in multi-domain training strategy , i.e. , the deep model should not over-fit every single domain and keep recognition on learned knowledge shared with other domains . Finally , DSL achieves superior cross-domain effect than FWT ( Section 4.2 ) . Catastrophic forgetting . Catastrophic forgetting plays an important role in the mechanism of DSL . When the deep model is trained sequentially on multiple tasks ( domains ) , there is a tendency that the knowledge learned on former tasks ( domains ) is abruptly lost , yielding the so-called catastrophic forgetting Kirkpatrick et al . ( 2017 ) ; McCloskey & Cohen ( 1989 ) ; French ( 1999 ) . In DSL , when the deep model switches from a domain to another , it forgets much of the knowledge on the former domain , due to catastrophic forgetting . Forgetting the domain-specific knowledge is beneficial , while forgetting the domain-general knowledge is harmful . In response , next time the deep model comes across the former domain again , we 1 ) supplement it with the corresponding knowledge with a domain-specific prompter to prevent over-fitting the domain-specific knowledge and 2 ) use a domain-general teacher to reinforce the common knowledge shared among all the former domains . In a word , in its mechanism , DSL actually utilizes catastrophic forgetting to improve cross-domain generalization . 3 METHODS . 3.1 PRELIMINARIES . In the cross-domain few-shot learning task , there is a significant domain gap between the training set and the testing set . The testing set consists the support set and query set , which are sampled from the same novel categories ( which are unseen in the training set ) . The support set provides very few labeled samples for recognizing the unlabeled samples in the query . Concretely , the support set contains C classes and each class has K samples , which is a C-way K-shot setup in few-shot learning . Our work is based on a popular fine-tuning pipeline Chen et al . ( 2019 ) ; Tian et al . ( 2020 ) ; Afrasiyabi et al . ( 2020 ) . The fine-tuning baseline uses the training set D to learn a deep model consisted of a feature encoder F and a feature classifier Y . F is parameterized with θ and Y is parameterized with a weight matrix W . During training , given a sample x , the feature encoder first maps x into a feature F ( x ; θ ) ∈ Rd . Then the feature classifier Y predicts the softmax probability of x belonging to each training category as P ( i|x ; θ , W ) = exp ( w T i F ( x ; θ ) ) ∑ j exp ( w T j F ( x ; θ ) ) ( wi is the weight vector for the i-th category ) . The training is supervised with a popular classification loss function , i.e. , the cross-entropy loss , which is formulated as : LCE ( x ) = − log ( P ( y|x ; θ , W ) ) = − log exp ( wTy F ( x ; θ ) ) ∑ j exp ( w T j F ( x ; θ ) ) , ( 1 ) where y is the label for x , wy is the weight vector corresponding to the ground-truth category . During the testing stage , we replace the feature classifier Y with a new linear classifier L. L is a C-way classifier , corresponding to the total number of classes in the support set . We freeze the parameters of the feature encoder F and fine-tune the parameters of Lwith samples from the support set . Finally , we use the feature encoder F and the learned L as the predictor for the query . This paper focuses on the training stage , i.e. , learning the feature encoder F and the classifier Y on the training set .

Summary: This paper proposes a novel training scheme termed Domain Switch Learning (DSL) to pursue the few-shot learning problem under the cross-domain scenario. In practice, DSL uses the data from one domain for a training iteration and switches to another domain for the next iteration. During the switching, two regularization terms are also introduced to balance the domain-specific knowledge and domain-general knowledge. The extensive experimental result demonstrates the effectiveness of the proposed method on several standard benchmarks.