Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators

1 INTRODUCTION . Few-shot learning in the perspective of meta-learning aims to train models which can quickly solve novel tasks or adapt to new environments with limited number of examples . In case of few-shot classification , models are usually evaluated on a held-out dataset which does not have any common class with the training dataset . In the real world , however , we often face harder problems in which novel tasks arise arbitrarily from many different domains even including previously unseen ones . In this study , we propose a more practical few-shot classification algorithm to generalize across domains beyond the common assumption , i.e. , meta-training and meta-testing within a single domain . Our approach to cover a complex multi-domain task distribution is to construct a pool of multiple models and learn to select the best one given a novel task through meta-training over various domains . This recasts task-specific adaption across domains as a simple selection problem , which could be much easier than manipulating high-dimensional parameters or representations of a single model to adapt to a novel task . Furthermore , we enforce all models to share some of the parameters and train per-model modulators with model-specific parameters on top of that . By doing so , each model could keep important domain-invariant features while the model pool has representational diversity as a whole without a significant increase of model parameters . We train and test our algorithms on various image classification datasets with different characteristics . Experimental results show that the proposed selection scheme outperforms other state-of-theart algorithms in few-shot classification tasks from many different domains without being given any knowledge of the domain which the task belongs to . We also show that even few-shot classification tasks from previously unseen domains , i.e. , domains which have never appeared during meta-training , can be done successfully by averaging outputs of all models . 2 METHODS . 2.1 PROBLEM STATEMENT . We follow the common setting of few-shot classification in the meta-learning community ( Vinyals et al. , 2016 ) . For a N -way , K-shot classification task , an episode which consists of a support set S = { ( xsi , ysi ) } NKi=1 and a query set Q = { ( x q i , y q i ) } Ti=1 is sampled from a given dataset , where xsi , x q i , y s i and y q i represent examples and their correct labels respectively and T is the number of query examples . Once a model has been trained with respect to a number of random episodes at meta-training time , it is expected to predict a correct label for an unlabeled query given only a few labeled examples ( i.e. , support set ) even if all these came from classes which have never appeared during meta-training . Based on this setting , we try to build a domain-agnostic meta-learner beyond the common metalearning assumptions , i.e. , meta-training within a single domain and meta-testing within the same domain . We perform meta-training over multiple diverse domains , which we call source domains DS1 , DS2 , · · · , DSM , where M is the number of source domains , expecting to obtain a domaingeneralized meta-learner . Since we presume that one particular dataset defines its own domain , we realize this idea by training this meta-learner on various tasks from many different datasets . In our study , the trained meta-learner is meta-tested on a target domain DT for two types of crossdomain few-shot classification tasks . One is a task which is required to classify from held-out classes of multiple source domains ( i.e. , DT ∈ { DS1 , DS2 , · · · , DSM } ) without knowing from which dataset each task is sampled . This could be used to evaluate whether the meta-learner is capable of adapting to a complex task distribution across multiple domains . We also tackle a task sampled from previously unseen datasets during the meta-training ( i.e. , DSi ∩ DT = ∅ for all i ) , which requires to generalize over out-of-distribution tasks in domain-level . 2.2 BUILDING A POOL OF EMBEDDING MODELS WITH DISPARATE MODULATORS . Basically , we perform metric-based meta-learning to learn a good metric space in which the support and query examples from the same class are located closely over various domains . While recent meta-learning methods have been proposed to train a single model commonly applicable to various potential tasks and to learn to adjust the model to a particular task for further improvement ( Rusu et al. , 2019 ; Oreshkin et al. , 2018 ; Ye et al. , 2018 ; Triantafillou et al. , 2019 ) , we train a pool of multiple embedding models each of which defines a different metric space and a meta-model to select the best model from them given a novel task . This makes task-specific adaptation easier and more effective to learn because our approach is required to solve only a simple classification problem , i.e. , choose one of all pre-trained models , instead of learning to manipulate high-dimensional model components , such as model parameters or activations , directly to adapt to novel tasks from various domains . Rather than training each model separately , we take an approach that all models share a considerable amount of parameters and they are differentiated by adding per-model modulators as done usually in multi-task learning ( Ruder , 2017 ) . The rationale behind this is to let our model pool capture good domain-invariant features by the shared parameters as well as have diversity , which is desirable to represent the complex cross-domain task distribution , without a significant increase of the number of parameters . To realize this idea , we first build a base network fE ( · ; θ ) shared among all models . One large virtual dataset is constructed by aggregating all source datasets . The base network is trained on this dataset following typical supervised learning procedure ( Figure 1 ( a ) ) . In the next step , we build one model per source domain by adding a per-model modulator with a parameter set αi on top of the frozen base network . We then train each modulator on one dataset DSi by performing metric-based metalearning in the same way as the Prototypical Networks ( ProtoNet ) ( Snell et al. , 2017 ) ( Figure 1 ( b ) ) . Finally , we have a pool of embedding models which are ready for non-parametric classification in the same way as ProtoNet . As shown in Figure 2 , we add the modulator components to the base network in a per-layer basis following the idea proposed in ( Rebuffi et al. , 2018 ) . This way has turned out to be more effective than the conventional approach , i.e. , adding a few extra layers for each model , for domain-specific representation . Moreover , this allows each modulated model to have the same computational cost at inference time as the base network ’ s because all modulator components can be fused into existing convolution 3×3 operations . We try two modulator architectures , convolution 1×1 and channel-wise transform ( i.e. , per-channel trainable scale and bias ) . The former shows slightly better performance whereas the latter uses much less parameters only incurring negligible memory overhead to the pool . More details of the architecture including the number of parameters can be found in Appendix B . 2.3 LEARNING TO SELECT THE BEST MODEL FOR A TARGET TASK . After the construction of the pool , we build a meta-model to predict the most suitable model from all constituent models in the pool for a given task as the final step of our training . By training this model over a number of episodes sampled randomly from all available source datasets , we expect this ability to be generalized to novel tasks from various domains including unseen ones during meta-training . As depicted in Figure 1 ( c ) , this meta-model parameterized by φ , which we call a model selection network , is trained in order to map a task representation ztask for a particular task to an index of the best model in the model pool . The task representation is obtained by passing all examples in the support set of the task through the base network and taking a mean of all resulting embedded vectors to form a fixed-length summary of the given task . During meta-training , the index of the best model , which is the ground truth label for training the selection network , is generated by measuring the classification accuracy values of all models in the pool given the query set and picking an index of one which has the highest accuracy . In our setup , the task-specific adaptation is reduced to a ( M+1 ) -way classification problem when we have M+1 embedding models including the base network learned from M available source datasets . Learning this classifier could be far simpler than learning to adapt model parameters ( Rusu et al. , 2019 ) , embedded vectors ( Ye et al. , 2018 ) or per-layer activations ( Oreshkin et al. , 2018 ) to a particular task because their dimensions are usually much larger than that of our selection network outputs , i.e. , the number of the pre-trained models . The overall training procedure for constructing the pool and training the selection network is summarized in Algorithm 1 in Appendix C . 2.4 INFERENCE WITH THE POOL AT META-TESTING TIME . One way of the inference is to predict a class with the best single model chosen by the selection network fS ( · ; φ ) for a given support set ( Figure 3 ( a ) ) . Following the method proposed in ProtoNet as shown in Equation 1 , a class prediction ŷ to a query example xq is made by finding a class whose prototype cy is the closest one to an embedded vector of the given query example . Specifically , the prototype for the class y is defined as a mean of embedded vectors of all examples in the support set belonging to a class y and squared Euclidean distances dyi to these prototypes are compared between the classes . ŷ = argmin y dyi ( x q ) = argmin y ‖cy − fE ( xq ; θ , αi ) ‖2 . ( 1 ) Another way to benefit from the model pool is to combine outputs from all available embedding models for inference ( Figure 3 ( b ) ) . We take the simplest approach to average outputs of all models in the level of class prediction probability . As described in Equation 2 , we collect output probabilities p ( y | xq ; i ) based on the relative distances to class prototypes dyi for a given task . Then , our final prediction ŷ would be a class to maximize a mean of these probabilities from all M + 1 models . ŷ = argmax y 1 M + 1 M∑ i=0 p ( y | xq ; i ) = argmax y 1 M + 1 M∑ i=0 softmax ( −dyi ( x q ) ) ( 2 ) As the last step , we adopt test-time ‘ further adaptation ’ proposed in ( Chen et al. , 2019 ) , which turned out to make additional performance improvement in most cases . Both experimental results with and without the further adaptation are presented in Section 3 and Appendix D with its implementation details in Appendix C . 3 EXPERIMENTS . 3.1 SETUP . 3.1.1 DATASETS . We use eight image classification datasets , denoted as Aircraft , CIFAR100 , DTD , GTSRB , ImageNet12 , Omniglot , SVHN , UCF101 and Flowers , introduced in the Visual Decathlon Challenge ( Rebuffi et al. , 2017 ) for evaluation . These are considered as eight different domains in our experiments . All datasets are resplit for the few-shot classification , i.e. , no overlap of classes between meta-training and meta-testing . More details about the datatsets can be found in Appendix A .

Summary: In this paper, the authors proposed to address the few-shot learning problem, especially for the cross-domain setting where a newly coming task originates from a different distribution (or in this work implemented by sampling from an unseen dataset). Basically, the authors constructed a model zoo based on source datasets at hand, and learned an “argmax” meta-selector which takes embedding of a task as input and outputs the model selection index. The idea is very intuitive, and the implementation is kind of an incremental combination of (Rebuffi et al., 2018) building models for multi-tasks and (Oreshkin et al., 2018) tailoring based on task embeddings.