Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators

1 INTRODUCTION . Few-shot learning in the perspective of meta-learning aims to train models which can quickly solve novel tasks or adapt to new environments with limited number of examples . In case of few-shot classification , models are usually evaluated on a held-out dataset which does not have any common class with the training dataset . In the real world , however , we often face harder problems in which novel tasks arise arbitrarily from many different domains even including previously unseen ones . In this study , we propose a more practical few-shot classification algorithm to generalize across domains beyond the common assumption , i.e. , meta-training and meta-testing within a single domain . Our approach to cover a complex multi-domain task distribution is to construct a pool of multiple models and learn to select the best one given a novel task through meta-training over various domains . This recasts task-specific adaption across domains as a simple selection problem , which could be much easier than manipulating high-dimensional parameters or representations of a single model to adapt to a novel task . Furthermore , we enforce all models to share some of the parameters and train per-model modulators with model-specific parameters on top of that . By doing so , each model could keep important domain-invariant features while the model pool has representational diversity as a whole without a significant increase of model parameters . We train and test our algorithms on various image classification datasets with different characteristics . Experimental results show that the proposed selection scheme outperforms other state-of-theart algorithms in few-shot classification tasks from many different domains without being given any knowledge of the domain which the task belongs to . We also show that even few-shot classification tasks from previously unseen domains , i.e. , domains which have never appeared during meta-training , can be done successfully by averaging outputs of all models . 2 METHODS . 2.1 PROBLEM STATEMENT . We follow the common setting of few-shot classification in the meta-learning community ( Vinyals et al. , 2016 ) . For a N -way , K-shot classification task , an episode which consists of a support set S = { ( xsi , ysi ) } NKi=1 and a query set Q = { ( x q i , y q i ) } Ti=1 is sampled from a given dataset , where xsi , x q i , y s i and y q i represent examples and their correct labels respectively and T is the number of query examples . Once a model has been trained with respect to a number of random episodes at meta-training time , it is expected to predict a correct label for an unlabeled query given only a few labeled examples ( i.e. , support set ) even if all these came from classes which have never appeared during meta-training . Based on this setting , we try to build a domain-agnostic meta-learner beyond the common metalearning assumptions , i.e. , meta-training within a single domain and meta-testing within the same domain . We perform meta-training over multiple diverse domains , which we call source domains DS1 , DS2 , · · · , DSM , where M is the number of source domains , expecting to obtain a domaingeneralized meta-learner . Since we presume that one particular dataset defines its own domain , we realize this idea by training this meta-learner on various tasks from many different datasets . In our study , the trained meta-learner is meta-tested on a target domain DT for two types of crossdomain few-shot classification tasks . One is a task which is required to classify from held-out classes of multiple source domains ( i.e. , DT ∈ { DS1 , DS2 , · · · , DSM } ) without knowing from which dataset each task is sampled . This could be used to evaluate whether the meta-learner is capable of adapting to a complex task distribution across multiple domains . We also tackle a task sampled from previously unseen datasets during the meta-training ( i.e. , DSi ∩ DT = ∅ for all i ) , which requires to generalize over out-of-distribution tasks in domain-level . 2.2 BUILDING A POOL OF EMBEDDING MODELS WITH DISPARATE MODULATORS . Basically , we perform metric-based meta-learning to learn a good metric space in which the support and query examples from the same class are located closely over various domains . While recent meta-learning methods have been proposed to train a single model commonly applicable to various potential tasks and to learn to adjust the model to a particular task for further improvement ( Rusu et al. , 2019 ; Oreshkin et al. , 2018 ; Ye et al. , 2018 ; Triantafillou et al. , 2019 ) , we train a pool of multiple embedding models each of which defines a different metric space and a meta-model to select the best model from them given a novel task . This makes task-specific adaptation easier and more effective to learn because our approach is required to solve only a simple classification problem , i.e. , choose one of all pre-trained models , instead of learning to manipulate high-dimensional model components , such as model parameters or activations , directly to adapt to novel tasks from various domains . Rather than training each model separately , we take an approach that all models share a considerable amount of parameters and they are differentiated by adding per-model modulators as done usually in multi-task learning ( Ruder , 2017 ) . The rationale behind this is to let our model pool capture good domain-invariant features by the shared parameters as well as have diversity , which is desirable to represent the complex cross-domain task distribution , without a significant increase of the number of parameters . To realize this idea , we first build a base network fE ( · ; θ ) shared among all models . One large virtual dataset is constructed by aggregating all source datasets . The base network is trained on this dataset following typical supervised learning procedure ( Figure 1 ( a ) ) . In the next step , we build one model per source domain by adding a per-model modulator with a parameter set αi on top of the frozen base network . We then train each modulator on one dataset DSi by performing metric-based metalearning in the same way as the Prototypical Networks ( ProtoNet ) ( Snell et al. , 2017 ) ( Figure 1 ( b ) ) . Finally , we have a pool of embedding models which are ready for non-parametric classification in the same way as ProtoNet . As shown in Figure 2 , we add the modulator components to the base network in a per-layer basis following the idea proposed in ( Rebuffi et al. , 2018 ) . This way has turned out to be more effective than the conventional approach , i.e. , adding a few extra layers for each model , for domain-specific representation . Moreover , this allows each modulated model to have the same computational cost at inference time as the base network ’ s because all modulator components can be fused into existing convolution 3×3 operations . We try two modulator architectures , convolution 1×1 and channel-wise transform ( i.e. , per-channel trainable scale and bias ) . The former shows slightly better performance whereas the latter uses much less parameters only incurring negligible memory overhead to the pool . More details of the architecture including the number of parameters can be found in Appendix B . 2.3 LEARNING TO SELECT THE BEST MODEL FOR A TARGET TASK . After the construction of the pool , we build a meta-model to predict the most suitable model from all constituent models in the pool for a given task as the final step of our training . By training this model over a number of episodes sampled randomly from all available source datasets , we expect this ability to be generalized to novel tasks from various domains including unseen ones during meta-training . As depicted in Figure 1 ( c ) , this meta-model parameterized by φ , which we call a model selection network , is trained in order to map a task representation ztask for a particular task to an index of the best model in the model pool . The task representation is obtained by passing all examples in the support set of the task through the base network and taking a mean of all resulting embedded vectors to form a fixed-length summary of the given task . During meta-training , the index of the best model , which is the ground truth label for training the selection network , is generated by measuring the classification accuracy values of all models in the pool given the query set and picking an index of one which has the highest accuracy . In our setup , the task-specific adaptation is reduced to a ( M+1 ) -way classification problem when we have M+1 embedding models including the base network learned from M available source datasets . Learning this classifier could be far simpler than learning to adapt model parameters ( Rusu et al. , 2019 ) , embedded vectors ( Ye et al. , 2018 ) or per-layer activations ( Oreshkin et al. , 2018 ) to a particular task because their dimensions are usually much larger than that of our selection network outputs , i.e. , the number of the pre-trained models . The overall training procedure for constructing the pool and training the selection network is summarized in Algorithm 1 in Appendix C . 2.4 INFERENCE WITH THE POOL AT META-TESTING TIME . One way of the inference is to predict a class with the best single model chosen by the selection network fS ( · ; φ ) for a given support set ( Figure 3 ( a ) ) . Following the method proposed in ProtoNet as shown in Equation 1 , a class prediction ŷ to a query example xq is made by finding a class whose prototype cy is the closest one to an embedded vector of the given query example . Specifically , the prototype for the class y is defined as a mean of embedded vectors of all examples in the support set belonging to a class y and squared Euclidean distances dyi to these prototypes are compared between the classes . ŷ = argmin y dyi ( x q ) = argmin y ‖cy − fE ( xq ; θ , αi ) ‖2 . ( 1 ) Another way to benefit from the model pool is to combine outputs from all available embedding models for inference ( Figure 3 ( b ) ) . We take the simplest approach to average outputs of all models in the level of class prediction probability . As described in Equation 2 , we collect output probabilities p ( y | xq ; i ) based on the relative distances to class prototypes dyi for a given task . Then , our final prediction ŷ would be a class to maximize a mean of these probabilities from all M + 1 models . ŷ = argmax y 1 M + 1 M∑ i=0 p ( y | xq ; i ) = argmax y 1 M + 1 M∑ i=0 softmax ( −dyi ( x q ) ) ( 2 ) As the last step , we adopt test-time ‘ further adaptation ’ proposed in ( Chen et al. , 2019 ) , which turned out to make additional performance improvement in most cases . Both experimental results with and without the further adaptation are presented in Section 3 and Appendix D with its implementation details in Appendix C . 3 EXPERIMENTS . 3.1 SETUP . 3.1.1 DATASETS . We use eight image classification datasets , denoted as Aircraft , CIFAR100 , DTD , GTSRB , ImageNet12 , Omniglot , SVHN , UCF101 and Flowers , introduced in the Visual Decathlon Challenge ( Rebuffi et al. , 2017 ) for evaluation . These are considered as eight different domains in our experiments . All datasets are resplit for the few-shot classification , i.e. , no overlap of classes between meta-training and meta-testing . More details about the datatsets can be found in Appendix A .

Summary: This paper aims to tackle few-shot classification with many different domains. The idea is to build a pool of embedding models, which are based on the same base network. The models are diversed by their own modulators. The high-level intuition is to let the model pool capture good domain-invariant features by the shared parameters and domain-specific features by the selection network, which is desirable to represent the complex cross-domain task distribution, without a significant increase in the number of parameters.