Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators
Authors: Yongseok Choi
Junyoung Park
Subin Yi
Dong-Yeon Cho
Venue:  (2020)
Keywords: Meta-learning
few-shot learning
multi-domain

Abstract: Although few-shot learning research has advanced rapidly with the help of meta-learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from a single domain. We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.
The key idea is to build a pool of embedding models which have their own metric spaces and to learn to select the best one for a particular task through multi-domain meta-learning. This simplifies task-specific adaptation over a complex task distribution as a simple selection problem rather than modifying the model with a number of parameters at meta-testing time. Inspired by common multi-task learning techniques, we let all models in the pool share a base network and add a separate modulator to each model to refine the base network in its own way. This architecture allows the pool to maintain representational diversity and each model to have domain-invariant representation as well. 
Experiments show that our selection scheme outperforms other few-shot classification algorithms when target tasks could come from many different domains. They also reveal that aggregating outputs from all constituent models is effective for tasks from unseen domains showing the effectiveness of our framework.

Under review as a conference paper at ICLR 2020
FEW-SHOT CLASSIFICATION BY LEARNING DISPARATE MODULATORS
Anonymous authors Paper under double-blind review
ABSTRACT
Although few-shot learning research has advanced rapidly with the help of meta- learning, its practical usefulness is still limited because most of the researches assumed that all meta-training and meta-testing examples came from single do- main. We propose a simple but effective way for few-shot classification in which a task distribution spans multiple domains including previously unseen ones during meta-training.
The key idea is to build pool of models, each of which is responsible for a different group of tasks, and to learn to select the best one for particular task through multi-domain meta-leamingg This reduces task-specific adaptation over a complex task distribution to a simple selection problem rather than modifying the model with number of parameters at meta-testing time. Inspired by common multi-task learning techniques, we let all models the pool share a base network and add separate modulator to each model to refine the base network in its own way. This architecture allows the pool to maintain representational diversity and each model to have domain-invarte
representation well.
Experiments show that our selection scheme outperforms other few-shot classifi- cation algorithms when target tasks come from many different domains. They also reveal that aggregating outputs from all constituent models effective for tasks from unseen domains indicating the effectiveness of our framework.
INTRODUCTION
Few-shot learning in the perspective of meta-learning aims train models which can quickly solve novel tasks or adapt to new environments with limited number of examples. In case of few-shot held-out dataset which does not have any common
classification, models are usually evaluated class with the training dataset. In the real world, however, we often face harder problems in which novel tasks arise arbitrarily from many different domains even including previously unseen ones.
In this study, we propose a more practical few-shot classification algorithm to generalize across do- mains beyond the common assumption, i.e., meta-training and meta-testing within single domain. Our approach to cover a complex multi-domain task distribution is to construct a pool of multi- ple models and learn to select the best one given novel task through meta-training over various domains. This recasts task-specific adaption across domains as a simple selection problem, which could be much easier than manipulating high- nigh-dimensional parameters or representations of single model to adapt to novel task.
Furthermore, we enforce all models share some of the parameters and train per-mode modula- tors with model-specific parameters top of that. By doing so, each model could keep important domain-invariant features while the model pool has representational diversity as a whole without significant increase of model parameters.
We train and test our algorithms on various image classification datasets with different characteris- tics. Experimental results show that the proposed selection scheme outperforms other state-of-the- art algorithms in few-shot classification tasks from many different domains without being given any knowledge of the domain which the task belongs to. We also show that even few-shot clas- sification tasks from previously unseen domains, ie.. domains which have never appeared during meta-train meta-training, be done successfully by averaging outputs of all models.

---

Under review as a conference paper at ICLR 2020
(a)
(b)
Model pool
Modulators
(x/),
0y/,
O1
(x/h) (xg9)
6,y1, byy"
Base network
Org
0g
(c)
f
fs
1xy_
0
Mean
2
tank
Softmax
Selection network
(18)
Cross entropy
Model pool
Compare
Accuracy, 6
[x, (8,.")
0,99
Compare
Accuracy ? Accurucy,
Best
Compure
Angmax
model
index
Compare
Accuracy
Figure 1: Training (a) a base network f=(:; 8,-)) (b) modulators fE(: a4), (c) a selection network (ts Φ). (1% an example, Vm: label, D/ prediction, zm an embedded vector of support (n=s) or query (n=q) set in domain m.)
2 METHODS
2.1 PROBLEM STATEMENT
We follow the common setting of few-shot classification in the meta-learning community (Vinyals al. 2016). For N-way, K-shot classification task, an episode which consists of support set (o ) NN and a query set Q = { [x ) is sampled from a given dataset, where ay x.. Vi and y/ represent examples and their correct labels respectively and T' is the number of query examples. Once model has been trained with respect to number of random episodes at meta-training time, it is expected predict a correct label for an unlabeled query given only a few labeled examples (i.e.. support set) even if all these came from classes which have never appeared during meta-training.
Based on this setting, we to build domain-agnostic meta-leamne beyond the common meta- learning assumptions, i.e., meta-training within single domain and meta-testing within the same domain. We perform meta-training over multiple diverse domains, which we call source domains number of source domains, expecting to obtain a domain- generalized meta-learner. Since we presume that one particular dataset defines its own domain, we realize this idea by training this meta- on various tasks from many different datasets.
Ds, Dsa:" DSar where
In our study, the trained meta-learner is meta-tested on target domain Dr for two types of cross- domain shw-shot classification tasks. One is a task which is required to classify from held-out classes of multiple source domains (i.e., Dr {Ds, Ds.; ... D. }) without knowing from which dataset each task is sampled. This could be used to evaluate whether the meta-learner is capable of adapting complex task distribution across multiple domains. We also tackle task sampled from previously unseen datasets during the meta-training (i.e., Ds; n Dr 0 for all i), which requires to generalize over -ut-of-distribution tasks in domain-level.
2.2 BUILDING A POOL OF EMBEDDING MODELS WITH DISPARATE MODULATORS
Basically, perform metric-based meta-learning to leam good metric space in which the support and query examples from the same class are located closely over various domains. While recent meta-learning methods have been proposed train single model commonly applicable to various
2

---

Under review as a conference paper at ICLR 2020
Chennel-wise transfom
512
ReLU
Macro
block
258
Moduletor
Macro
block
BN
128
3x3
Cony
Macro block
Residual
block
ReLU
ReLU
Macro block
Residual
BN
Modulator
block
64
BN
Cony 3x3
Cony 3x3
Modulator
Cony
Figure 2: Architecture of one embedding model fs(:: 8, a;).
potential tasks and to learn to adjust the model to a particular task for further improvement (Rusu al. 2019 Oreshkin et al. 2018, Ye et 2018 Triantafillou et al., 2019). we train a pool of mul- tiple embedding models each of which defines different metric space and meta- to select best model from them given a novel task. This makes task-specific adaptation easier and more effective to learn because our approach is required to solve only simple classification problem, ie., choose one of all pre-trained models, instead learning to manipulate gigh-dimensiona model components, such as model parameters activations, directly to adapt to novel tasks from various domains.
Rather than training each model separately, take approach that all models share considerable amount of parameters and they are differentiated by adding per-mode modulators as done usually in multi-task learning (Ruder (0177). The rationale behind this is to let model pool capture good domain-invarianl features by the shared parameters as well as have diversity, which is desirable to represent the complex cross-domain task distribution, without significant increase of the number of parameters.
To realize this idea, we first build base network fE(; shared among all models. One large virtual dataset is constructed by aggregating all source datasets. The base network is trained on this dataset following typical supervised learning procedure (Figure aa)). In the next step, we build one model source domain by adding per-model modulator with a parameter set o; on top of the frozen base network. We then train each modulator on one dataset Dss, by performing metric-based meta- learning in the same way as the Prototypical Networks (ProtoNet) (Snell et al. 2017) (Figure [b))- Finally, we have pool of embedding models which are ready for non-paran non-parametric classification in the same way as ProtoNet.
As shown Figure 2, we add the modulator components to the base network a per-layer basis following the idea proposed in (Rebuffi et al., 2018). This way has turned out to be more effective than the conventional approach. i.c., adding a few extra layers for each model, for domain-specine representation. Moreover, this allows each modulated model to have the same computational cost at inference time as the base network's because all modulator components can be fused into existing convolution 3x3 operations. We try two modulator architectures, convolution 1x and channel-wise transform (i.e., per-channel trainable scale and bias). The former shows slightly better performance whereas the latter uses much less parameters only incurring negligible memory overhead the pool. More details of the architecture including the number of parameters can be found in Appendix B
2.3 LEARNING SELECT THE BEST MODEL FOR TARGET TASK
After the construction of the pool, we build meta-model to predict the most suitable model from all constituent models in the pool for a given task as the final step of our training. By training this model over number of episodes sampled randomly from all available source datasets, expect this ability to be generalized to novel tasks from various domains including unseen ones during meta-training.
3

---

Under review as a conference paper at ICLR 2020
As depicted in Figure Ic), this meta-model parameterized by Ø, which we call model selection network, is trained in order to map a task representation Stask for a particular task to an index of the best model in the model pool. The task representation is obtained by passing all examples in the support set of the task through the base network and taking a mean of all resulting embedded vectors to form fixed-length summary of the given task. During meta-training, the index of the best model, which is the ground truth label for training the selection network, is generated by measuring the classification accuracy values of all models in the pool given the query set and picking an index of one which has the highest accuracy.
In our setup, the specifi adaptation is reduced (M+1)-way classification problem when we have M+1 embedding models including the base network learned from M available source datasets. Learning this classifier could far simpler than learning to adapt model parameters (Rusu et al. 2019), embedded vectors YEe et al.) 2018) or per-layer activations (Oreshkin et al. (0188) to particular task because their dimensions are usually much larger than that of our selection network outputs, i.e., the number of the pre-trained models.
The overall training procedure for constructing the pool and training the selection network is sum- marized in Algorithm|] in Appendix C
2.4
INFERENCE WITH THE POOL AT META-TESTING TIME
(b)
fs
Model pool
Model pool
fE
0
Mean
CCg
Figure 3: Inference (a) with one model chosen by the selection network fs(;i) (b) by averaging output probabilities from all constituent models in the pool.
One way of the inference is to predict a class with the best single model chosen by the selection network fs(:: Φ) for given support set (Figure (a)) Following the method proposed in ProtoNet as shown in Equation a class prediction ú to a query example x is made by finding a class whose prototype Cy the closest one to embedded vector of the given query example. Specifically, the prototype for the class is defined as a mean of embedded vectors all examples in the support set belonging a class y and squared Euclidean distances de to these prototypes are compared between the classes.
argmin de (x9) = argmin ||c, Sn(=";
(I)
Another way to benefit from the model pool is to combine outputs from all available embedding models for inference (Figure b)). We take the simplest approach to average outputs of all models in the level of class prediction probability. As described in Equation[] we collect output probabilities ply ( based on the relative distances class prototypes de for given task. Then, our final prediction ý would be a class to maximize mean of these probabilities from all M + 1 models.
M Σ
M
i = argmax
puu 12 = argmax y M
fod
M+1
1
As the last step, we adopt test-time 'further adaptation' proposed in (Chen et al. 2019). which turned out make additional performance improvement in most cases. Both experimental results with and without the further adaptation are presented in Section 3 and Appendix D with its implementation details in Appendix
4

---

Under review as a conference paper at ICLR 2020
3 EXPERIMENTS
3.1 SETUP
3.1.1 DATASETS
We use eight image classification datasets, denoted as Aircraft, CIFARIOO, DTD, GTSRB, Ima- geNet12, Omniglot, SVHN, UCF101 and Flowers, introduced in the Visual Decathlon Challenge (Rebuth et al., 2017) for evaluation. These are considered as eight different domains in our exper- iments. All datasets resplit for the few-shot classification, i.e., overlap of classes between meta-training and meta-testing. More details about the datatsets can be found in Appendix A
3.1.2 ALGORITHMS
We denote our methods using the model picked by the selection network as DoS (Domain- generalized method by Selection) and DoS-Ch, which modulate the base network with convolution 1x 1 and channel-wise transform respectively. We also explore our averaging-baser methods, DoA Domain--eneralinn method by Averaging) and DoA-Ch, which generate an output by averaging class prediction probabilities of all constituent models modulated by the proposed two types of mod- ulators.
Our algorithms are compared with Fine-tune, Simple-Avg, ProtoNet (Snell et al. 2017). FEAT YYe al., (2188) and ProtoMAML (Triantafillou et al.) (019). Fine-tune a baseline method, which adds a linear classifier on top of the pre-trained base network and fine-tune it with the support set examples for 100 epochs meta-testing. In Simple-Avg, we train an embedding model independently on each source domain without sharing any parameters and perform inference by averaging class prediction probabilities of all these models. FEAT and ProtoMAML are the state-of-the-ar algorithms focusing on single domain and cross-domain setups respectively. TADAM (Oreshkin et al. 2018 was also tested but excluded from the results because its training did not converge in our setup. All these algorithms are tested by own implementations.
3.1.3 TRAINING AND TESTING
We pre-train all models of the compared algorithms with our base network since the pre-training has shown additional performance gain empirically. Then, we meta-train these models in an algorithm- specific way randomly generated episodes except the Fine-tune, the non-episodic baseline. At each episode, a target domain is chosen randomly from source domains, then a target task is sampled from that domain. The trained models are tested for 5-way classification tasks with 1-shot and 5-shot The configurations, The test results are averaged over 600 test episodes with 10 queries per class. Other details about the training are explained in Appendix C
3.2 RESULTS
3.2.1
FEW-SHOT CLASSIFICATION ON SEEN DOMAINS
We evaluate the above-mentioned algorithms in a multi-domain test setup constructed using all available datasets. Specifically, we meta-train a model for each algorithm on all available eight datasets. Then, we meta-test the trained model for various tasks sampled from these eight datasets without knowing which dataset the task comes from. F depicts test accuracy for each target dataset and the average accuracy over the eight datasets.
Our selection methods, DoS and DoS-Ch, outperform other few-shot classification methods in most cases. Two sfate-of-the art algorithms, FEAT and ProtoMAMI, not seem as effective as ours under this complex task distribution across domains. ProtoMAML shows comparable or better results in some cases, but much inferior results in other cases. ProtoNet seems relatively stable, but does not produce better results in any case. FEAT works worse than these two algorithms in most cases.
Although our averaging methods, DoA and DoA-Ch, seem competitive in many cases, they are out- performed by the selection methods always. This implies that the learned selection network is work- ing properly, which is highly likely to select the model with the modulator trained on the same domain the given task even without any information about the domain testing time. Another
5

---

Under review as a conference paper at ICLR 2020
Fine-tune
Protonet
FEAT
ProL
Simple-Avg
DoS
DoS-Ch
DoA
DoA-Ch
505
459
705
935
60%
5090
479
407
306
55%
30%
302
436
Airmma
CIPARIOX 100
UTD
Airera
CIFAR
DTD
90%
-075
3096
60%
2950
10%
70%
45%
407
IraagSNa122
Ohnnightt
GTSRB
InoageNet12
Quusigltt
709
1002
475
10%
60%g
2016
55%
755
19%
LCPI11
Flowers
Averago
UCPIOL
Flowets
Aveeaaee
(a) 5-way 1-shot
(h) 5-way 5-shot
Figure 4: Test accuracy various seen domains. (Accuracy values shown in Table AS
Fine tune
Protonet
FEAT
ProtoMAML
Simple-Avg
DoS
DoS Ch
DoA
DoA-Ch
259
439
45%
36%
90A 905
XHA
54%
495
25%
2395
159
329
45%
475
Airmvi
CIFARI00
DTTD
Aimrnc
CIEARIO)
DTD
9009
1079
2995
80%
15%
59%
271-
10%
80%
XVS
60%
75%
Iragghhe112
Ohnnightt
Imagehet12
Olnniglot
40%6
70%
73%
60%
55%
7002
55%
306
456
NS
30%
60%
50%
1CCF101
Flowers
Average
UCP101
Hoowers
Average
(a) 5-way 1-shot
(b) 5-way 5-shot
Figure 5: Test accuracy on various unseen domains. (Accuracy values are shown in Table A6
implication is that the best single model might be better than the averaging approach if a model from the same domain exists. is also worth noting that DoS-Ch is quite competitive despite much less number of parameters than DoS.
3.2.2 FEW-SHOT CLASSIFICATION ON UNSEEN DOMAINS
We also report the results on unseen domains in Figure 5 Given a target dataset for test, we train all models on other seven datasets. Therefore, end up with eight different models for each algorithm because there exist eight different combinations of the source datasets.
Our methods still outperform other algorithms in this more challenging setting. This reveals that our approach can be generalized to novel domains as well. Differently from the seen domain cases, averaging methods, DoA and DoA-Ch. perform better than all other algorithms including selection methods. It seems to make sense since the averaging could induce natural synergy be- tween beneficial models even if we do not know which models good for given task. However,

---

Under review as a conference paper at ICLR 2020
averaging methods significantly outperform Simple-Arg, which implies that way of the pool construction to encourage keeping domain-invariant invariant features is another key factor to the high per- formance of our averaging methods.
3.2.3 FEW-SHOT CLASSIFICATION ON VARYING NUMBER OF SOURCE DOMAINS
We conduct experiments with varying number of source datasets. Following the common real-world situation, we add from the largest dataset to the smallest one to our sources for meta-training. Tables 1 and 2 show the experimental results with 2, 4, and 6 source datasets on seen and unseen domains respectively.
Our selection and averaging methods outperform others consistently seen and unseen domains similarly to the previous results. Apart from comparing between the algorithms, observed over all algorithms that the added source often harms the performance. For example, the CIFARI00 tasks tend to work poorer as the number of source datasets increases in the seen domain case. This means that should pay more attention to avoiding negative transfer between heteroge- neous domains.
commonly
Table 1: Few-shot classification accuracy of varying number of sources on seen target domains.
METHODS
PROTONET
PROTOMAML
CH
DoA
C.I.
54. 72%
65.47% 55.88%
65.39%
69.51% 56.37%
73. 04% 57.71%
63%
71.19%
69.58%
57. 57.50%
52.53%
57.84%
59.39%
58.05%
AVERAGE
56. 11%
60.68%
58.96%
62.94%
65.38%
65. 65.24%
65.29%
63.82%
54.244%
58.31%
63.82%
68. 93%
71. 11.11%
70.15%
68. 14%
66.67%
C,G.T.O
G
81.21%
90.58%
95.40%
96.61%
96. 96.96%
96.36%
92. 94%
01.55%
50.70%
52.04%
50.58%
53.72%
55.78%
56. 10%
58. 04%
57.74%
94.82%
97.14%
95.10%
99. 99%
98.94%
98.73%
97 07.86%
97.64%
AVERAGE
71.19%%
74.52%
76.23%
79. 79.70%
80. 70%
80.33%
79.25%
78. 10%
40.68%
51.92%
62.77%
70.78%
61.94%
57.49%
47.77%
51.033%
58. 68%
56.56%
G0. 60.17%
69.5 19.5%1%
70 70.32%
66.48%
61%1
A.C.G.
76. 76.35%
89. 89.95%
95.64%
96.50%
97.71%
96. 90.78%
89.65%
89.62%
1.O.U
53. 53.37%
52.80%
48.62 48.62%
51. 74%
54.599%
56 03.0%
57.34%
03%
94.06% 0
96.94%
95.30%
99. 11%
98.89%
98. 98.76%
97.20%
97.92%
62.20%
67.11%
7U.23%
71.42%
AVERAGE
63.42%
69.30%
70.18%
74.29%
75.52%
74.94%
71.27%
71.54%
S: source datasets, T: target datasct
A: Aircraft, C: CIFARIOO, G: GTSRB, ImageNet12, O: Omniglot, U: UCF101, F: Flowers
Table Few-shot classification accuracy varying number of sources on unseen target domains.
METHODS
S
T
FINE-TUNE
PROTONET
FEAT
PROTOMAML
DoS
Dos-CH
DoA
DOA-CH
A
38, 63%
39.38%
36.71%
35.37%
38. 38.43%
39. 39.27%
39.14%%
40.57%
D
53.89%
35.27%
32.39%
51.04%%
54.80%
54.90%
57.15%
36.99%
G
70.01% %
81.37%
77.67%
81.25%
83.839
82.65%
80.13%
79.93%
O
92.72%
92.62%
90.61%
91.37%
93. 93.
93.31%
94.00%
94.34%
U
03.80%
63.25%
60.04%
59.890 89%
63.67%
G4. 64.81%
66.28%
67.15%
79.28%
81.77%
78.86%
80.50%
81.26%
80.841%
82,90 82.96%
82.85%
AVERAGE
66.39%
68.94%
66.05%
66.63%
69.18%
09.30%
69. 09.94%
70.33 70.31%
A
38, 12%
36.88%
35.11%
31.96%
37.90%
38. 0-1%%
39.66%
39.99%
C.G.L.O
D
55.04%
49. 96.96%
49.55%
49. 49.79%
55.85%
56.53%
56. 56.73%
55.67%
U
25%
56.53%
60.93%
61.12%%
61.32%
61.8/1%
67 67.58%
67.69%
79.88% F
76.79%
77.67%
81.40%
80.69%
79.54%
83. 45.25%
82.20%
AVERAGE
59.07%
55.04%
55.82%
57 37.57%
59.69%
50.74%
61. .81%%
61.39%
A.C.G
D
53.60%
51.52%
51.86%
50.94%
55.54%
55, 35.69%
56.71%
57.07%
LO.U
80.80%
80.58%
79.70%
79.40%
81.62%
81.66%
84.30%
84.50%
AVERAGE
07.20%
66.05%
65.78%
65. 17%
68.58%
68. 08.68%
70.51%
70%7%%
source datasets, target dataset
A: Aireraft, C: CIFARI00, GTSRB, ImageNet12, O: Omniglot, UCF101, F: Flowers
7

---

Under review as a conference paper at ICLR 2020
4 RELATED WORKS
Few-shot learning has been studied actively as an effective means for better understanding of human learning or as practical learning method only requiring a small number of training examples
(Lake et al., 2015 Li al. 2006). Meta-learning one of the most popular techniques to solve the
few-shot learning problems, which include learning a task iiss-invariantt metric space (Snell et al. [2117]
Vinyals et al.
2016), learning to optimize (Andrychowicz et al. 2016, Ravi & Larochelle| (017) or
learning good weight initialization (Finn et al. Nichol et al. 2018) for forthcoming tasks.
Follow-up studies showed that the metric- based meta-learning could be improved further by learning to modulate that metric space in a task- dusk-specific manner (Gidaris & Komodakis) 2018] Oreshkin et al.,
2018 Qiao et al. 2018: Ye et al. 2018). Similarly, it has been reported that the task-common initial parameters could d be refined for a given task producing task-specific initialization (Rusu et al. 2019
Vuorio et al. 2018 Yao et al.) 2019).
Recent few-shot learning studies have tried to tackle challenging problems under more realistic assumptions. Some studies dealt with few-shot learning under domain shift between training and testing (Kang & Feng, 2018, Wang & Hebert 2016). A more realistic evaluation method was pro- posed for few-shot learning to overcome limitations of the current popular benchmarks including the lack of domain divergence (Triantafillou al., 2019). One study performed an extensive and fair comparative analysis of well-known few-shot learning methods (Chen et al. 2019).
Our network architecture is inspired by the parameter sharing strategies for multi-task learning [Ruder] 2017) and multi-domain learning with domain-specific adaptation (Rebuffi et al., (018) because they have been known to lead to efficient parameterization and positive knowledge trans- fer between heterogeneous entities. Similar to our approach, a few suggestions combined multiple models to benefit from their diversity (Dvornik et al. 2019 Liu et al. 2019 Park et al., 2019). Our work also has something in common with the mixture of of-experts approach (Shazeer et al. (017]] in a sense that a part of large scale model would be executed conditionally benefiting from a large amount of the learned knowledge at low computational cost.
Our research is also related to domain adaptation or generalization (Ganin et al. 2016: Motiian cet al. 2017). However, most of the researches about these topics assume tasks with the same classes in both training and testing whereas our methods do not impose such limitations. Interestingly, some studies showed that the episodic training which is commonly adopted in many few-shot learning techniques, was also useful for domain generalization (Li et al. 2018 2019).
5 CONCLUSION AND FUTURE WORKS
We proposed a new few-shot classification method which is capable of dealing with many different domains including unseen domains. The core idea was to build a pool of embedding models, each of which diversified by its own modulator while sharing most of parameters with others, and to learn to select the best model for a target task through cross-domain meta-learning. The simpli- small classification problem made our dnlection-based
fication of the task-specific adaptation algorithm easy to learn, which in turn helped the learned model to work more effectively for multi- domain few-shot classification. The architecture with one shared model and disparate modulators encouraged our pool to maintain domain-invariant knowledge as well as cross-domain diversity. It helped our algorithms to generalize to heterogeneous domains including unseen ones even when we used one best model solely or all models collectively.
We believe that there still a large room for improvement in this challenging task. It would be one promising extension to find the optimal way to build the pool without the constraint on the number of models (i.e., one model per dataset) so that it can work even with a single source dataset with large diversity. Soft selection or weighted averaging can be also thought as one of future research directions because a single model or uniform averaging is less likely to be optimal. We can also consider a more scalable extension to allow continual expansion of the pool only by training a mod- ulator for an incoming source domain without re- re-training all existing models in the pool. Although the number of parameters does not increase much by virtue of the parameter sharing between mod- els, the computational cost in the averaging- based methods needs to be improved over the current linear increase with the number of models.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz. Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffmann, David Pfau, Tom Schaul, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems 29: An- nual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, Pp. 3981-3989, 2016. URL http::/paper: 6461- 6461-learning-to- learn by seradient-descee serprrt-----rrrreeeeedeeeeeeee
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look few-shot classification. In International Conference Learning Representations, 2019.
URL
net/fooumm forummiddileeeee......... id=HkXLXnACCO
M. Cimpoi, S. Maji, I. Kokkinos, Mohamed, and A. Vedaldi. Describing textures in the wild. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.
Nikita Dvomikk Cordelia Schmid, and Julien Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. CoRR, abs/1903. 11341, 2019. URL http: org/
1903. 114
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning, ICML 2017. Sydney, NSW, Australia, 6-11 August 2017, pp. 1126-1135, 2017. URL http:
//proceedings mlr preas / 7 finnl7a. html
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adverssrial training of neu-
ral networks. Journal of Machine Learning Research, 17:59:1-59:35, 2016. URL http:
jjmrr.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forget- ting. In 2018 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2018, doi: 10.1109/CVPR.
Salt Lake City, UT, USA, June 18-22, 2018, pp. 4367-4375. 2018.
2018.00459.
thecvf. com/ content _cvpr 2018/htme Gidaris Dynamic Few-Shot- Visual CVPR 2018 wwpppaapppaaa.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778, 2016. doi: 19.100//VP..2006..90 URI. org/10 wttp://...1111...11.1111
2016.90
Bingyi Kang and Jiashi Feng. Transferable meta learning across domains. In Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, Cali- formia, USA, August 6-10, 2018, pp. 177-187, 2018. URL pddf
org/ 2018/
proceediigs//pppe..e
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 3332 1338 2015. ISSN 0036-8075. doi: 10. 1e.11///cceecce.abbb.. aab3050. URL
350/ 62666116333
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M. Hospedales. Learning to generalize: Meta- learning for domain generalization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3490-3497, 2018. 16067
URL https://
aaai, org/ ocs/ wnwex.idexxe///e php/ AAAI I/ AAAII8 www.aa///ppee.veeeee.e.
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M. Hospedales. Episodic training for domain generalization. CoRR, abs//102.00011, 2019. URL http:
//arxiv. wwwpxii.orr/abb//11e 900...

---

Under review as a conference paper at ICLR 2020
Fei-Fei Li, Robert Fergus, and Pietro Perona. One-shot learning of object categories. 1EEE Trans. Pattern Anal. Mach. Intell., 28(4):594 611, 2006. doi: 10.1109/TPAM..200..779 URL https:
//doi. wwo..orgg10.0o 1109/ TPAMI 2006. 79
Yaoyao Liu, Qianru Sun, An-An Liu, Yuting Su, Bernt Schiele, and Tat-Seng Chua. LCC: learning to customize and combine neural networks for few-shot learning. CoRR, abs/1904.08479, 2019.
URL
org/ abs 1904. 08479
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi, Fine- visual classification of aircraft. CoRR, abs/1 306,551,,,,,, 2013. URL http:://a httt:://arxiv.v00 org/
abs/1306. 5151
Saeid Motiaan, Marco Piccirilli, Donald A. Adjeroh, and Gianfranco Doretto. Unified deep su- pervised domain adaptation and generalization. In IEEE International Conference on Com- puter Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 5716-5726 2017. doi: URLbttps://ddat wwopl.oogg/1... http://...1111//////o. 2017. 609
10.1109/ICCV
S. Munder and D. M. Gavrila. An experimental study pedestrian classification. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 28(1 28(11):1863-188 1868, Nov 2006. ISSN 0162- 8828. doi: 10.1 109/TPAM1.20002211
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. NIPS, 01 2011.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta leamiig algorithms. CoRR, abs/1803 0003,0299, 2018 URL http::/t///xi.....////
02999
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. ICVGIP '08, pp. 722-729, Washington, DC, USA, 2008. IEEE Computer doi: 10.1 10.1109/CVGPP22008..48 URL https: //dci. org/
Society. ISBN 978-0-6995-34700
10. 1109 ICVGIP 2008.400044 47
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018. NeurIPS 2018, 3-8 December 2018. Montréal,
Canada., pp. 719-729, 2018.
URL
7352-tadame www.ddaa--s--deeeeeeaeeeee ott.::/:..........--aaaa.eeeeee-
Minseop Park, Jungtack Kim, Saehoon Kim, Yanbin Liu, and Seungjin Choi. Mxml: Mixture of meta-learners for few-shot classification. CoRR, abs/1904.05658, 2019. http::t/////xx
org htt//bbs/1900 05658
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial In- telligence (IAAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelli- gence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3942-3951, 2018. URL https:/ www aaai. wwg/ index. pttp://AAA/AAAA///PPaPaa.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. 2018 IEEE Conference on Computer Vision and Pattern Recogni- fion, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 pp.
2018. doi: 10. 109/ cvpr__0187 html
CVPR.201 000555. URL hptpacccssppcccccsssss thwv com http///ooteettttoeeeeeeee
html httml/Qiace haamll/@aa_FFexee Fee Shot_
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In 5th In- ternational Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL https: http:///pppeeeiiieee/m net. wtw.i/ewww.e///ouuuttm umr
id=rJY0-Kc11
10

---

Under review as a conference paper at ICLR 2020
Sylvestre-Alviss Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual do- mains with residual adapters. In Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Processing Systems 2017. 4-9 December 2017, Long Beach, CA, USA, pp. 506-516, 2017. URL. sual doma ins ch- -ewp--re-iddaladdaaaaaa-a adapters
n cc cc/paper
1154-learning-mulee
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 8119-8127, 2018. doi: 10.1109/CVPR.2018 0.00477. URL / evpr httt////////...// Effi Efficient, Paramet paper. html
thecvf com/ content.
of Stcpiionooooo________ee___ CVPR_2012
Sebastian Ruder. An overview of multi-task learning in deep neural networks. abs/1706.05098, 2017. /tL/ wwwporgg/ggg......o//////// 05098
CoRR,
O. Russakovsky, J. Deng, Hao. Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei. Im- ageNet Large Scale Visual Recognition Challenge. International Journal of Computer Viston (JJCV), 1 115(3):211 252,, 2015. doi: 12.1007/11223--15-000
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Confer- ence Learning Representations, 2019. URL httph:tt/////....... hnttps://openrvitt http///pppee/veeeeeee net/ weet/fortte forum? ?id=
BJak ThAcK7
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E, Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-ccperts layer. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. forum? (rrumidd=BccMMB ckMDQqg
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot Advances in Neural Information Processing Systems 30: Annual Con- ference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 4080-4090, 2017. URL
learning.
nips cc/paper/
6996 1995-prototypiccl--eeete wwwpppiccaneetttt...... for few- shot learning
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0402, 2012. URL http://arxiv
arg/abs/1214 0402]
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man computer: Bench- marking machine learning algorithms for traffic sign recognition. Newral Networks, 32:323-332, 2012. doi: mf mem.e...emuee..22222200000 URLhttps: petttps://doi.rrgge /ww.//oiiioooo wtt.di..og/...111//5/...3 neunet.. 2012. 016
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Car- les Gelada, Kevin Swersky, Picrre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. CoRR, abs/1903.00 030999 2019. URL.
http: arxiv.aiv............ org. abs 1903..90... 03096
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances in Neural Information Processing Sys- tems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. (630-3638, 2016. URL htth::ttp/pppeee/e nips cc/paper/ for one shot Sirt--eernning
6385 1785-mace 1105--mttchinggneetokeee.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Toward multimodal model-agnostic meta- meta-learning learning. CoRR, abs/1812.07172, 2018. URL http: wttpaaittww.....////oo
77
11

---

Under review as a conference paper at ICLR 2020
Yu- Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016. Proceedings, Part VI, pp. 616 634, 2016. doi: 10. 6. -
11--3-39--4646-44 37. URL. https:
37
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 7045-7054, 2019. URL http:::/procceeeiiiii ngs.
m wtwpprrss////ye00 90..99.hhh ht.ml
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Learning adbettinngaaapttttinnn for few-shot learning. CoRR, abs/18 88 11.006664, 2018. URL http:::/aaxi....tog/t/a.o /ttp://xww...///////////22.....0
12

---

Under review as a conference paper at ICLR 2020
A DATASETS
In our experiments, we use the Visual Decathlon Challenge dataset (Rebufh et al. 2017 which consists of ten datasets for image classification listed below.
FGVC-Aircraft Benchmark (Aircraft, A) Maji et al. 2013
CIFARIO0 (CIFAR100, C) Krizhevsky 2009
Daimler Mono Pedestrian Classification Benchmark (DMPCB) (Munder Gavrila 2006)
Describable Texture Dataset (DTD, D) (Cimpoi al. 2014)
German Traffic Sign Recognition Benchmark (GTSRB, G)
Stallkamp et al. 2012)
ImageNet ILSVRC12 (ImageNet12, 1) (Russakovsky et al.
2015
Omniglot (Omniglot, O) (Lake et al.
2015)
Street View House Numbers (SVHN)
Netzer et 2011
UCF101 (UCF101, U) (Soomro et al.
2012)
Flowers103 (Flowers, F) (Nilsback & Zisserman 2008
The categories and the number of images each dataset as well as the image sizes are significantly different. All images have been resized isotropically 72 72 pixels that each image from various domains has the same size.
Daimler Mono Pedestrian Classification has only 2 classes, pedestrian and non-pedestrian. We cluded it from our experiments as we are considering 5-way classification tasks. SVHN was also excluded since SVHN has only 10 digit classes from 0 to 9, which were few split for meta- training and meta-testing. To use the remaining eight datasets for multi-domain few-shot classifica- tion, we divide the examples into roughly 70% training. 15% validation, and 15% testing classes. For ILSVRC12, we follow the split of Triantafillou et al. (Triantafillou et al. 2019) to adopt class hierarchy, and we use random class splits for other datasets. The number of classes at each split shown in Table A1 We only use train and validation sets of the Visual Decathlon because the labels of the test set not publicly available.
Aesott
CIFAR-100
Desombable Textures
Germa Traflle Sign
Inapehent 2012
Vas-
Onnight
UCF101 Dynanis Images
Table A1: The details of datasets used in our experiments.
SPLITS
DATASET
# DATA
# CLASSES
TRAIN
VAL
TEST
AIRCRAFT
6667
100
70
15
15
CIFAR100
50000
100
70
15
15
DTD
3760
47
7
8
GTSRB
39209
43
IMAGENETI2
1281167
1000
712
158
130
OMNIGLOT
25968
1623
1136
243
244
UCFI01
9537
101
70
15
16
FLOWERS
2040
102
16
16
B ARCHITECTURES
Figure 2 shows the architecture of the embedding network fr(; a,),, which processes an input image and produces 51 5dl-dinmnso embedding vector. The embedding network is based on the
13

---

Under review as a conference paper at ICLR 2020
CONVOLUTION 1x1
CHANNEL- TRANSFORM
MODULATORS {aik-=1
9,795.584
61,440
BASE NETWORK 0 SELECTION NETWORK Φ
11,176,512
11,176,512
66, 696
66, 696
SUM
21, 038, 792
11, 304, 648
Table A2: The comparison of the number of parameters for convolution x and channel-wise transform modulators. This is the case when the number of source domains is 8.
ResNet- 18 architecture (He et al. 2016), which consists of one convolutional layer with 64 7 x filters followed by 4 macro blocks, each having 14-128-256-512 3 3 filters. Figure 2aa and Figure b)) depict how the base network is modulated by the convolution x modulator and the channel- wise transform modulator, respectively. These modulators placed within each residual block of the macro blocks, same as the previous works in (Rebuffi et al. 2018 and (Perez et al. 2018).
The number of parameters for two modulators are shown in Table A2 The values on the first row the number of modulator parameters that are additionally applied to the embedding network. Note that the channel-wie transform modulator has much fewer parameters than the convolution 1 modulator. In particular, the channel-wise transform modulator has negligible number of parameters compared to that of the base network, i.e., R8sNet-18. For each embedding model, the convolution 1 x 1 modulator has about 10% of the number of parameters that the base network has whereas the channel-wise transform modulator requires only less than 1%.
The selection network fs(:; Φ) is two-layered MLP (multi-layer perceptron) network, which re- ceives an embedding vector produced by the embedding network as an input and performs the best model index prediction. Two layers are linear layer of 512x 128 and a linear layer of 128 x (M+1), where M is the number of source domains.
C TRAINING DETAILS
Algorithm describes the overall training procedure to construct the model pool and the selection network. Although we trained three components in a sequential manner, joint training of these com- ponents seems to make sense also.
For the fair comparison with Fine-tune method, also apply algorithm-specifie refinement meta- testing time, inspired by 'further adaptation' in (Chen et al., 2019), to all other algorithms including ours. linear classifier is placed on top of the embedding network of the ProtoNet. the self-attention module of the FEAT' or the modulated embedding network of our models. During meta-testing, other parameters are fixed and the classifier fine-tuned using the support examples for 100 iterations per episode. In case of FEAT, the classifier is trained for 100 epochs per query example not per episode because FEAT modulates a representation space for each query. We also adjust the number of adaptation of the ProtoMAML 100 for the better atsk-adaptation as done in (Chen al. 2019).
The hyperparameters including the learning rate are selected by grid search based on the validation accuracy. For FEAT and ProtoMAML. Adam optimizer is used for training and the learning rate and weight decay are set to be 0.0001. Other models are also trained using Adam optimizer with the learning rate 0.001 but without any regularization including the weight decay.
In Simple-Arg, each embedding model is trained on a separate source dataset following the method proposed in the Prototypical Networks (Snell et al.) 2017). We can obtain a little higher performance with this approach than training each model following the standard supervised learning procedure.
All reported test accuracy values averaged over 600 test episodes with 10 queries per class.
14

---

Under review as a conference paper at ICLR 2020
Algorithm The overall training procedure
Input:
Training data from Ds - (Ds J embedding networks fE(+), selection network (-).).
Output: Learned parameters 8, {ar Φ.
Step 1: Build base network
Build one large classification dataset (Zagg: Vagg) by aggregating all classes from Ds.
2: Learn 0 by optimizing {=; 0,))) for the aggregated dataset (oy: no modulation).
Step 2: Add modulators through intra-domain episodic training
1: while not converged do
2: Sample one domain Ds, from Ds, then sample one episode (S, Q) from Ds.-
3: Learn a; by optimizing fe (z; a4) for (S while keeping 0 fixed.
4: end while
Step 3: Build selection network through cross-domain episodic training
1: while not converged do
2: Sample one domain Ds, from Ds. then sample one episode (S, Q) from Ds..
3: Get task representation Ptask by averaging embedding vectors of from the base network.
4: Measure accuracies of M + 1 available embedding models {fs (2; 0 ox:) for (S, Q)
5: Set the best model index Yset the index of the model with the highest accuracy.
6: Learn é by training [s(ataski 6) so as to predict Yset for (S, Q).
7: end while
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 RESULTS WITHOUT FURTHER ADAPTATION
We present the experimental results when do not apply the further adaptation scheme introduced in (Chen et al. 2019). Specifically, ProtoNet, FEAT, and our models are tested without additional linear classifiers [: v The number of parameter update steps in ProtoMAML is reduced 3, which is not enough to have the models fine-tuned. Tables A3 and A4 show the results tested on seen and unseen domains, respectively. We can see that accuracy drops in almost all cases compared to corresponding cases with further adaptation whose results are in Section 3 in the main text, but our models generally do better than other methods in any experimental settings.
Table A3: 5-way 5-shot classification accuracy domains without further adaptation.
METHODS
TARGET
F-T
PROTO
FEAT
PMAML
S-AvG
DoS
DoS-CHI
DoA
DOA-CH
A
39.53%
19.61%
58.53%
66. 06.39%
37.019
02.86%
59.39%
42.82%
42.91%
C
53.94%
57.48%
53.85%
59.37%
48.60%
69.94%
71.57%
62.96%
64.37%
52.91%
50.81%
42.98%
68.
56.50%
56.54%
G 78.67%
86.21%
92.74%
95. 23%
74.39%
96.49%
96. 96.63%
82.93%
83.31%
54.64%
52.23%
49.91%
46.80%
41.69%
55.73%
56.49%
67.83%
57.42%
92. 92%96%
96 96.20%
93.18%
98. 98.9
444%
86%
98 80%
95.5-1%
96, 96.70%
U
62.78%
06.31%
74%7
66.17%
15.
69.78% 78%
69.45%
69 19.68%
68.93%
83. 83.46%
69.98%
83. 83.60%
84.12%
AVERAGE
61.76%
68.13%
67.58%
69.95%
58.53%
49.49%
73.99%
69 09.23%
69.18%
A: Aircraft, C: CIFAR100, G: GTSRB. I: ImageNet12, O:Omniglot, U:UCFI01, F.VGG-Flowers F-T: Fine-tune, Proto: ProtoNet, PMAML: ProtoMAML, S-Avg: Simple-Avg
D.2 TEST ACCURACY IN A NUMERIC FORM
In Table A5 and A6 we present the exact values of test accuracy with the largest number of source domains, which are shown in Figures 4 and S the main text.
D.3
COMPARATIVE ANALYSIS ABOUT THE AVERAGING METHODS
As an effort for better understanding the averaging methods, we investigate how each model in the pool contributes to the final prediction. Figures Al and (T) show how many correct predictions
15

---

Under review as a conference paper at ICLR 2020
Table A4: 5-way 5-shot classification accuracy on unseen domains without further adaptation.
METHODS
TARGET
F-T
PROTO
FEAT
PMAML
S-AVG
DOS
DoS-CH
DoA
DOA- CH
A
36.40%
35.42%
33. 33.30%
32.87%
33. 33.18%
31.73%
35.63%
36 36.93%
37. 28%
C
53.26%
17.17%
52.60%
58.01%
49.23%
00.79%
50.09%
58.30%
59.22%
D
55. 55.18%
51.033%
30.37%
45.46%
45.99 15.99%
54.11%%
48.81%
54.88%
34.37%
G
78.01% 1%
76.33%
75.79%
78.81%
69.49%
77.03%
77.18%
77.36%
77.31 %
92.90%
33.50%
37.79%
34.34%
32.73%
35.27%
O
92. .9%
92.64%
91.99%
83.80%
91.45%
93.68%
91.99%
94.95%
94.84%
U
62.16%
59.74%
58.54%
40%40%
59.06%
62.04%
62.21%
65.93%
67. 65.25%
81.00%
79.42%
80.82%
69.47%
75.39%
81.85%
82.69%
82.71%
83.89%
AVERAGE
61.69%
60.33%
59.62%
57.91%
57.74%
61.70%
60.17%
63.19%
63.68%
A: Aircraft, C: C0F,R100, G: GTSRB, I: ImageNet1 O:Omniglot, U:UCF101, F.VGG-Flowers
F-T: Fine-tune, Proto: ProtoNet, PMAML: ProtoMAML, S-Avg: Simple-Avg
Table A5: Few-shot classification accuracy on seen domains. All eight domains were used for train- ing.
(a) 5-way s-shot test accuracy.
METHODS
TARGET
F-T
PROTO
FEAT
PMAML
S-AVG
DoS
Dos-CH
DoA
DOA-CH
27. 97.90%
39.05%
45.39%
53.03%
30.45%
19.14%
44.02%
32.93%
32.63%
34.51%
40.41%
33.22%
49.58%
43.90%
D
38 08%
38.77%
34.16%%
38.45%
29.52%
41.15%
40.53%
39.35%
38. 88.86%
54.37%
78.57%
84.85%
88.85%
67.97%
91.08%
89.02%
71.60%
70.68%
92%
18%
33.44%
33.53%
31.03%
40.60%
40.76%
41.55%
41.56%
75-52%
86.35%
81.44%
94.30%
86.88%
95.37%
95.25%
87.69%
89.74%
44.6%%
48.40%
46.63%
51-46%
40.36%
50.16%
53.19%
50.03%
19. 22%
59.32%
62.11%
54.03%%
57.20%
49.62%
65.36%
66.94%
65.34%
65.93%
AVERAGE
16.71%
76%
51.81%
57.15%
46.13%
60.41%
59.91%
53.92%
54.06%
A: Aircraft, C: CIFARIO0 100, G: GTSRB. I: ImageNet12, O:Omniglot, U:UCFI01, F:VGG-Flowers
F-T: Fine-tune, Proto: ProtoNet, PMAML: ProtoMAML, S-Avg: Simple-Avg
(b) 5-way 5-shot test accuracy.
METHODS
TARGET
F-T
PROTO
FEAT
PMAM1.
S-AvG
Do$
Dos-CH
DoA
DOA-CH
A
39.53%
49.82%
57.62%
65.89%
40.79%
62.65%
59.31%
47.09%
46.51%
57.45%
53.30%
60.90%
70.73%
71.29%
64.21%
65.70%
D
55.68%
52.43%
52.49%
50.94%
44.42%
59.27%
56.800 80%
59.68%
57.74%
G
78.67%
89.28%
92, 92%1
95. 95.25%
81. 59%
80%
96 91%
89.38%
89.21%
54.64%
51.86%
47.20%
42.35%
54.95%
55.36%
54%
O
92.96%
96 65%
93.10%
98. 98%
95. 36%
78%7
98 88%
96.53%
97.55%
U
62.76%
66.02%
64.77%
66.09%
62.66%
70.32%
70.46%
69.93%
70.57%
79.92%
83.30%
71.38%
75.17%
72.00%
81.90%
81.68%
85.11%
81.66%
AVERAGE
64.70%
08.35%
67.08%
70.05%
63%
80%
74.21%
71.25%
71.00%
A: Aircraft, C: CIFAR100, G: GTSRB, I: ImageNet12, O:Omniglot, U:UCF101, F.VGG-Hlowers
F-T: Fine-tune, Proto: ProtoNet, PMAML: ProtoMAML, S-Avg: Simple-Avg
are made by each model with the Simple-Avg and our DoA methods respectively given queries per episode for 40 episodes.
The measured numbers show that the individual models of our DoA perform better than those in the Simple-Avg, which explains the higher performance of the proposed method partly. Additionally, we can observe that major contributors (i.e., the models with higher accuracy) tend to change every episode in our DoA whereas only two models to play dominant roles regardless of the given episode. This implics that our method for constructing the model pool provides the averaging model with more beneficial diversity.
16

---

Under review as a conference paper at ICLR 2020
Table A6: Few-shot classification accuracy on unseen domains. Seven domains other than the target domain were used for training.
(a) 5-way I-shot test accuracy.
METHODS
TARGET
F-T
PROTO
FEAT
PMAM1.
S-AvG
DoS
DoS-CH
DOA
DDA-CH
26.78%
26.88%
27.50%
25.38%
26.05%
27.05%
26.76%
27.82%
28.01%
C
36.43%
36.35%
37.63%
30.04%
31.57%
34.56%
30.93%
40.05%
39.78%
37.73%
34.25%
29.06%
36.56%
37.24%
G
53. 53%
64.07%
47. 17.83%
08%
45. 10%10%
55.77%
56 16.96%
58 81%
57. 57.28%
27. 18%
26.03%
25.97%
27.18%
27.53%
25.38%
26.45%
27.56%
28.88%
75.11%
08.38%
75.09%
55.28%
71.16%
70.73%
71.26%
80.26%
80.06%
U
42.11%
39 39.06%
38. 12%
37.16%
34.63%%
39.62%
43.04%
46.20%
45. 15.88%
59.32%
55.94%
56.34%
45.93%
46.02%
56.38%
58.98%
64.56%
63.51%
AVERAGE
44.76%
42.52%
42.96%
37.78%
38.89%
43.37%
43.87%
48.03%
47.45%
A: Aircraft, C: CIFAR G: GTSRB, I: ImageNet12, O:Omniglot, U:UCF101, F.VGG-Flowers
F-T: Fine-tune, Proto: ProtoNet, PMAML: ProtoMAML, S-Avg: Simple-Avg
(b) 5-way 5-shot test accuracy.
METHODS
TARGET
PROTO
FEAT
PMAML
S-AvG
DoS
DoS-CH
DoA
DOA-CH
36.40%
36.12%
34.81%
33.32%
34.75%
36.68%
37.87%
39.03%
39.41%
C
53.26%
54.98%
52.40%
56. 14%
48.52%
57.25%
51.1
59.83%
60.99%
55%
51.91%
51.90%
15.12%
55.49%
51.65%
56.47%
55.25%
78.01%
80.23%
76.
79.96%
76.29%
80.98%
81.21%
81.90%
82.00%
31.81%
32. 42%
31.27%
36.50%
38. 15%
34.16%
32.68%
35.33%
36. 36.31%
92.69%
92.87%
89.93%
91.79%
95.29%
U
62. 16%
60 60.32%
57.53%
57. 49%
59. 59.12%
62.33%
62.81%
67. 67.57%
67. 67.92%
F
81.00%
79.81%
76.62%
69.55%
76.57%
83.39%
82.37%
84.41%
84.58%
AVERAGE
61.69%
61.08%
38.83%
57.71%
58.79%
63.05%
61.44%
64.98%
65.27%
A: Aircraft, C: CIFARIOO, G: GTSRB, I: ImageNet12, O:Omniglot, U:UCFI0L, F.VGG-Flowers F-T: Fine-tune, Proto: ProtoNet, PMAML: ProtoMAML, S-Avg: Simple-Avg
1
7
4
5
6
7
8
9
10
11
19
19
14
15
16
17
18
19
20
21
27
25
24
25
26
27
2A
70
30
$1
30
51
54
35
36
37
38
3g
40
Mede
41
40
52
43
42
36
57
36
41
36
48
41
56
47
46
46
41
48
37
43
40
39
67
43
40
43
40
45
37
41
38
44
37
40
46
30
40
42
39
40
Model
21
21
18
14
24
28
23
17
18
18
28
18
98
18
18
18
16
39
25
97
95
96
26
32
28
23
22
20
17
16
Model
22
32
17
29
30
18
20
24
29
19
20
22
23
32
26
28
24
97
28
25
29
26
23
24
36
28
31
28
29
21
27
28
28
Model
59
42
38
41
42
36
41
38
44
36
46
47
88
46
48
43
44
64
42
45
68
41
45
48
40
64
41
40
40
42
44
42
48
48
38
41
44
38
Model e
21
13
18
18
20
12
14
12
19
16
16
192
19
13
11
10
19
19
19
11
20
21
11
11
11
11
19
16
11
15
10
11
200
14
11
121
16
16
Model
21
23
18
18
22
17
18
18
20
29
21
18
15
22
25
17
29
20
22
91
22
21
29
91
17
21
18
24
23
28
22
19
19
281
29
18
19
111
15
27
(a) Simple averaging (Simple-AN -Agel)eee)
19
18
Model
42
36
4.1
40
38
42
37
41
48
42
42
46
30
41
65
40
36
42
61
3Y
62
41
43
34
66
A4
43
80
35
38
40
41
40
36
83
40
41
33
67
Model
42
40
411
43
56
54
54
41
43
59
37
37
31
4A
46
31
36
41
45
36
42
A5
41
34
64
A5
41
99
96
10
57
38
43
43
59
40
41
41
32
45
Model
40
38
40
46
30
36
37
43
43
38
40
40
36
41
41
41
33
35
43
35
41
38
43
35
89
38
99
44
33
38
38
38
43
40
32
37
38
38
34
42
Model
42
40
47
38
40
57
38
39
36
88
48
35
36
42
36
39
35
83
K7
38
45
40
43
43
57
Model
65
36
40
34
45
46
40
38
40
36
41
45
39
37
43
41
39
42
43
42
38
66
43
44
45
37
42
38
40
44
44
45
40
48
(b) Proposed averaging (DoA)
Figure A1: Contributions of individual models in model averaging methods.
E THE LOSS FUNCTION FOR THE SELECTION NETWORK
Equation 3 shows the loss (lossset) used to train the selection network ()) Here, accm is the classification accuracy of the model with the modulator parameterized by a., in the pool. The accuracy is measured for query examples in given episode by making prediction in the same way
17

---

Under review as a conference paper at ICLR 2020
with the Protypical Networks (Snell et al. 2017), ) where the class whose prototype is the closest to the embedding vector given query example is picked as the final prediction.
NK
1
Σ
()(
Ftack
NK
Ysel softmax (fstmaax((sfaa (Etaak 6))
(3)
argmax( facem (f0), Vi 2 of))),))-
Ysel
l0s8se! cross_entropy (nntroy((jett,rrroo Uset)
18


Reviews:
[{"review_id": "Sylxa5s6tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed to extend meta learning for few-shot classification to the multi-domain setting. The problem looks interesting, but the proposed method is incremental. \n\nSpecifically, there are three main components of the proposed method in training: 1) a base network, 2) a model pool, and 3) a selection network. The base network is trained by simply combing training data from all the domains, which is the most common way to learn a shared model across different domains. The model pool is constructed highly based on existing works [Snell et al., 2017; Rebuffi et al., 2018], and thus is out of novelty. The main technical contribution should lie on the selection network, where a key research issue is how to represent different tasks. However, in this work, the authors just simply used the mean of outputs of the base network over all instances of a specific task to represent the task. Why is this strategy good for task representation? The authors failed to explain it. Neither theoretical nor empirical studies are provided. Therefore, the motivation behind this is unclear and not convincing. \n\nIn testing or inference, the authors proposed to use a class-prototype based approach or a weighted average of ensemble outputs to make predictions on the target-domain instances. Theses techniques are indeed widely used in the literature, which are again not novel.\n\nIn summary, though the problem studied in this paper is interesting, the proposed method is incremental, and has limited novelty contributions. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkeKQwaatS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed to address the few-shot learning problem, especially for the cross-domain setting where a newly coming task originates from a different distribution (or in this work implemented by sampling from an unseen dataset). Basically, the authors constructed a model zoo based on source datasets at hand, and learned an “argmax” meta-selector which takes embedding of a task as input and outputs the model selection index. The idea is very intuitive, and the implementation is kind of an incremental combination of (Rebuffi et al., 2018) building models for multi-tasks and (Oreshkin et al., 2018) tailoring based on task embeddings.\n\nPros:\n-\tThe problem that this work aims to address, i.e., domain heterogeneity, is significant.\n-\tThe author investigated several variants of the proposed method, varying the architecture of the modulator and the inference scheme.\n-\tThe paper makes clear points and is quite easy to follow.\n\nCons:\n-\tThe concern at first priority is the practicability of the proposed method. The authors likely misunderstand why almost all existing SOTA algorithms modulate parameters/activations themselves – it is because of their advantages of easily being efficiently updated across tasks. However, the proposed method has to store an increasing number of models as a task comes, which takes huge storage cost. Also, when a new task arrives, will all models be trained from scratch to enforce the feature extractor to also be shared by this new task? It is an economically infeasible solution for meta-learning/few-shot learning, to my best knowledge.\n-\tActually while I was reading the paper from the beginning, I was expecting a meta-scheme that learns the weights of the models in the pool. In that case, the inference scheme of averaging in Eqn. (2) could be more intuitively correct, by paying more attention to those models which are similar. \n-\tThe third concern comes from the empirical results. \n    o\tBaselines: the two baselines compared, FEAT and ProtoMAML, are actually un-published. The results of a handful of SOTA algorithms recently published, including (Oreshkin et al., 2018) and (Yao et al., 2019), should definitely be incorporated. I do not see any specific part in your setup, and I cannot understand why (Oreshkin et al., 2018) does not converge. According to my experiences, it is quite easy to converge. Besides, (Yao et al., 2019) is specifically designed to tackle the cross-domain (for an unseen dataset) setting.\n    o\tHow can you validate the effectiveness of the task embedding? It is better to conduct ablation studies to consider those like autoencoder embeddings. \n    o\tCould you introduce the setting of fine-tuning in more details? I am confused why fine-tuning seems always superior than ProtoNet?\n    o\tCould you give more discussion on the results in Table 1? As the number of source datasets increases, an effective meta-model of course should contribute more to the target dataset, while it is not in this work. Does this signify that the proposed method is not that effective?\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgONWIA_B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "###Summary###\nThis paper aims to tackle few-shot classification with many different domains. The idea is to build a pool of embedding models, which are based on the same base network. The models are diversed by their own modulators. The high-level intuition is to let the model pool capture good domain-invariant features by the shared parameters and domain-specific features by the selection network, which is desirable to represent the complex cross-domain task distribution, without a significant increase in the number of parameters. \n\nThe model includes the embedding networks f_E and a selection network f_S. The overall training process of the proposed method is:\n(1) Train the base embedding network f_E with the aggregated dataset from multiple domains.\n(2)Sample one domain to optimize the corresponding embedding model. \n(3) Build a selection network through cross-domain episodic training. The task representation is calculated by average the embedding vectors from the base network, which resembles the prototype. Then the model with the highest accuracy on the query set is selected to compute the labels for the target domain.\n\nFor the embedding model f_E, the paper provides two architectures, one with convolution 1x1 and the other with Channel-wise transform, denoted as DoS and DoS-CH, respectively. Instead of selecting the model which has the highest accuracy, the averaging model generates an output by averaging class prediction probabilities of all constituent models, denoted by DoA and DoA-CH, respectively.\n\n The baseline used in this paper includes Fine-tune, Simple-Avg, ProtoNet, FEAT, ProtoMAML. \n\nThe paper performs extensive experiments on a batch of datasets, including Aircraft, CIFAR100, DTD, GTSRB, ImageNet12, Omniglot, SVHN, UCF101, and Flowers. The 5-way 1 shot and 5 way 5 shot experimental results demonstrate that the proposed DoS and DoS-CH can outperform other baselines in the \"see domains\" setting. However, the results on \"unseen domains\" experiments are worse than the averaging baselines (DoA, DoA-Ch). \n\nThe paper also surveys the few-shot classification results on a varying number of source domains to show that DoA and DoA-ch are robust to deal with different settings.\n\n### Novelty ###\n\nThe paper composes of two sub-network, with one baseline network to extract the commonsense knowledge for different datasets and another selection network to select the best model from the model pools for each specific domain. \n\nThe idea of leveraging multiple modulators for domain-agnostic image recognition is interesting and heuristic, thus the proposed framework shows some novelty.\n\n###Clarity###\n\nOverall, the paper is readable and logically clear. The images are well-presented and well-explained by the captions and the text. \n\nWhile this paper misses many prior works both in the track of domain-agnostic learning and few-shot learning. I would recommend the authors to the following materials:\na). Domain Agnostic Learning with Disentangled Representations, ICML 2019. https://arxiv.org/pdf/1904.12347.pdf\nb). Generalizing from a Few Examples: A Survey on Few-Shot Learning, \nhttps://arxiv.org/pdf/1904.05046.pdf\n\nThere also exist some grammar mistakes and typoes in the paper. It will be better to revise and polish the paper. \n\n###Pros###\n\n1) The paper proposes a framework that includes two parts, i.e. the base network and the selection network. The idea is to make deep models more robust to different image domains, which is interesting and heuristic.\n2) The paper provides extensive experiments on a wide range of datasets. The experiments demonstrate the effectiveness of the proposed method.\n3) The paper is applicable to many practical scenarios since the data from the real-world application is complicated. \n\n###Cons###\n\n1) The critical component of the proposed method is the selection network. However, the experiments on the \"novel domains\" show that the proposed DoS and DoS-Ch works worse than just averaging the outputs of the models in the model pool. \n2) The paper should explain the details about the baseline experiments. Since the proposed models have much more parameters than the baselines, what's the effect of the auxiliary parameters? Is the comparison between the baselines and the proposed method fair?\n3) The presentation of the proposed paper should be polished. Many critical techniques used in this paper are not well-explained, such as the ProtoNet. There exist some grammar mistakes and typos that need a revision. \n\n\nBased on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of few-shot classification across multiple domains. The main algorithmic contribution consists of a selection criteria to choose the best source domain embedding for a given task using a multi-domain modulator. 

All reviewers were in agreement that this paper is not ready for publication. Some key concerns were the lack of scalability (though the authors argue that this may not be a concern as all models are only stored during meta-training, still if you want to incorporate many training settings it may become challenging) and low algorithmic novelty. The issue with novelty is that there is inconclusive experimental evidence to justify the selection criteria over simple methods like averaging, especially when considering novel test time domains. The authors argue that since their approach chooses the single best training domain it may not be best suited to generalize to a novel test time domain. 

Based on the reviews and discussions the AC does not recommend acceptance. The authors should consider revisions for clarity and to further polish their claims providing any additional experiments to justify where appropriate.

Decision: Reject