Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed to address the few-shot learning problem, especially for the cross-domain setting where a newly coming task originates from a different distribution (or in this work implemented by sampling from an unseen dataset). Basically, the authors constructed a model zoo based on source datasets at hand, and learned an “argmax” meta-selector which takes embedding of a task as input and outputs the model selection index. The idea is very intuitive, and the implementation is kind of an incremental combination of (Rebuffi et al., 2018) building models for multi-tasks and (Oreshkin et al., 2018) tailoring based on task embeddings.

Pros:
-	The problem that this work aims to address, i.e., domain heterogeneity, is significant.
-	The author investigated several variants of the proposed method, varying the architecture of the modulator and the inference scheme.
-	The paper makes clear points and is quite easy to follow.

Cons:
-	The concern at first priority is the practicability of the proposed method. The authors likely misunderstand why almost all existing SOTA algorithms modulate parameters/activations themselves – it is because of their advantages of easily being efficiently updated across tasks. However, the proposed method has to store an increasing number of models as a task comes, which takes huge storage cost. Also, when a new task arrives, will all models be trained from scratch to enforce the feature extractor to also be shared by this new task? It is an economically infeasible solution for meta-learning/few-shot learning, to my best knowledge.
-	Actually while I was reading the paper from the beginning, I was expecting a meta-scheme that learns the weights of the models in the pool. In that case, the inference scheme of averaging in Eqn. (2) could be more intuitively correct, by paying more attention to those models which are similar. 
-	The third concern comes from the empirical results. 
    o	Baselines: the two baselines compared, FEAT and ProtoMAML, are actually un-published. The results of a handful of SOTA algorithms recently published, including (Oreshkin et al., 2018) and (Yao et al., 2019), should definitely be incorporated. I do not see any specific part in your setup, and I cannot understand why (Oreshkin et al., 2018) does not converge. According to my experiences, it is quite easy to converge. Besides, (Yao et al., 2019) is specifically designed to tackle the cross-domain (for an unseen dataset) setting.
    o	How can you validate the effectiveness of the task embedding? It is better to conduct ablation studies to consider those like autoencoder embeddings. 
    o	Could you introduce the setting of fine-tuning in more details? I am confused why fine-tuning seems always superior than ProtoNet?
    o	Could you give more discussion on the results in Table 1? As the number of source datasets increases, an effective meta-model of course should contribute more to the target dataset, while it is not in this work. Does this signify that the proposed method is not that effective?

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors proposed to extend meta learning for few-shot classification to the multi-domain setting. The problem looks interesting, but the proposed method is incremental. 

Specifically, there are three main components of the proposed method in training: 1) a base network, 2) a model pool, and 3) a selection network. The base network is trained by simply combing training data from all the domains, which is the most common way to learn a shared model across different domains. The model pool is constructed highly based on existing works [Snell et al., 2017; Rebuffi et al., 2018], and thus is out of novelty. The main technical contribution should lie on the selection network, where a key research issue is how to represent different tasks. However, in this work, the authors just simply used the mean of outputs of the base network over all instances of a specific task to represent the task. Why is this strategy good for task representation? The authors failed to explain it. Neither theoretical nor empirical studies are provided. Therefore, the motivation behind this is unclear and not convincing. 

In testing or inference, the authors proposed to use a class-prototype based approach or a weighted average of ensemble outputs to make predictions on the target-domain instances. Theses techniques are indeed widely used in the literature, which are again not novel.

In summary, though the problem studied in this paper is interesting, the proposed method is incremental, and has limited novelty contributions. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: ###Summary###
This paper aims to tackle few-shot classification with many different domains. The idea is to build a pool of embedding models, which are based on the same base network. The models are diversed by their own modulators. The high-level intuition is to let the model pool capture good domain-invariant features by the shared parameters and domain-specific features by the selection network, which is desirable to represent the complex cross-domain task distribution, without a significant increase in the number of parameters. 

The model includes the embedding networks f_E and a selection network f_S. The overall training process of the proposed method is:
(1) Train the base embedding network f_E with the aggregated dataset from multiple domains.
(2)Sample one domain to optimize the corresponding embedding model. 
(3) Build a selection network through cross-domain episodic training. The task representation is calculated by average the embedding vectors from the base network, which resembles the prototype. Then the model with the highest accuracy on the query set is selected to compute the labels for the target domain.

For the embedding model f_E, the paper provides two architectures, one with convolution 1x1 and the other with Channel-wise transform, denoted as DoS and DoS-CH, respectively. Instead of selecting the model which has the highest accuracy, the averaging model generates an output by averaging class prediction probabilities of all constituent models, denoted by DoA and DoA-CH, respectively.

 The baseline used in this paper includes Fine-tune, Simple-Avg, ProtoNet, FEAT, ProtoMAML. 

The paper performs extensive experiments on a batch of datasets, including Aircraft, CIFAR100, DTD, GTSRB, ImageNet12, Omniglot, SVHN, UCF101, and Flowers. The 5-way 1 shot and 5 way 5 shot experimental results demonstrate that the proposed DoS and DoS-CH can outperform other baselines in the "see domains" setting. However, the results on "unseen domains" experiments are worse than the averaging baselines (DoA, DoA-Ch). 

The paper also surveys the few-shot classification results on a varying number of source domains to show that DoA and DoA-ch are robust to deal with different settings.

### Novelty ###

The paper composes of two sub-network, with one baseline network to extract the commonsense knowledge for different datasets and another selection network to select the best model from the model pools for each specific domain. 

The idea of leveraging multiple modulators for domain-agnostic image recognition is interesting and heuristic, thus the proposed framework shows some novelty.

###Clarity###

Overall, the paper is readable and logically clear. The images are well-presented and well-explained by the captions and the text. 

While this paper misses many prior works both in the track of domain-agnostic learning and few-shot learning. I would recommend the authors to the following materials:
a). Domain Agnostic Learning with Disentangled Representations, ICML 2019. https://arxiv.org/pdf/1904.12347.pdf
b). Generalizing from a Few Examples: A Survey on Few-Shot Learning, 
https://arxiv.org/pdf/1904.05046.pdf

There also exist some grammar mistakes and typoes in the paper. It will be better to revise and polish the paper. 

###Pros###

1) The paper proposes a framework that includes two parts, i.e. the base network and the selection network. The idea is to make deep models more robust to different image domains, which is interesting and heuristic.
2) The paper provides extensive experiments on a wide range of datasets. The experiments demonstrate the effectiveness of the proposed method.
3) The paper is applicable to many practical scenarios since the data from the real-world application is complicated. 

###Cons###

1) The critical component of the proposed method is the selection network. However, the experiments on the "novel domains" show that the proposed DoS and DoS-Ch works worse than just averaging the outputs of the models in the model pool. 
2) The paper should explain the details about the baseline experiments. Since the proposed models have much more parameters than the baselines, what's the effect of the auxiliary parameters? Is the comparison between the baselines and the proposed method fair?
3) The presentation of the proposed paper should be polished. Many critical techniques used in this paper are not well-explained, such as the ProtoNet. There exist some grammar mistakes and typos that need a revision. 


Based on the summary, cons, and pros, the current rating I am giving now is weak reject. I would like to discuss the final rating with other reviewers, ACs.


review_title: Official Blind Review #1

Meta-review: This paper addresses the problem of few-shot classification across multiple domains. The main algorithmic contribution consists of a selection criteria to choose the best source domain embedding for a given task using a multi-domain modulator. 

All reviewers were in agreement that this paper is not ready for publication. Some key concerns were the lack of scalability (though the authors argue that this may not be a concern as all models are only stored during meta-training, still if you want to incorporate many training settings it may become challenging) and low algorithmic novelty. The issue with novelty is that there is inconclusive experimental evidence to justify the selection criteria over simple methods like averaging, especially when considering novel test time domains. The authors argue that since their approach chooses the single best training domain it may not be best suited to generalize to a novel test time domain. 

Based on the reviews and discussions the AC does not recommend acceptance. The authors should consider revisions for clarity and to further polish their claims providing any additional experiments to justify where appropriate.

Decision: Reject