Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel training scheme termed Domain Switch Learning (DSL) to pursue the few-shot learning problem under the cross-domain scenario. In practice, DSL uses the data from one domain for a training iteration and switches to another domain for the next iteration. During the switching, two regularization terms are also introduced to balance the domain-specific knowledge and domain-general knowledge. The extensive experimental result demonstrates the effectiveness of the proposed method on several standard benchmarks.
main_review: Strengths
+ The paper is well written and easy to follow.
+ The proposed switch learning schedule seems novel for the cross-domain few-shot task.
+ The proposed Binary KLD Loss and Prompter loss seem to contribute a lot to the accuracy improvement.
+ The overall framework is consistent and the accuracies on standard benchmarks are acceptable.

Weaknesses
- According to Table1 and Table2, the authors directly copied the performance of RelationNet_ATA from [a], but why did not compare the performance of TPN+ATA. Compared with TPN+ATA, the strength of the proposed method would be limited, especially for the Places dataset. Could the authors give some explanation and analysis about that?

- Actually, the Tabel3 made me confused about the experimental results. (1) As mentioned in the first paragraph of Sec.4.2, the authors adopted the leave-one-out-setting, does it means the total number of training samples for multi-domain is larger than that for the single-domain? How to ensure the fairness of comparison？ （2）In Table3, I can't align the performance of the domain-mix scheme with other methods. The performance shown in Table3 has already been higher than some baseline models in Table1 and Table2, why?

- As shown in Table3, even though the accuracy of domain-mix was lower than domain-switch, I am still curious about the performance of domain-mix+teacher-regularization. In such a case, you could also use the average of several adjacent models as the teacher network to do the knowledge distillation. Further, if we use the teacher network to calculate both BKLD loss and RCE loss for the domain-mix scheme, how about the performance? 

[a] Cross-Domain Few-Shot Classification via Adversarial Task Augmentation
summary_of_the_review: For weaknesses 1 and 2, I hope the author could provide more detailed explanations.
For the last one, I am still concerned if the performance gain is provided by the switch scheme or just from the carefully designed BKLD loss and RCE loss.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose to improve domain generalization capability by training with a fast switching manner. The intuition is that when the model is trained with the capability to adapt and retain information, such ability can be extended to adapt to a different test time distribution. 
main_review: Strength:
* the high-level intuition of incorporating adaptability into the training scheme is heuristic to the research community
* the proposed method is applicable to the real-life scenario of few-shot domain shift
* extensive experiments and ablation studies are conducted to demonstrate the performance of the proposed method
* the paper is very well written, easy to understand, well organized, and supported with nicely-drawn figures 

Weakness:
* since the proposed method strictly requires multiple source domain, it is confusing how single domain baselines are relevant here. (it is included in all the experiment tables)
* the authors should consider adding more discussion on how the proposed method of domain-switch is better at extracting general information than domain-mix methods. Since in the domain-mix setting, different domains are given, it can be easier for the model to learn the general information shared by these domains, which is not possible in the domain switch setting.
* comparison with GNN would be helpful


summary_of_the_review: based on the above discussion, I recommend borderline accept for this paper. I would like to adjust my rating for the paper after discussing with the other reviewers and AC.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address the problem of cross-domain few-shot classification. The main contribution is that it introduces domain-switch learning that simulates cross-domain scenario for training, and thus improves the model’s generalization. To this end, it uses multiple domains during training – it only trains on a single domain at each iteration and switches the domain in consecutive training iterations. This training mechanism is different from widely-used strategies like mixing different domains, and it forces the model to learn domain-general knowledge. In addition, two constraints are proposed to further improve the generalization. First, a domain-specific prompter module is introduced, so that the model is not overfitting to the domain in the current iteration. Second, a domain-general teacher module is introduced, so that the model does not forget the already-learned knowledge of other domains. The approach is tested on four datasets, including CUB, Cars, Places, and Plantae, and compared with competing results.
main_review: Paper Strengths:

The authors tackle an important and challenging problem of cross-domain few-shot classification. The proposed approach is simple. Experimental evaluations clearly demonstrate the effect by introducing the domain switching training strategy.

Paper Weaknesses:

1) The approach is evaluated on CUB, Cars, Places, and Plantae in a leave-one-out manner. These four datasets are all fine-grained and consist of natural images. I was wondering how does the proposed domain-switch strategy work on more diverse, heterogenous datasets, like Meta-dataset [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, ICLR 2020].

2) The comparisons in the paper are a little bit weak. It is only marginal better than NASE. How is the performance of NASE in the multi-domain setting? Also, comparisons with some other existing cross-domain few-shot learning methods are missing, for example [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, ICLR 2020] [Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification, ECCV 2020] [A Universal Representation Transformer Layer for Few-Shot Image Classification, ICLR 2021].

3) In Remarks 1, 2, and 3, the relationships between the predicted probability of the prompter, the learner, and the teacher are discussed. While the reasoning intuitively makes sense, there is no mathematically rigorous proof. The current statements sound like they are already-proved facts.

4) While in the introduction it is stated that the model is only trained on one domain at each iteration, one critical implementation detail is that the approach requires a large-scale basic training set (mini-ImageNet) to stabilize the training procedure. I think this should be mentioned in the introduction. Moreover, an ablation study should be provided – what is the performance without mini-ImageNet? Also, if the model is pre-trained on mini-ImageNet, and then trained on the individual domain in a domain-switch manner, does that work?

5) Currently, the training is conducted periodically with pre-defined domain order. How is the sensitivity to the domain order? How is the performance if the domain ordering is completely random in the entire training procedure? In that case, probably the design of the prompter and teacher should be modified as well. 

6) Domain-mix and domain-switch can be viewed as two extremes, where domain-mix uses all the domains at each iteration, while domain-switch uses only one domain at each iteration. How is the performance if we use some strategy in between? For example, at each iteration, some domains are randomly combined as a compound domain, and there is no overlapping domain between two consecutive training iterations.
summary_of_the_review: The paper introduces domain-switch learning that simulates cross-domain scenario for training, and thus improves the model’s generalization. The evaluations and comparisons in the paper are a little bit weak. Some ablation studies and analysis are missing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The goal of this paper is to improve the generalization ability of the few-shot model on novel domains. To this end, this work introduces a learning manner named domain-switch learning (DSL). To make DSL feasible, the work further proposes two modules (i.e., domain-specific prompter and the domain-general teacher). The experimental results show the effectiveness of the proposed framework.
main_review: 



**Strengths** 
1. Overall, this paper is easy to follow. The introduction nicely outlines the method.
2. The domain-specific prompter and the domain-general teacher are straightforward and reasonable. The two modules are needed under the proposed domain-switch learning manner.
3. Domain-general teacher seems a general module that can be used in other few-shot frameworks (although the experiment does not validate this point).
4. The experimental evaluation nicely stuides the hyper-parameters.

**Weaknesses** 

- **[Working mechanism needs more discussion]** 

	The proposed Domain-Switch Learning (DSL) includes only a single domain into a training iteration and switches to another domain for the following iteration. The work highlights several times that such a way makes “the deep model favors domain-general knowledge and is prone to ignoring the domain-specific knowledge”. This point is not very clear and needs more validations. 

	***First***, the domain-mix pipeline learns images from different domains in each iteration. In order to learn well from these images, the model tends to learn common information. This also makes the model favors domain-general knowledge. Thus, it is still not clear why domain-switch learning should be better than domain-mix learning.

	***Second***, In Sec. 3. 2 the work includes a sentence to eliminate the concern of the contradiction when using mini-ImageNet images at each iteration. However, it is still not very convincing.
	A. One could also regard each mixed batch in domain-mix learning as a single “compound” domain. 
	B. Also, following the logic of the motivation, using mini-ImageNet in each iteration could make model memory the information of mini-ImageNet (which is domain-specific). Such mini-imagenet specific information might already be general, which is helpful when testing on bird/car/place images. This might be the reason why including mini-ImageNet images in each iteration helps to learn. 

	***Third***, experimental comparison between domain-switch and domain-mix needs clarification (Please see the first point in the following).

- **[Experimental evaluation needs clarification]**

	***First***, Table 3 needs more clarification. 1) When reporting results on CUB, does the model use four domain images (mini-ImageNet, Car, Places, Plantae) for training? 2) when using the domain-mix manner, does each mini-batch contain mini-ImageNet images? 3) one possible baseline could be first pertaining on mini-ImageNet, and then finetuning using domain-mix manner or domain-switch manner. 4) do the methods listed in Tables 1 and 2 also use mini-ImageNet image at each training iteration or just finetuning?

	***Second***, Tables 1 and 2 list basic baselines. GNN (Tseng et al. (2020)) should be included as well. Moreover,  the results of NASE and BSR under the multi-domain setting (“m” in the table) are missing.

	***Third***, when using the teacher network, why BKLD is better (as shown in Fig4 (a))? It would be better if this work could provide a discussion on this.

- Minor:

	[-] The captions of Tables 1 and 2 are not clear. The definitions of TF, ATA, s, m are missing.
	[-] Summarization of the datasets (domains) used in the experiment could be included in the appendix.

***---Post Rebuttal---*** 

**First**, ***the major concern of "working mechanism" is well resolved by the clarification***. The authors tune down their claim and clarify that "The superiority of domain-switch (without teacher or prompter) is not because it learns the domain-general knowledge better. Instead, it is because domain-switch learner better suppresses the domain-specific knowledge". This is reasonable and much clearer.

***One additional comment is that the introduction does not fully highlight this point***. The corresponding text should be modified. 

**Second**, ***the evaluation is well clarified.*** ***One suggestion*** is the main paper should clearly mention the experiments in the appendix. These results are helpful to understand the effectiveness of the proposed method.

In addition, I notice that the experimental results on the ***newly-introduced test sets (Aircraft and Traffic Sign) also show the effectiveness of DSL***. ***It would be better*** if the authors could add them in the revised paper (e.g., appendix), which helps understand the effectiveness of DSL.

Based on the above points, I am inclined to accept this paper. 
summary_of_the_review: This work bases on the domain switch learning (DSL) manner. Around this, the work proposes two modules (i.e., domain-specific prompter and the domain-general teacher). Two modules are reasonable and needed under the DSL. 

The major concern is the working mechanism of DSL is not well explained. More discussion/explanation could help eliminate the concern.  Moreover, experimental evaluation needs clarification.

***---Post Rebuttal---*** After reading the response and revised paper, my major concerns are well addressed by the authors.
review_title: null

Meta-review: This paper proposes an approach to improve cross-domain generalization in few-shot learning, using an objective that attempts to fight overfitting on the observed domain at any given iteration while maintaining the general learned information so far from all domains. The approach uses a domain-cycling procedure, where each iteration sees a single domain and, pseudo-labels coming from predictions of a previous iterate of the model and from a parameter-averaged general model are used in a combined training objective.

Three of the reviewers support acceptance (one strongly), while the fourth leans weakly towards rejection, despite an extensive response from the authors that include new results. One concern was a lack of comparison on Meta-Dataset, which the authors went some way towards addressing during the rebuttal, though they also argued Meta-Dataset couldn't really support the kind of cross-domain evaluation they were targeting. The reviewer was not convinced by the authors' argument, and I too am not, in particular when you consider that Meta-Dataset evaluations now often include evaluations on MNIST and CIFAR-10/100, in addition to MS-COCO and TrafficSigns (all not included in the training split of Meta-Dataset). That said, the experimental protocol favored in the authors' experiments certainly is sound and challenging for cross-domain generalization, so I'd hesitate to penalize them for that alternative choice.

Overall, I find the ideas behind this work neat, interesting and well motivated. Even if the basic ideas aren't completely novel, I found their combination thought provoking and creative.

Therefore, in the end, I feel this work will be beneficial to the body of literature on few-shot learning and would merit to appear at ICLR.
Decision: Accept (Poster)