Switch to Generalize: Domain-Switch Learning for Cross-Domain Few-Shot Classification
Authors: Zhengdong Hu
Yifan Sun
Yi Yang
Venue: ICLR 2022 Poster (2022)

Abstract: This paper considers few-shot learning under the cross-domain scenario. The cross-domain setting imposes a critical challenge, i.e., using very few (support) samples to generalize the already-learned model to a novel domain. We hold a hypothesis, i.e., if a deep model is capable to fast generalize itself to different domains (using very few samples) during training, it will maintain such domain generalization capacity for testing. It motivates us to propose a novel Domain-Switch Learning (DSL) framework. DSL embeds the cross-domain scenario into the training stage in a ``fast switching'' manner. Specifically, DSL uses a single domain for a training iteration and switches into another domain for the following iteration. During the switching, DSL enforces two constraints: 1) the deep model should not over-fit the domain in the current iteration and 2) the deep model should not forget the already-learned knowledge of other domains. These two constraints jointly promote fast generalization across different domains. Experimental results confirm that the cross-domain generalization capacity can be inherited from the training stage to the testing stage, validating our key hypothesis. Consequentially, DSL significantly improves cross-domain few-shot classification and sets up new state of the art.

Published as a conference paper at ICLR 2022
SWITCH TO GENERALIZE: DOMAIN-SWITCH LEARN- ING FOR CROSS-DOMAIN FEW-SHOT CLASSIFICATION
Zhengdong Hu 12 Yifan Sun?, Yi Yang"
1 State Key Laboratory of Industrial Control Technology, Zhejiang University
2 Baidu Research, China
CCAL, College of Computer Science and Technology, Zhejiang University
Izhengdonghu, yangyi @zjuu.dd..
edu. cn, sunyf158tsinghie org.
ABSTRACT
This paper considers few-shot learning under the cross-domain scenario. The cross-domain setting imposes critical challenge, ie., using very few (support) samples to generalize the already-learned model to a novel domain. We hold a hypothesis, i.c., if deep model is capable to fast generalize itself to different domains (using very few samples) during training, it will maintain such domain generalization capacity for testing. It motivates us to propose novel Domain- Switch Learning (DSL) framework. DSL embeds the cross-domain scenario into the training stage in a "fast switching" manner. Specifically, DSL uses single domain for a training iteration and switches into another domain for the following iteration. During the switching, DSL enforces
constraints: 1) the deep model should not over-fit the domain in the current iteration and 2) the deep model should not forget the already-leamed knowledge of other domains. These two constraints jointly promote fast generalization across different domains. Experimental results confirm that the cross-domain generalization capacity can inherited from the training stage to the testing stage, validating our key hypothesis. Consequentially. DSL significantly improves cross-domain few-shot classification and sets up new state of the art.
INTRODUCTION
This paper challenges a realistic problem, ie.,, the cross-domain scenario, for few-shot learning- Basically, the few-shot learning task uses classifier learned on the training set to recognize novel classes with very few support samples. In real-world applications, there is usually a domain gap between the training samples and the support samples Tseng et al. (2020); Guo et al. (2020); Chen et al. (2019). The domain gap further imposes a critical challenge: since the support samples are very rare, they do not suffice for mitigating the domain gap between the support set (for novel classes) and the training set. Consequentially. the domain gap significantly compromises the recognition accuracy of the novel classes. Therefore, learning model with strong cross-modality generalization capacity is important for cross-domain few-shot learning Tseng et al. (2020).
We argue that may enhance the desired cross- mross-modality generalization by "learning-to generalize". To be more specific, we hold hypothesis intuition that if a deep model learns to fast generalize itself different domains (using very few samples) during training, will maintain the good domain generalization capacity for testing.
We model our intuition into a novel Domain-Switch Learning (DSL.) framework, illustrated in Fig. 1. DSL uses multiple (M > 1) domains to construct cross-domain training scenario in a "fast switching" manner. Instead of simply mixing all the domains construct a mini-batch (Fig. (a)), DSL includes only single domain into every training iteration and switches to another domain for the following iteration (Fig. (b)). Moreover, each iteration contains very few samples per category, so to imitate the few-shot setting. Therefore, after every switch, the deep model crosses into domain different from the former one, yielding the cross-domain few-shot scenario.
"Zhengdong Hu makes his part of work during internship in Baidu Research.
"orresponning author.

---

Published as a conference paper at ICLR 2022
domain-1
general
specific
domain-1
general
specific
domain-2
general
specific
domain-2
general
specific
domali--M
general
specific
TM
domain-M
general
specific
domain-1
general
specific
Twe
exclude
learned
learned knowledge
knowledge
domain-mix
(b) domain-switch
The comparison between Domain-Switch Learning (DSL) and a domain-min pipeline. (a) mixes up all the M domains in each training iteration. The deep model memorizes all the domain- specific knowledge (apart from the domain-genera knowledge), which hinders cross-domain gen- eralization. (b) DSL includes only single domain into training iteration and switches another domain for the following iteration. Since the domain-specific knowledge does not fit other domains, the deep model is prone discarding the domain-specifie knowledge when switching another domain. We note that in our experiments, we use multiple relatively small datasets e.g. CUB Welinder al. (2010), Cars Krause al. (2013), Places Zhou al. (2018), and Plantae Horn al. (2018) as the switchable domains while using mini- Mumi--mmaeNNet as a basic training set which appears in each iteration.
Figure I:
An important advantage of this domain-switte learning manner is that it can suppress learning from domain-specifte knowledge, as illustrated in Fig. 1. Specifically, we consider each domain contains both domain- general and comain-speecife knowledge. If we mix all the domains for each training iteration (Fig. (a)). the deep model may memorize all the domain-speent comain-generas knowledge. Consequentially, during testing, the comain--specffi knowledge hinders generalization towards the novel domain and thus compromises few-shot classification accuracy. In contrast, DSL (Fig. (b)), the model only learns from one single domain an iteration. Since the Nomain-specinn knowledge of the current domain does not fit the following domain, the deep model is prone to discarding these conain-specific knowledge during the following iteration.
knowledge, as well as the
To further promote cross-domain generalization in this domain-switch learning scheme, DSL en- forces two constraints as follows:
1) The deep model should not over-fit the domain in the current iteration, because over-fitting the current domain makes the model memorize much of the domain-specific knowledge. The first con- straint implemented with domain-specife prompter module (consisted of multiple prompters). After the model learns from the i-th training domain D; and gets updated, its prediction accuracy on D; increases and is relatively high. We store this edition of model as the prompter for D,. Next time before the training domain switches to D; again, the model just gets updated from D and becomes relatively inaccurate on D,. Given the accurate prompter and the inaccurate learner (w. D.), we average their prediction for supervision so that the penalty on the learner will be suppressed. The details are be accessed Section 3.3.
2) The deep model should not forget the already-learned knowledge of former domains, so that next time the model crosses into these domains again, can directly re-use the corresponding knowl- edge for prediction. The second constraint is implemented with a domain- Specifically, we collect several historical models and average their parameters to get a mean model. During training, we use the softmax prediction of the mean model the auxiliary supervising sig- nals for the learner (apart from the ground truth label). The mean model serves as teacher distilling the already-leamde bias towards any single domain, we name as a domain-genenn teacher. The details of the domain- general teacher module be accessed in Section 3.4.
teacher module.
knowledge of former domains to the learner. Since this teacher has obvious
2

---

Published as a conference paper at ICLR 2022
These two constraints achieve complementary benefits for DSL, and jointly reinforce the cross- domain generalization. Extensive experiments under four cross-domain scenarios show that DSL consistently improves cross-domain few-shot learning and achieves performance par with the state-of-the-al methods.
Our main contributions are summarized as follows:
We propose a novel Domain-Switch Learning (DSL) framework for cross cross-domain few-shot learn- ing. DSL uses multiple domains for training and switches the domain in consecutive training itera- tions. It provides a cross-domain learning scenario where the deep model learns to generalize across different domains.
Under the DSL framework, we integrate two modules, i.e., the domain-specific prompter and the -omeinl teacher. These two modules achieve complementary benefits for DSL and jointly reinforce the cross-domain generalization.
We conduct extensive experiments validate the effectiveness of the proposed DSL. Experimental results confirm that the fast generalization capacity can be inherited from training to testing and thus improves roos--dmmii few-shot classification. On all the four popular benchmarks, DSL achieves performance on par with the state of the art.
2 RELATED WORKS
Few-Shot Learning. Few shot learning methods could be roughly categorized into two branches: meta-learning and fine-tuning methods. Some meta- metazlleamiggSttorras & Estrach (2018); Mishra et al. (2018); Ren et al. (2018); Snell et al. (2017); Sung al. (2018); Vinyals et al. (2016) methods map the few-shot samples into non-linear embedding space and evaluate the similarity between support samples and the query samples. Recently, fine-tuning methods Chen al. (2019); Tian et al. (2020); Afrasiyabi et al. (2020) propose to pre-train the model on the training set (typically in a classification learning approach), and fine-tune the classifier on the novel classes. The fine- tuning methods achieve competitive performance compared with the state-of-the-arr meta-learning methods. This work considers the sho- learning task under the cross-domain scenario and adopts the fine-tuning pipeline as the baseline.
Domain Generalization and Adaptation may be viewed as two closely related concepts. Do- main adaptation uses abundant (unlabeled) samples to adapt the already-learned model from the source domain to the target domain Ganin & Lempitsky (2015); Tzeng et al. (2017); Long et al. (2017; 2015; 2016); Tzeng al. (2014). In contrast, domain generalization aims to generalize the knowledge from multiple source domains to the target domain without learning any samples of the target domain during the training stage Balaji et al. (2018); Li al. (2018). In cross-domain few- shot learning, there is a domain gap between the training set and the testing set. To promote the cross-domain generalization, Tseng et al. (2020) considers the challenge is more of domain gener- alization problem than a domain adaptation problem. This consideration is reasonable because the support samples are very rare and do thus not suffice for typical domain adaptation.
From the viewpoint of domain generalization, FWT Tseng et al. (2020) proposes to simulate the cross-domain scenario to improve the domain generalization capacity. indeed inspires us of the multi-domain training strategy. In spite of sharing the multi-domain training strategy, DSL signifi- cantly differs from FWT: First, FWT uses changes the training domain spoch-by-epoet It does not rely on fast switching for better generalization. In contrast, DSL is featured for the novel fast switch- ing manner and finds that this manner promotes learning domain-general knowledge. Second, DSL further considers two constraints in multi-domain training strategy, i.e., the deep model should not over-fit fit every single domain and keep recognition on learned knowledge shared with other domains. Finally, DSL achieves superior cross-domain effect than FWT (Section 4.2).
Catastrophic forgetting. Catastrophic forgetting plays an important role in the mechanism of DSL. When the deep model is trained sequentially multiple tasks (domains), there is tendency that the knowledge learned on former tasks (domains) is abruptly lost, yielding the so-called catastrophic forgetting Kirkpatrick et al. (2017); McCloskey & Cohen (1989); French (1999).
In DSL, when the deep model switches from domain to another, it forgets much the knowledge on the former domain, due catastrophic forgetting. Forgetting the domain-speciffe knowledge is
3

---

Published as a conference paper at ICLR 2022
beneficial, while forgetting the domain-genee knowledge is harmful. In response, next time the deep model comes across the former domain again, we 1) supplement it with the corresponding knowledge with domain-specific prompter to prevent over-fitting the domain-specific knowledge and 2) use a domain-genere teacher to reinforce the common knowledge shared among all the for- mer domains. In word, in its mechanism, DSL actually utilizes catastrophic forgetting improve cross-domain generalization.
METHODS
3.1 PRELIMINARIES
In the cross-domain few-shot learning task, there is significant domain gap between the training set and the testing set. The testing set consists the support set and query set, which are sampled from the same novel categories (which are unseen in the training set). The support set provides very few labeled samples for recognizing the unlabeled samples in the query. Concretely, the support set contains C' classes and each class has K samples, which is a C-way K'shot setup in few-shot learning.
Our work is based on a popular fine-tuning pipeline Chen et al. (2019); Tian et al. (2020); Afrasiyabi et al. (2020). The fine-tuning baseline uses the training set D to Iearn a deep model consisted of a feature encoder F and feature classifier Y. F is parameterized with and Y is parameterized with a weight matrix W. During training, given a sample æ, the feature encoder first maps æ into a feature F(x; 0) E Rd. Then the feature classifier Y predicts the softmax probability of belonging each training category P(ilæ; O, W)
exp(w=" oxp(u="[(x::0) F(=:0))
Σ, e F(=:0)) (w, is the weight vector for the i-th category). The training is supervised with a popular classification loss function, i.e., the cross-entropy loss, which is formulated as:
exp(w, (w) F():0))
(CE(*) log(P(y) W)) =llog
Σ, exp(w) F(x:0))'
where y is the label for æ, wy is the weight vector corresponding to the ground-truth category.
During the testing stage, we replace the feature classifier Y with new linear classifier L. L is a C-way classifier, corresponding to the total number of classes in the support set. We freeze the parameters of the feature encoder F and fine-tune the parameters of L with samples from the support set. Finally, we use the feature encoder F and the learned as the predictor for the query.
This paper focuses on the training stage, i.e., learning the feature encoder F and the classifier Y on the training set.
3.2 OVERVIEW OF DOMAIN-SWITCHH LEARNING
Domain-Switch learning (DSL) employs M datasets to construct the training set, i.e., D (D1, D2...., DM}, }. D; is an individual domain, which may be selected from some popular few- shot datasets, e.g., CUB Welinder al. (2010), Cars Krause al. (2013), Places Zhou et al. (2018), Plantae Horn et al. (2018). These switchable training sets are relatively small, and does not suffice for learning discriminative feature encoder. Therefore, following Tseng et al. (2020), we employ a relatively large-scale dataset mini-Imagel ImageNet Deng et al. (2009) as a "basic" training set. During training. each mini-batee batch actually contains samples from the basic mini- Imai-ee and a switchable training set D,. We note that such implementation does NOT contradict our motivation of using single domain for each training iteration, because we may view the mixture of mini-ImageNet and D, as single "compound" domain.
During training, DSL uses samples from single switchable domain D, (and the mini-ImageNNe) for each iteration, illustrated in Fig. 2. Since mini- ImageNe appears for every iteration, we omit in Fig. 2 for simplicity. The domain-switching process is formulated as:
B, - D. D, E D; i = M
M
(2)
where By is the training batch at the j-th iteration, D; is the i-th domain, I.] is the rounding down operation. We define enumeration of all the M domains switching round.
4

---

Published as a conference paper at ICLR 2022
iteration
Reweighted CE Loss
Binary KLD Loss
Reweighted CE Loss
Press-Entropy
Learner
Teacher
Prompter
Ap+ (1 A)p"
iteration i (cunrant)
Dj
iterations www.
iteration (i-1)
Di
model updated by Dy
learner softmax
softmax
iteration (i-M+1)
model updated Da
Feature encoder
weighted average
Iteration (i-M)
D model updated by Dj
Figure The overview of Domain-Switch Learning. DSL uses a single domain for training iteration and switches to another domain for the following iteration. During the current iteration (the red area), DSL uses a domain-specific prompter and a domain-general teacher to assist the learner. The prompter prevents the learner from over-fitting the domain-specific knowledge of the current domain using a Re-weighted Cross-Entropy loss (Section 3.3). The teacher helps the learner memorize the domain-general knowledge shared among former domains using novel Binary Kullback-Leilerr divergence loss (Section 3.4).
In order to further improve the cross-domain generalization in the above "fast switching", DSL enforces two constraints: 1) the deep model should not over-fit the domain the current iteration and the deep model should not forget the knowledge of other domains. These two constraints are implemented with Domain-Specific Prompter (Section.3.3) and a Domain-General Teacher (Section.3.4) module, respectively.
3.3 DOMAIN-SPECIIN PROMPTER
Definition of the prompter:
Let consider in the j-th iteration, the model has already learned from D, and gets updated. We store this edition of model as the prompter for the (j + M iteration (when the learner is learn from D, again), illustrated in Fig. 2. The prompter aims to prevent the learner from over-fitting the current domain D..
Given sample in current domain, DSL averages the softmax prediction of the prompter (Ply 0", WW), where the upper-script P denotes the prompter) and the softmax prediction of the learner (P(y) [x; 0, W)), which is formulated as:
XP(y x; W) + (1 ((y)))())x; 0P, WP),
arg max P(ila, 0", WP) =y
P(y|æ; 0, W),
otherwise
where is a hyper-parameter. The condition arg max P(i|e; 0", WP) = y is to ensure the prompter makes the correct prediction.
With the weighttt softmax prediction Pb, the cross-entropy loss for supervising the classification transformed into a Re-weighted Cross-Entropy loss (RCE loss in Fig. 2):
CRCE - log((P)
(4)
We explain how the domain-specinfe prompter prevents the learner from over-fitting the current domain with two following remarks (Remark1 and Remark2).
Remark1: Compared with the learner, the prompter has relatively higher predicted probability on the ground-truth category, i.e., Plylæ: 0?, WP) 2 P(ylæ; W).

---

Published as a conference paper at ICLR 2022
08
08
accuracy
accuracy
04
0.4
02
learner on (2())
teacher D,(*)
D.(*)
teacher on D2
learner on D2 learner
prompter on 0y(*)
teacher
350
epochs
opochs
(a) Comparison between prompter and learner
(b) Comparison between teacher and learner
Figure 3: Analysis on the predictions of prompter and teacher. "D1(*)" indicates that the current training domain D1- (a) The prompter fits D₁ better than the learner before the learner gets updated on D1. (b) The teacher memorizes the general knowledge from all the domains and thus fits the former domains (e.g., D2, D3) better, while the learner fits better D1.
Remark1 is intuitive. Without losing generality, we assume the current learning domain in Fig. is Dj. The domain-specific prompter is the former model updated by D2 and thus fits D1 well. In contrast, the current learner newly updated by Das and forgets some knowledge of D1, due to the well-known "catastrophic forgetting". Therefore, the prompter has higher predicted probability on the ground-truth category than the learner does. We empirically validate the above analysis in Fig. 3, from which we clearly observe that the prompter has higher prediction accuracy than the learner.
Remark2:
Since Ply|æ; 0", WP) 2 P(ylæ; 0, W) (i.e., Remark1), the re-weighted procedure in Eq. alleviates the penalty on the learner, which is beneficial to exclude domain-specifti knowledge. Please refer to the Appendix for the conceptual proof. Consequentially, the prompter counters over- fitting the current domain.
3.4 DOMAIN-GENERAL TEACHER
Definition of the domain-general teacher: We recall that switching round is a period within which all the M domains are enumerated. As illustrated in Fig. 2, we average the parameters of all the M encoders updated in the past switching round to generate the general teacher for the current domain D., which is formulated
TT
1 0
(5)
The teacher aims to preserve and distill the domain-general knowledge learned in former domains the current learner.
We note that the domain- domain-general teacher is different from the popular temporal moving average for semi-supervised and unsupervised task Tarvainen & Valpola (2017); He al. (2020); Grill et al. (2020). In temporal moving average, the teacher absorbs parameters from all the historical models (with higher weighting factor for the more recent model). contrast, our domain general teacher absorbs parameters from only M historical models with equal weighting factor.
Remark3: When the learner gets updated from D,, it has higher prediction accuracy on D; than the teacher does. Meanwhile, the teacher has higher prediction accuracy former domains Dj(j + i).
Remark3 is intuitive. is because the teacher absorbs knowledge from M historical learners, there- fore preserving historical knowledge of the former domains. We empirically validate this intuition in Fig. 3 (b), which consistent with Remark3. Morcover, since the teacher averages the parameters of all the M historical learners, it has obvious bias towards any single domain. Therefore, the teacher has relatively smaller variation of accuracy on all the domains, as shown in Fig. (b).
Binary Kullback-Leibler divergence loss. Given sample in the current domain, output of the teacher P(ylar [ OT, WT) as an auxiliary supervising signal for the learner. To this end, propose novel Binary Kullback-Leibler divergence (KI.)) Loss follows:
the

---

Published as a conference paper at ICLR 2022
Methods
CUB
Cars
Places
Plantae
MatchingNet
$5.89:0.
30.77 = 0.47
49.86 0.79
32.70 0.60
RelationNet
S
0.77
29.11 0.60
4..64
33.17 0.64
MatchingNett+FT
36.61 0.53
29.82 + 0.44
$
34.48 + 0.50
Net+Fion
0.77
28.63 + 0.59
50.68 + 0.87
33.14 0.62
RelationNet ATA
43.02 0.40
31.79 = 0.30
51.16 0.50
33.72 + 0.30
MatchingNNt
+.55
28.96 = 0.45
49.01 + 0.65
33.21 0.51
RelationNet
m
0.59 44.33
29.53 = 0.45
47.76 0.63
33.76 - 0.52
MatchingNet + LFT
43.29
0.59
30.62 = 0,48
52.51 0.67
35.12 + 0.54
RelationNet + LFT
m
48.38
0.63
32.21 = 0.51
50.74 = 0.66
35.00 + 0.52
DSL
50.15 + 0.80
37.13 0.69
53.16 + 0.88
41.17 0.80
Table 1: Comparison with the state of the arts for 5-way I-shot task. "FT ("LFT") denotes method
in Tseng et al. (2020). "ATA" denotes method in Wang & Deng (2021), "s"("m") denotes single- domain (multi-domain) learning.
Methods
CUB
Cars
Places
Plantac
MatchingNNt
51.37 + 0.77
38,99 = 0.64
63.16
46.53 + 0,68
RelationNet
57.77 0.69
37.33 = 0.68
0.76
44.00 0.60
MatchingNett+FT
55.23
41.24 0.65
6.75
41.69 + 0.63
RelationNet+FF
59.46 0.71
39.91 = 0.69
45.08 0.59
RelationNet ATA
42.95 + 0.40
66.90 0.440
45.32 + 0.30
NSAE
0.76
54.91 + 0.74
71.02 + 0.72
59.53 0.74
Matching
+
39.87 = 0.51
61.82 + 0.57
47.29 0.51
RelationNet
m
62.13
0.74
40.64 = 0.54
64.34 + 0.57
46.29 0.56
MatchingNet LFT
61.41 0.57
43.08 = 0.55
64.99 0.59
48.32 + 0.57
RelationNet + LFT
m
64.99 + 0.54
43.44 - 0.59
67.35 + 0.54
50.39 + 0.52
DSL
m
73.57
0.65
58.53 0.73
74.10 + 0.72
62.10 0.75
Table 2: Comparison with the state of the arts for 5-way 5-shot task. "FT" ("LFT") denotes method
in Tseng et (2020). "ATA" denotes method in Wang Deng (2021), "s"("""") denotes single- domain (multi-domain) learning.
pT LBKLD PTT log + - (1- log 1
PT
(6)
Ps
where P, = P(ylar; O. W) and PT P(ylar OT, WT).
The difference between the proposed BKLD loss and popular KLDD loss is: KL.D aims to make the student consistent with the teacher at all the entries of the softmax prediction. In contrast, the proposed BKLD loss only focuses on the consistency at the ground-truth entry. In the experiment in Section 4.4, empirically show BKLD loss is better than the KLD loss for DSL.
Overall Training. We recall that besides switchable domains, mini- mmae--mm is employed as "basic" training set, which is sampled into all the training iterations. For mini- mmai-em, adopt the cross-entropy loss as basic loss Basse- The overall training loss is as follows:
L L + a LRCE + (1 - a)) LARLD
where a is a weighting factor.
EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
We evaluate the proposed DSL five datasets: i.e., 4 fine-grained datasets (CUB Welinder et al. (2010), Cars Krause et al. (2013), Places Zhou et al. (2018), and Plantae Hom et al. (2018)) and popular large scale dataset mini-ImageNe Deng al. (2009). We use ResNet-10 He al. (2016) without pretraining as the backbone network. Please refer to the Appendix for more experimental setups and implementation details.
experimnan
7

---

Published as a conference paper at ICLR 2022
Methods
CUB
Cars
mini-ImageNet (single)
63.76 + 0.60%
51.21 + 0.40%
domain-mis (multi)
64.73 0.68%
51.78 0.60%
domain-switch (multi)
66.33 + 0.71
53.61 + 0.68%
Table 3: Comparison between two multi-domain learning scheme for 5-way 5-shot task.
4.2 COMPARISON WITH THE STATE OF THE ARTS
We compare DSL with the existing state of the arts on 4 benchmarks. We adopt the leave-one-out setting, i.e, one of the fine-grained datasets is chosen as testing set and the other 3 datasets (out of the datasets) are used as the training sets. Meanwhile, we employ mini Amai-em as "basic" training set, which is available during the whole training stage. The results of 5-way 1-shot and 5-way 5-shot are summarized Table and Table respectively.
From Table 1, we clearly observe the superiority of DSL under 5-way 1-shot classification. First, comparing DSL with all the single-domain training methods, we find that DSL achieves significant superiority. For example, DSL outperforms the most competing single-domain method ("Relation- Net + ATA (s)") Wang Deng (2021) by +7. 13%, +5.34% +2%, +7.45% on CUB, Cars, Places and Plantae, respectively. Second, while the multi-domain training methods achieve considerable improvement (over their single-domain counterpart), DSL still outperforms them with clear mar- gin. For example, DSL outperforms the most competing multi domlin method ( ("RelationNet + LFT (m)") by +1.77%, +4.92% +2.42% 17% on CUB, Cars, Places and Plantae, respectively.
From Table classification. Most of the competing methods already appear in the comparison of 5-way 1-shot setting (Table 1). Compared with all these methods (except NSAE Liang al. (2021)), DSL main- tains significant superiority on all the four benchmarks. For example, DSL surpasses "RelationNet LFF(())"Tseng et al. (2020) by +8.58% +15.09% +6.75% +11.71 % on CUB, Cars, Places and Plantae, respectively.
clearly observe that DSL achieves competitive accuracy under the 5-way 5-shot
NSAE in Table 2 is very recent and competitive single-domain training method. It splits the support samples of the same category into two sub-sets for similarity learning. Therefore. it requires more than support samples for each category and is incompatible to the 1-shot setting. With this unique comilarity-learning manner, it achieves very competitive performance (e.g.. 68.51 % on CUB). Compared with NSAE, DSL obtains consistent superiority For example, DSL surpasses NSAE by 5.06% on CUB.
4.3 THE EFFECTIVENESS OF THE DOMAIN-SWITHH OPERATION
A key characteristic of DSL is the domain-switch operation. In Table 3, compare the single- domain baseline (on mini-ImageNNt),, the domain-mix and the proposed domain-switch scheme (without domain-general teacher and the domain-specifie prompter CUB and Cars. We draw two following observations:
First, compared with single-domain training, the domain-mix learning scheme only brings slight improvement. Although it employs multiple domains (as well as multiple datasets) for training, the benefit of more training data trivial (less than 1% improvement). We infer because in the domain-mix learning scheme, the disadvantage of learning domain-specific knowledge largely offsets the advantage of more training data.
Second, comparing "domain-mis with "domain-switch". we observe that domain-switeh is su- perior multi-domain learning manner. Specifically, domain-switch surpasses domain-mix by 1.60% accuracy under 5-way 5-shot setting on CUB. It is consistent with our conceptual analysis, i.e., the domain-switch learning manner better ignores the domain-specific knowledge, therefore improving the cross-domain generalization.
4.4 ABLATION STUDIES
Ablation the prompter and teacher module. While the domain-switch framework already improves cross-domain generalization, DSL uses the domain-specific prompter and the domain- general teacher to prevent over-fitiing domain-specific knowledge and to reinforce the domain-
8

---

Published as a conference paper at ICLR 2022
5-way 1-shot
5-way 3-shot
5-way 1-shot
5-way 5-sbot
78
73.17
74
73.57
51
30.15
73
51
30:15
73
49.36
7LB8
40.32
72
7198
70
70
BKLD
KLD
no teacher
BKLD
KLD
no teacher
round 2 rounds 3 rounds
round 2 rounds 3 rounds
(a) Comparison between BKLD loss and KLD loss
(b) Comparison between different switching rounds
Figure 4: Analysis on Binary KLD loss and the number switching round of teacher module.
Components
5-way 1 I-shot
5-way 5-shot
domain-switch
46.34
0.73%
66.33 = 0.71%
domain-switch specific prompter
49.05 + 0.60%
71.13 + 0.68%
domain-switch + general teacher
48.88 0.61 %
70.45 0.71%
All (DSL.
50.15
0.80 %
73.57 + 0.73%
Table Ablation studies of our proposed method on individual components.
general knowledge, respectively. We investigate their benefits through ablation in Table 4, from which we draw two observations:
First, both the domain-spectioo prompter and the domain- general teacher are beneficial. Based on the doitin-switch operation, we find that adding the prompter or the teacher brings accuracy improvement. For example, under the 5-way 5-shot setting on CUB, adding the prompter and the teacher increases the accuracy by 4.80% and 4. 4.12%, respectively.
Second, the domain-specife prompter and the domain- semmain-general teacher are complementary to each other. While both modules already bring independent improvement, combining them ("All (DSL)") yields even higher accuracy. It validates that these two modules achieve complementary benefits (i.e., countering domain -specfcc knowledge and promoting domain-generss knowledge).
Analysis some important contigurations of the teacher module. We investigate two configura- tions: 1) using a novel Binary KLD loss (instead of the popular KLD loss) for knowledge distillation and 2) using the historical models within single switching round for generating the teacher.
Fig. (a) shows that the domain -eenrillgeeeral teacher favors the Binary KLD loss than the canonical KLD loss. Under the 5-way 5-shot setting CUB, the canonical KLD barely improves over "no teacher", while the BKLD surpasses KL.D by 1.69% accuracy. We infer it is because the knowledge from other domains is not accurate for the non-target classes. Therefore, foreing the learner to approximate the teacher w.r.t all the non-target classes is inappropriate.
Fig. 4 (b) shows that using more switching rounds gradually compromises the effectiveness of the teacher, compared with using a single round. We infer it is because increasing switching rounds includes some severely out-of-date models, whose knowledge largely diverges from the up-of-date model. Therefore, we use a single switching round for generating the teacher model.
5 CONCLUSION
This paper proposes a novel Domain-Switch Learning (DSL) method for cross-domain few-shot classification. DSL embeds the cross-domain scenario into the training stage in fast switching manner. We show that during the domain switching procedure, the deep model favors domain- general knowledge and is prone to ignoring the domain-specific knowledge, so to fast adapt itself to different domains. Moreover, DSL employs a domain-spechi prompter and domain domain-general general teacher module to further promote the cross- cross-domain generalization capacity. Experiments conducted on multiple benchmarks demonstrate that DSL improves cross-domain few-shot classification and the achieved results on par with the state of the arts.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper can help to improve the model generalization capacity in recognizing novel classes with limited labeled samples (e.g., the rare birds, the rare plants,etc) under a significant domain gap between training sets. It may be applied to some environmental protection projects which require monitoring on the diversity of species. Specifically, can help recognize some rare species in the wild reserve. We will explore more application scenarios of few-shot learning, well as cross- domain few-shot learning. Moreover, we will try to improve the reliability of few-shot learning systems to reduce the potential problems.
REPRODUCIBILITY STATEMENT
The DSL is reproducible. In main text, we describe the utilized datasets DSL, í.e., fine- grained datasets (CUB, Cars, Places, and Plantae) and popular large scale dataset mini- Mini--mmaeNet. We provide the details about the experimental implementation, the proof of the proposed remark and the analysis of some hyper parameters in appendix.
REFERENCES
Arman Afrasiyabi, Françoiss Lalonde, and Christian Gagné. Associative alignment for few-shot image classification, 2020.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gen- eralization using meta-regularization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. International Conference on Learning Representations, 2019. 2019.
Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Patlern Recognition, pp- 248-255, 2009.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pp. 1180-1189, PMLR, 2015.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Yunhui Guo, Noel C. Codella, Leonid Karlinsky, James V. Codella, John R. Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Grant Van Hor, Oisin Mae Aodha, Yang Song. Yin Cui, Chen Sun. Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset, 2018.
10

---

Published as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabssa-Barwwiska. al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3222 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine ane-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) Workshops, June 2013.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C. Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5400-5409, 2018.
Hanwen Liang, Qiong Zhang, Peng Dai, and Juwei Lu. Boosting the generalization capability in cross-domain few-shot learning via noise-enhanced supervised autoencoder, 2021.
Bingyu Liu, Zhen Zhao, Zhenpeng Li. Jianan Jiang, Yuhong Guo, and Jieping Ye. Feature transfor- mation ensemble model with batch spectral regularization for cross-domain few-shot classifica- tion. arXiv preprint arXiv:2005. 200::2005.08463,, 2020.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference machine learning, pp. 97-105, PMLR,, 2015.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. arXiv preprint 1o0iv: 2016.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. arXiv preprint 1rXiv:1755.10667 2017.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12 (1):145-151, 1999.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-leaming for semi semi-supervised few-shot classifica- tion. In International Conference on Learning Representations, 2018
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference Learning Representations, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp- 4080-4090, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1199-1208. IEEE, 2018.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. arXiv preprint arXiv:1703.0172 2017.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: good embedding is all you need?, 2020.
11

---

Published as a conference paper at ICLR 2022
Hung- Yu Tseng, Hsin- Hsin-Ying Lee, Jia-Bin Huang, and Ming Husan Yang. Cross-domain few-shot classification via learned feature-wise transformation. International Conference on Learning Representations, 2020.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprini arXiv: 14123444, 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7167-7176, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3637-3645, 2016.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task aug- mentation. arXiv preprint arXiv:2104.14:: 2021.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report
California Institute of Technology, 2010.
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. Places: 10 million image database for scene recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 40(6): 1452-1464, 2018.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
In the appendix, we supply the details which are not described in the main text due to space limita- tion. Section A.I, we provide the details about the experimental implementation. In Section A.2, we prove the Remark2 in Section 3.3. In Section A.3, we investigate DSL on different backbones. In the main experiments, DSL and all the competing methods adopt the standard inductive infer- ence. In Section A.4, we further provide some comparison under the transductive setup (e.g., with a label propagation method Liu et al. (2020)). Section A.5, we analyze the impact of some hyper-parameters.
A.1 IMPLEMENTATION DETAILS.
We use ResNet-10 He et al. (2016) without pretraining as the backbone network. When training the model on the training set, we use the Adam Kingma & Ba (2015) optimizer. We train the model with 400 epochs and set the initial learning rate as le-3. In each training iteration, the mini-batch size switchable domain and the "basic" training set are both 64, In each mini-batch, we randomly sample 16 classes from each domain (4 images per class).
evaluate the classification accuracy over 1000 experiments on test set. When fine-tuning model on the support set, randomly sample C classes and K samples per class, according the C-way K-shot setting(e.g. C=5, K=1 or 5). We use the SGD Qian (1999) to optimize the linear classifier layer and the initial learning rate is 0.01. The weight decay of SGD is 0.001 and the SGD) momentum is 0.9.
A.2 REMARK2 PROOF.
We first recall the domain semeiff prompter in Section 3.3, Given a sample æ current domain, we average the softmax predictions of the prompter P(y|ar; 0?, WP) and learner P(y ax; 0, W) get re- re-weighted softmax prediction P. It can prevent the learnr from over-fitting the current domain.
Remark2: Given P 0", WP) 2 P(ylæ; 0, W), we have the re-weighted softmax prediction P > (z O, W). It thus reduces final prediction errors (on the current domain), alleviating the penalty on the learner. The proof is as illustrated by:
Plyl:; OP, WP)
P(yl:; O, W)
A P( 0" WP
A) Plyla: O", WP)
P(y m: O. W
A
S
A- P(y/ 0T
WP) + (1 - A) Plylæ; Or, WP)
Ply x; W)
dCRCE
dCce
dp
d æ; OF, WP)
where denotes the value vector. E [0, 1] is a hyper parameter.
5-way 1-shot
5-way 5-shot
36
baseline
80
baseline
DSL
713
463
70
66.99
45
4003
61.88
60
55,58
35
30
Conv4
Resnet18
Figure 5: Illustration of our proposed method on both shallow or deeper backbone settings.
13

---

Published as a conference paper at ICLR 2022
A.3 DSL ON VARIOUS BACKBONES.
We further analyze the adaptation of our methods on Conv4 and ResNet18, The performance on CUB are shown Fig. 5. We adopt "domain-mix" learning manner as the baseline. We observe that the proposed method can still achieve considerable improvements on both shallow or deeper backbone settings, which proves the robustness the proposed method.
A.4 ADDITIONAL EXPERIMENTS.
We adopt simple label propagation method Liu et al. (2020) for DSL and compare "DSL+LP" with label- proppaattonnn based methods.
Methods
CUB
Cars
Places
Plantac
ATA+
65.31 + 0.40
46.95 + 0,40
72.12 1 0.40
55.08 1 0,40
GNN
69.26 + 0.68
48.91
0.67
72.59 + 0.67
58.36 + 0.68
GNN+LFT
73.11 = 0.68
49.88
= 0.67
77.05
+ 0.65
58.84 + 0.66
DSL LP
76.72 + 0.60 62.21
0.70
77.10 + 0.70
65.70 + 0.75
Table 5: Comparison with the state of the arts for 5-way 5-shot task. "ATA" denotes Adversarial Task Augmentation." "LFT" denotes learning-to-leanen feature-wise transformation.
From Table 5, we observe that when all the methods use transductive setup (e.g., label propagation), DSL achieves competitive accuracy under the 5-way 5-shot classification. 'DSL+LP' outperforms GNN + LFT by +3.61 %, +12.33%, +0.05% +6. 86% CUB, Cars, Places and Plantae, respectively.
A.5 HYPER-PARAMENT
ANALYSIS.
We analyze the impact of three important cepeerparameterssi.. in re- re-predicted procedure(Eg. 3), the in overall loss(Eq. 7) and the temperature coefficient y in computing the predictions P
Plyl xx: 0T, WT) of domain general teacher.
5w55
3-w 1-s
744
5- 1-5 (%)
70
66
16
02
Figure 6: Analysis on the hyper-parameter X.
In Fig. 6, we evaluate the impact of hyper-parameter X, which controls the weight of predictions from domain-specifie prompter in Eq. 3. We observe that the accuracy first increases (when increases from 0 to 0.4) and then decreases (when À further increases to 1.0). We set 0.4 as the weight factor.
5w33
54
5- (%)
71
88
46
0
01
02
03
04
0.5
06
0.7
0.8
09
Figure 7:
Analysis on the hyper-parameter a.
14

---

Published as a conference paper at ICLR 2022
In Fig. evaluate the impact of syper-parameter which controls the weight of Binary KLD loss and RCE loss in Eq. 7. We observe that the accuracy first increases (when a increases from to 0.7) and then decreases (when further increases to 1.0). We set 0.7 the weight factor.
- 5- S4
- 5w lss
74
5- S-saco (%
72
30
68
64
1225
15
1755
2
Figure 8: Analysis on the pyper-parameter μt.
In Fig. 8, we evaluate the impact of hyper-paramett μu, which denotes the temperature coefficient in computing predictions p/ aaylar 0² WT) of domain-geners teacher. When the temperature coefficient μ 2 I, consider that the predictions of domain- -enerall teacher are more smooth and thus transfer more general knowledge shared with other domains. We set μ to vary from 2. observed that the accuracy undergoes increase (when μ increases from 1 to 1.25) and then decrease (when μ > 1.25). Therefore, we set μ 1.25 as the optimized threshold for domain-general teacher.
15


Reviews:
[{"review_id": "vOxNV13hT9", "reviewer": "Reviewer_xu2A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The paper is well written and easy to follow.\n+ The proposed switch learning schedule seems novel for the cross-domain few-shot task.\n+ The proposed Binary KLD Loss and Prompter loss seem to contribute a lot to the accuracy improvement.\n+ The overall framework is consistent and the accuracies on standard benchmarks are acceptable.\n\nWeaknesses\n- According to Table1 and Table2, the authors directly copied the performance of RelationNet_ATA from [a], but why did not compare the performance of TPN+ATA. Compared with TPN+ATA, the strength of the proposed method would be limited, especially for the Places dataset. Could the authors give some explanation and analysis about that?\n\n- Actually, the Tabel3 made me confused about the experimental results. (1) As mentioned in the first paragraph of Sec.4.2, the authors adopted the leave-one-out-setting, does it means the total number of training samples for multi-domain is larger than that for the single-domain? How to ensure the fairness of comparison？ （2）In Table3, I can't align the performance of the domain-mix scheme with other methods. The performance shown in Table3 has already been higher than some baseline models in Table1 and Table2, why?\n\n- As shown in Table3, even though the accuracy of domain-mix was lower than domain-switch, I am still curious about the performance of domain-mix+teacher-regularization. In such a case, you could also use the average of several adjacent models as the teacher network to do the knowledge distillation. Further, if we use the teacher network to calculate both BKLD loss and RCE loss for the domain-mix scheme, how about the performance? \n\n[a] Cross-Domain Few-Shot Classification via Adversarial Task Augmentation", "summary_of_the_paper": "This paper proposes a novel training scheme termed Domain Switch Learning (DSL) to pursue the few-shot learning problem under the cross-domain scenario. In practice, DSL uses the data from one domain for a training iteration and switches to another domain for the next iteration. During the switching, two regularization terms are also introduced to balance the domain-specific knowledge and domain-general knowledge. The extensive experimental result demonstrates the effectiveness of the proposed method on several standard benchmarks.", "summary_of_the_review": "For weaknesses 1 and 2, I hope the author could provide more detailed explanations.\nFor the last one, I am still concerned if the performance gain is provided by the switch scheme or just from the carefully designed BKLD loss and RCE loss.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sItFnvsUQgp", "reviewer": "Reviewer_KRoA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n\n\n**Strengths** \n1. Overall, this paper is easy to follow. The introduction nicely outlines the method.\n2. The domain-specific prompter and the domain-general teacher are straightforward and reasonable. The two modules are needed under the proposed domain-switch learning manner.\n3. Domain-general teacher seems a general module that can be used in other few-shot frameworks (although the experiment does not validate this point).\n4. The experimental evaluation nicely stuides the hyper-parameters.\n\n**Weaknesses** \n\n- **[Working mechanism needs more discussion]** \n\n\tThe proposed Domain-Switch Learning (DSL) includes only a single domain into a training iteration and switches to another domain for the following iteration. The work highlights several times that such a way makes “the deep model favors domain-general knowledge and is prone to ignoring the domain-specific knowledge”. This point is not very clear and needs more validations. \n\n\t***First***, the domain-mix pipeline learns images from different domains in each iteration. In order to learn well from these images, the model tends to learn common information. This also makes the model favors domain-general knowledge. Thus, it is still not clear why domain-switch learning should be better than domain-mix learning.\n\n\t***Second***, In Sec. 3. 2 the work includes a sentence to eliminate the concern of the contradiction when using mini-ImageNet images at each iteration. However, it is still not very convincing.\n\tA. One could also regard each mixed batch in domain-mix learning as a single “compound” domain. \n\tB. Also, following the logic of the motivation, using mini-ImageNet in each iteration could make model memory the information of mini-ImageNet (which is domain-specific). Such mini-imagenet specific information might already be general, which is helpful when testing on bird/car/place images. This might be the reason why including mini-ImageNet images in each iteration helps to learn. \n\n\t***Third***, experimental comparison between domain-switch and domain-mix needs clarification (Please see the first point in the following).\n\n- **[Experimental evaluation needs clarification]**\n\n\t***First***, Table 3 needs more clarification. 1) When reporting results on CUB, does the model use four domain images (mini-ImageNet, Car, Places, Plantae) for training? 2) when using the domain-mix manner, does each mini-batch contain mini-ImageNet images? 3) one possible baseline could be first pertaining on mini-ImageNet, and then finetuning using domain-mix manner or domain-switch manner. 4) do the methods listed in Tables 1 and 2 also use mini-ImageNet image at each training iteration or just finetuning?\n\n\t***Second***, Tables 1 and 2 list basic baselines. GNN (Tseng et al. (2020)) should be included as well. Moreover,  the results of NASE and BSR under the multi-domain setting (“m” in the table) are missing.\n\n\t***Third***, when using the teacher network, why BKLD is better (as shown in Fig4 (a))? It would be better if this work could provide a discussion on this.\n\n- Minor:\n\n\t[-] The captions of Tables 1 and 2 are not clear. The definitions of TF, ATA, s, m are missing.\n\t[-] Summarization of the datasets (domains) used in the experiment could be included in the appendix.\n\n***---Post Rebuttal---*** \n\n**First**, ***the major concern of \"working mechanism\" is well resolved by the clarification***. The authors tune down their claim and clarify that \"The superiority of domain-switch (without teacher or prompter) is not because it learns the domain-general knowledge better. Instead, it is because domain-switch learner better suppresses the domain-specific knowledge\". This is reasonable and much clearer.\n\n***One additional comment is that the introduction does not fully highlight this point***. The corresponding text should be modified. \n\n**Second**, ***the evaluation is well clarified.*** ***One suggestion*** is the main paper should clearly mention the experiments in the appendix. These results are helpful to understand the effectiveness of the proposed method.\n\nIn addition, I notice that the experimental results on the ***newly-introduced test sets (Aircraft and Traffic Sign) also show the effectiveness of DSL***. ***It would be better*** if the authors could add them in the revised paper (e.g., appendix), which helps understand the effectiveness of DSL.\n\nBased on the above points, I am inclined to accept this paper. ", "summary_of_the_paper": "The goal of this paper is to improve the generalization ability of the few-shot model on novel domains. To this end, this work introduces a learning manner named domain-switch learning (DSL). To make DSL feasible, the work further proposes two modules (i.e., domain-specific prompter and the domain-general teacher). The experimental results show the effectiveness of the proposed framework.", "summary_of_the_review": "This work bases on the domain switch learning (DSL) manner. Around this, the work proposes two modules (i.e., domain-specific prompter and the domain-general teacher). Two modules are reasonable and needed under the DSL. \n\nThe major concern is the working mechanism of DSL is not well explained. More discussion/explanation could help eliminate the concern.  Moreover, experimental evaluation needs clarification.\n\n***---Post Rebuttal---*** After reading the response and revised paper, my major concerns are well addressed by the authors. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jyNYW5Wcfxw", "reviewer": "Reviewer_JgKH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n* the high-level intuition of incorporating adaptability into the training scheme is heuristic to the research community\n* the proposed method is applicable to the real-life scenario of few-shot domain shift\n* extensive experiments and ablation studies are conducted to demonstrate the performance of the proposed method\n* the paper is very well written, easy to understand, well organized, and supported with nicely-drawn figures \n\nWeakness:\n* since the proposed method strictly requires multiple source domain, it is confusing how single domain baselines are relevant here. (it is included in all the experiment tables)\n* the authors should consider adding more discussion on how the proposed method of domain-switch is better at extracting general information than domain-mix methods. Since in the domain-mix setting, different domains are given, it can be easier for the model to learn the general information shared by these domains, which is not possible in the domain switch setting.\n* comparison with GNN would be helpful\n\n", "summary_of_the_paper": "The authors propose to improve domain generalization capability by training with a fast switching manner. The intuition is that when the model is trained with the capability to adapt and retain information, such ability can be extended to adapt to a different test time distribution. ", "summary_of_the_review": "based on the above discussion, I recommend borderline accept for this paper. I would like to adjust my rating for the paper after discussing with the other reviewers and AC.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3I5gBBRUMs2", "reviewer": "Reviewer_gTju", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper Strengths:\n\nThe authors tackle an important and challenging problem of cross-domain few-shot classification. The proposed approach is simple. Experimental evaluations clearly demonstrate the effect by introducing the domain switching training strategy.\n\nPaper Weaknesses:\n\n1) The approach is evaluated on CUB, Cars, Places, and Plantae in a leave-one-out manner. These four datasets are all fine-grained and consist of natural images. I was wondering how does the proposed domain-switch strategy work on more diverse, heterogenous datasets, like Meta-dataset [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, ICLR 2020].\n\n2) The comparisons in the paper are a little bit weak. It is only marginal better than NASE. How is the performance of NASE in the multi-domain setting? Also, comparisons with some other existing cross-domain few-shot learning methods are missing, for example [Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, ICLR 2020] [Selecting Relevant Features from a Multi-domain Representation for Few-shot Classification, ECCV 2020] [A Universal Representation Transformer Layer for Few-Shot Image Classification, ICLR 2021].\n\n3) In Remarks 1, 2, and 3, the relationships between the predicted probability of the prompter, the learner, and the teacher are discussed. While the reasoning intuitively makes sense, there is no mathematically rigorous proof. The current statements sound like they are already-proved facts.\n\n4) While in the introduction it is stated that the model is only trained on one domain at each iteration, one critical implementation detail is that the approach requires a large-scale basic training set (mini-ImageNet) to stabilize the training procedure. I think this should be mentioned in the introduction. Moreover, an ablation study should be provided – what is the performance without mini-ImageNet? Also, if the model is pre-trained on mini-ImageNet, and then trained on the individual domain in a domain-switch manner, does that work?\n\n5) Currently, the training is conducted periodically with pre-defined domain order. How is the sensitivity to the domain order? How is the performance if the domain ordering is completely random in the entire training procedure? In that case, probably the design of the prompter and teacher should be modified as well. \n\n6) Domain-mix and domain-switch can be viewed as two extremes, where domain-mix uses all the domains at each iteration, while domain-switch uses only one domain at each iteration. How is the performance if we use some strategy in between? For example, at each iteration, some domains are randomly combined as a compound domain, and there is no overlapping domain between two consecutive training iterations.", "summary_of_the_paper": "This paper aims to address the problem of cross-domain few-shot classification. The main contribution is that it introduces domain-switch learning that simulates cross-domain scenario for training, and thus improves the model’s generalization. To this end, it uses multiple domains during training – it only trains on a single domain at each iteration and switches the domain in consecutive training iterations. This training mechanism is different from widely-used strategies like mixing different domains, and it forces the model to learn domain-general knowledge. In addition, two constraints are proposed to further improve the generalization. First, a domain-specific prompter module is introduced, so that the model is not overfitting to the domain in the current iteration. Second, a domain-general teacher module is introduced, so that the model does not forget the already-learned knowledge of other domains. The approach is tested on four datasets, including CUB, Cars, Places, and Plantae, and compared with competing results.", "summary_of_the_review": "The paper introduces domain-switch learning that simulates cross-domain scenario for training, and thus improves the model’s generalization. The evaluations and comparisons in the paper are a little bit weak. Some ablation studies and analysis are missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to improve cross-domain generalization in few-shot learning, using an objective that attempts to fight overfitting on the observed domain at any given iteration while maintaining the general learned information so far from all domains. The approach uses a domain-cycling procedure, where each iteration sees a single domain and, pseudo-labels coming from predictions of a previous iterate of the model and from a parameter-averaged general model are used in a combined training objective.

Three of the reviewers support acceptance (one strongly), while the fourth leans weakly towards rejection, despite an extensive response from the authors that include new results. One concern was a lack of comparison on Meta-Dataset, which the authors went some way towards addressing during the rebuttal, though they also argued Meta-Dataset couldn't really support the kind of cross-domain evaluation they were targeting. The reviewer was not convinced by the authors' argument, and I too am not, in particular when you consider that Meta-Dataset evaluations now often include evaluations on MNIST and CIFAR-10/100, in addition to MS-COCO and TrafficSigns (all not included in the training split of Meta-Dataset). That said, the experimental protocol favored in the authors' experiments certainly is sound and challenging for cross-domain generalization, so I'd hesitate to penalize them for that alternative choice.

Overall, I find the ideas behind this work neat, interesting and well motivated. Even if the basic ideas aren't completely novel, I found their combination thought provoking and creative.

Therefore, in the end, I feel this work will be beneficial to the body of literature on few-shot learning and would merit to appear at ICLR.
Decision: Accept (Poster)