Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts
Authors: Tao Zhong
Zhixiang Chi
Li Gu
Yang Wang
YUANHAO YU
Jin Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Domain generalization
Mixture-of-Experts
Meta-learning
Knowledge distillation
Test-time adaptation

Abstract: In this paper, we tackle the problem of domain shift. Most existing methods perform training on multiple source domains using a single model, and the same trained model is used on all unseen target domains. Such solutions are sub-optimal as each target domain exhibits its own specialty, which is not adapted. Furthermore, expecting single-model training to learn extensive knowledge from multiple source domains is counterintuitive. The model is more biased toward learning only domain-invariant features and may result in negative knowledge transfer. In this work, we propose a novel framework for unsupervised test-time adaptation, which is formulated as a knowledge distillation process to address domain shift. Specifically, we incorporate Mixture-of-Experts (MoE) as teachers, where each expert is separately trained on different source domains to maximize their specialty. Given a test-time target domain, a small set of unlabeled data is sampled to query the knowledge from MoE. As the source domains are correlated to the target domains, a transformer-based aggregator then combines the domain knowledge by examining the interconnection among them. The output is treated as a supervision signal to adapt a student prediction network toward the target domain. We further employ meta-learning to enforce the aggregator to distill positive knowledge and the student network to achieve fast adaptation. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art and validates the effectiveness of each proposed component. Our code is available at https://github.com/n3il666/Meta-DMoE.

Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts
Tao Zhong"), Zhixiang Chi Li Gu"?, Yang Wang2*,, Yuanhao Yu?, Jin Tang²
University of Toronto, Huaawei Noah's Ark Lab, 'Concordia University tao zhong@mail utoronto. ca yang. wwwg@concordia.coommmmmmm Sawawiinnhwwwe.cccmm com
(zhixiang.ch) li.gu, yuanhao.
Abstract
In this paper, we tackle the problem of domain shift. Most existing methods perform training on multiple source domains using a single model, and the same trained model is used all unseen target domains. Such solutions are sub-optimal as each target domain exhibits its own specialty, which is not adapted. Further- more, expecting single-model training learn extensive knowledge from multiple source domains is counterintuitive, The model is more biased toward learning only domain-invariant features and may result negative knowledge transfer. In this work, we propose a novel framework for unsupervised test-time adapta- tion, which is formulated as knowledge distillation process to address domain shift. Specifically, we incorporate Mixture-of-Exxee Expeett (MoE) as teachers, where each expert is separately trained on different source domains to maximize their specialty. Given a test-time target domain, a small set of unlabeled data is sam- pled to query the knowledge from MoE, As the source domains are correlated the target domains, transformer- basedd aggregator then combines the domain knowledge by examining the interconnection among them. The output is treated supervision signal to adapt student prediction network toward the target domain. We further employ meta-learning to enforce the aggregator to distill positive knowledge and the student network to achieve fast adaptation. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art and validates the effectiveness of each proposed component. Our code available at https://gihbbbcccoomm/iimmeaaaaa
Introduction
The emergence of deep models has achieved superior performance [32, 40, 47]. Such unprecedented success is built the strong assumption that the training and testing data are highly correlated (i.e., they are both sampled from the same data distribution). However, the assumption typically does not hold in real-world settings as the training data is infeasible to cover all the ever-changing deployment environments [39]. Reducing such distribution correlation is known as distribution shift, which significantly hampers the performance of deep models. Humans distribution shift, but artificial learning- based systems suffer more from performance degradation.
robust against the
One line of research aims to mitigate the distribution shift by exploiting some unlabeled data from target domain, which is known as unsupervised domain adaptation (UDA) [24, 51, 26]. The unlabeled data is estimation of the target distribution [86]. Therefore, UDA normally adapts to the target domain by transferring the source knowledge via a common feature space with less effect from domain discrepancy [79, 50]. However, UDA is less applicable for real-world scenarios
"cqual contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

as repetitive large-scale training is required for every target domain. In addition, collecting data samples from a target domain in advance might be unavailable the target distribution could be unknown during training. Domain generalization (DG) [54, 28, 6] is an alternative line of research but more challenging as assumes the prior knowledge of the target domains is unknown. DG methods leverage multiple source domains for training and directly use the trained model on all unseen domains. As the domain-specinf information for the target domains is not adapted, a generic model is sub-optimal [68, 17].
Test-time adaptation with DG allows the model to exploit the unlabeled data during testing to overcome the limitation of using flawed generic model for all unseen target domains. In ARM [86], meta-learning [25] is utilized for training the model as an initialization such that it can be adapted using the unlabeled data from the unseen target domain before making the final inference. However, we observed that ARM only trains a single model, which counterintuitive for the multi-source domain setring. There is a certain amount of correlations among the source domains while each of them also exhibits its own specific knowledge. When the number of source domains rises, data complexity dramatically increases, which impedes thorough exploration of the dataset. Furthermore, real-world domains are not always balanced data scales [39]. Therefore, the single-model training is more biased toward the domain-invariant features and dominant domains instead of the domain- specific features [12].
In this work, we propose to formulate the test-time adaptation as the process of knowledge distil- lation [34] from multiple source domains. Concretely, we propose to incorporate the concept of Mfxture-of-
(MoH), which is natural fit for the multi-source domain settings. The MoR models are treated as a teacher and separately trained on the corresponding domain to maximize their domain specialty. Given new target domain, few unlabeled data are collected to query the features from expert models. A transformer-basse the interconnection among queried knowledge and aggregate the correlated information toward the target domain. The output is then treated as a supervision signal to update a student prediction network to adapt to the target domain. The adapted student is then used for subsequent inference. We employ bi-level optimization as meta-learning to train the aggregator at the meta-level to improve generalization. The student network is also weta-trained to achieve fast adaptation via a few samples. Furthermore, we simulate the test-time out-of- -it-o---iittibutiinn scenarios during training to align the training objective with the evaluation protocol.
knowledge aggregator is proposed to examine
The proposed method also provides additional advantages over ARM: 1) Our method provides a larger model capability to improve the generalization power; 2) Despite the higher computational cost, only the adapted student network is kept for inference, while the MoE models are discarded after adaptation. Therefore, our method is more flexible in designing the architectures for the teacher or student models. (e.g., designing compact models for the power-constrained environment); 3) Our method does not need to access the raw data of source domains but only needs their trained models. So, we can take advantage of private domains in a real-world setting where their data is inaccessible.
We name our method as Meta- DistiiDiiillaoionn of MoE (Meta-DMoke Our contributions are as follows:
We propose novel unsupervised test-time adaptation framework that is tailored for multiple sources domain settings. Our framework employs the concept of MoE to allow each expert model to explore each source domain thoroughly. We formulate the adaptation process as knowledge distillation via aggregating the positive knowledge retrieved from MoE.
The alignment between training and evaluation objectives via meta-learning improves the adaptation, hence the test-time generalization.
We conduct extensive experiments to show the superiority of the proposed method among the state-of-the-arts and validate the effectiveness of each component of Meta-DMoE.
We validate that our method is more flexible in real-world settings where computational power and data privacy are the concerns.
2 Related work
Domain shift. Unsupervised Domain Adaptation (UDA) has been popular to address domain shift by transferring the knowledge from the labeled source domain to the unlabeled target domain [48, 41, 81]. It is achieved by learning
features via minimizing statistical discrepancy across

---

domains [5, 58, 70]. Adversarial learning is also applied to develop indistinguishable feature space [26, 51, 57]. The first limitation of UDA is the assumption of the co-existence of source and target data, which is inapplicable when the target domain is unknown in advance. Furthermore, most of the algorithms focus on unrealistic normally come from multiple domains. Splitting the source data into various distinct domains and exploring the unique characteristics of each domain and the dependencies among them strengthen the robustness [88, 76, 78]. Domain generalization (DG) is another line of research to alleviate the domain shift. DG aims to train a model on multiple source domains without accessing any prior information of the target domain and expects it to perform well unseen target domains. [28, 45, 53] aim to learn the domain- invarrinn feature representation. [63, 75] exploit data augmentation strategies in data or feature space. A concurrent work proposed bidirectional learning mitigate domain shift [14]. However, deploying the generic model to all unseen target domains fails to explore domain specialty and yields sub-optimal solutions. In contrast, our method further exploits the unlabeled target data and updates the trained model to each specific unseen target domain at test time.
adaptation as source data
Test-time adaptation (TTA). TTA constructs supervision signals from unlabeled data to update the generic model before inference. Sun et al. [68] uses rotation prediction to update the model during inference. Chi et al. [17] and Li et al. [46] reconstruct the input images to achieve internal-eeariig to restore the blurry images and estimate the human pose. ARM [86] incorporates test-time adaptation with DG, which meta-leams model that is capable of adapting to unseen target domains before making an inference. Instead of adapting to every data sample, our method only updates once for each target domain using fixed number of examples.
Meta-learning, The existing meta-learning methods can be categorised as model-based [62, 59, 8], metric-based [65, 30), and optimization- -based [25]. Meta-learning aims to learn the learning process by episodic learning, which is based on bi-level optimization ([13) provides a comprehensive survey). One of the advantages of bi-level optimization is to improve the training with conflicting learning objectives. Utilizing such a paradigm, [16, 85] successfully reduce the forgetting issue and improve adaptation for continual learning [49]. In our method, we incorporate meta-learning with knowledge distillation by jointly learning a student model initialization and a knowledge aggregator for fast adaptation.
Mfxture-of-expers.. The goal of MoE [37] is to decompose the whole training set into many subsets, which are independently learned by different models. It has been successfully applied in image recognition models to improve the accuracy [1]. MoE is also popular in scaling up the architectures. As each expert is independently trained, sparse selection methods are developed to select subset of the MoE during inference to increase the network capacity [42, 23, 29]. In contrast, our method utilizes all the experts to extract and combine the knowledge for positive knowledge transfer.
3 Preliminaries
In this section, describe the problem setting and discuss the adaptive model. We mainly follow the test-time unsupervised adaptation as in [86]. Specifically, we define a set of N source domains Ds = (Ds'' > and M target domains Dy = (Dr') } 3=11. The exact definition of a domain varies and depends on the applications or data collection methods. It could be specific dataset, user, or location. Let æ E X and y E )' denote the input and the corresponding label, respectively. Each of the source domains contains data in the form of onput-output pairs: Ds 3 = {(25) (()g.y))22 ) In contrast,
each of the target domains contains only unlabeled data: D,' ={<+t) [ For rell-designed datasets (e.g. [33, 20]) all the source or target domains have the same number of data samples. Such condition is not ubiquitous for real-world scenarios (i.e. Z,, * Zr, if i # ig and Kj, # K2, if ji # j2) where data imbalance always exists [39]. It further challenges the generalization with a broader range of real-world distribution shifts instead of finite synthetic ones. Generic domain shift tasks focus on the dutto--distribution setting where the source and target domains are non-overlapping (i.e. DDs DJ= Ø), but the label spaces of both domains are the same (i.e. V's ))))
Conventional DG methods perform training on Ds and make a minimal assumption on the testing scenarios 3, Therefore, the same generic model is directly applied to all target domains DT. which leads to sub-optimal solutions [68]. In fact, for each Dy', some unlabeled data are readily available which provides certain prior knowledge for that target distribution. Adaptive Risk Minimization (ARM) [86] assumes that batch of unlabeled input data x approximate the
3

---

Domain models
Query set (xa-Ya)
(6.4)
CaB
Support set XSV
Inowrada
M
Agprogator A(:
sgt
al aee
MC
Figure 1: Overview of the training of Meta-DMoE. We first sample disjoint support set Xsu and query set (x). from a training domain. Xsu is sent to the expert models M query their domain-specific knowledge. An aggregator A(:;)) then combines the information and generates supervision signal to update the f(;; () via knowledge distillation. The updated f(=;0') evaluated using the labeled query set to update the meta pertapprameters.
input distribution P= which provides useful information about Pylz- Based on the assumption, an unsupervised test-time adaptation [59, 27] is proposed. The fundamental concept is to adapt the model to the specific domain using x. Overall, ARM aims to minimize the following objective C(..))
over all training domains:
Σ E
Cly, f(x: 0)), where = 11(x Ø).
(1)
Ds'EDs (xy))Dss
y is the labels for x. [xx 0) denotes the prediction model parameterized by 0. lu(;; u) an adaptation function parameterized by db It receives the original 0 of ʃ and the unlabeled data x to adapt 0 to 0'.
The goal of ARM is learn both (0. To mimic the test-time adaptation (i.e., adapt before prediction), it follows the episodic learning as in meta-lcarning [25]. Specifically, each episode processes domain by performing unsupervised adaptation using x and h(; Φ) the inner loop to obtain f(:;0') The outer loop evaluates the adapted f(:;0') using the true label to perform meta-update. ARM general framework that can be incorporated with existing meta- meta-learning approaches with different forms of adaptation module h(::-) [25, 27].
However, several shortcomings observed with respect the generalization. The episodic learning processes one domain at a time, which has clear boundaries among the domains. The overall setting equivalent to the multi-source domain setting, which is proven to be effective than learning from a single domain [53, 87] as most of the domains are correlated to each other [2]. However, is counterintuitive learn all the domain knowledge one single model as each domain has specialized semantics or low-level features [64]. Therefore, the single-model method in ARM is sub-optimal due to: 1) some domains may contain competitive information, which leads negative knowledge transfer [66]. It may tend to learn the ambiguous feature representations instead of capturing all the domain- domain-specifie information [80]: not all the domains are equally important [76], and the learning might be biased as data in different domains are imbalanced in real-world applications [39].
4 Proposed approach
In this section, we explicitly formulate the test-time adaptation as a knowledge transfer process to distill the knowledge from MoE. The proposed method is learned via meta-learning to mimic the test-time distt---isttiittio scenarios and ensure positive knowledge transfer.
4.1 Meta-distillation from mixture-of-experts
Fig. shows the method overview. We wish to explicitly transfer useful knowledge from various source domains to achieve generalization on unseen target domains. Concretely, we define
Overview.
MoE as M = {M')N to represent the domain-specific models. Each MD is separately trained using standard supervised learning the source domain D&* to learn its discriminative features.

---

We propose the test- rest-time adaptation as the unsupervised knowledge distillation [34] to learn the knowledge from MoE. Therefore, we treat M as the teacher and aim to distill its knowledge to a student prediction network [(; 0) to achieve adaptation. To do so, we sample a batch of unlabeled x from a target domain, and pass it to M to query their domain-specific knowledge (M*(x)); That knowledge is then forwarded to knowledge aggregator A(: 6). The aggregator is learned to capture the interconnection among domain knowledge and aggregate the information from MoE. The output of A(:; Φ) is treated as the supervision signal to update f fx; 0). Once the adapted O' is obtained, f(:; 0*) used to make predictions for the rest of the data in that domain. The overall framework follows the effective few-shot learning paradigm where is treated as unlabeled support set [74, 65, 25].
Training Meta-DMoE. Properly training (0, Φ) is critical im- prove the generalization on un- seen domains. In our framework, A(: o) acts as a mechanism that explores and mixes the knowledge
Algorithm Training for Meta-DMoF
Require (Dx " data source domains; o learning rates; B: meta hatch
Preirain dommin-specific MoE models
2:
for i=I....NN do
3:
Train the domain specific model M using Ds'
4:
end for
from multiple source domains.
5:
// Metattraain aggregator ( d) and student model f( fz: 0c)
Conventional knowledge distilla- tion process requires large num-
6:
Initialize: d,. 8..
7:
while not converged do
8:
Sample batch source domaiiss (Ds " reset batch loss Lo
bers of data samples and learn-
9:
for each Dz" do
ing iterations [34, 2]. The repeti-
10:
Sample support and query set: (Nsu), (Ke. DDs
tive large-scale training is inappli-
11:
.M', (xssus Φi = [M, (xssr) +1 mask M (Nsu: with with
cable in real-world applications.
12:
Perform adaptation via knowledge distillation from MoE:
13:
o' = Ar Vo. [|A(M' (M8H1 4)) [N8u:: 62)]12
To mitigate the aforementioned
14:
Evaluate the udapted 6", using the query set and accumulatee the loss:
challenges, we follow the meta-
15:
Le = Lss + Les(D (xe; 0", 0c])
learning paradigm [25]. Such bi-
16:
end for
level optimization enforces the
17:
Update dh 0c. for current meta batch:
18:
(d. Pz. 92) 4- (0. 02) WV (o.
A(., Φ) to learn beyond any spe- cific knowledge [85] and allows the student prediction network f(: to achieve fast adaptation. Specifically, We first split the data samples in each source domain Ds' into disjoint support and query sets. The unlabeled support set (xsu) used to perform adapta- tion via knowledge distillation, while the labeled query set (X2+ yo) is used to evaluate the adapted parameters to explicitly test the generalization on unseen data.
19:
end while
The student prediction network ((; 0) can be decoupled as feature extractor 0, and classifier Oc-
Unsupervised knowledge distillation can be achieved via the softened output [34] or intermediate
features [84] from M. The former one allows the whole student network 0 = (0e, 0-) to be adaptive, while the latter one allows partial or complete 6g adapt to x. depending on the features utilized.
We follow [56] to only adapt dg in the inner loop while keeping the 0₂ fixed. Thus, the adaptation process is achieved by distilling the knowledge via the aggregated features:
(2) where a denotes the adaptation learning rate, M, is the feature extractor of MoE models, which extracts the features before the classifier, and ||-1|, measures the Le distance. The goal is to obtain an updated 0' such that the extracted features of fxxsu; (²) is closer to the aggregated features. The overall learning objective of Meta-DMofF is to minimize the following expected loss:
DIST(Xsu... Me, ó, 0e) = 0- -. Ve.. (A(MM.xsu))) o) (o000-(((ee fxsui 02)1
arg min Σ
Σ (Ds
Lorly o: (xx: 02, 0e)), where 0- DIST((su,, 0c),
B..0 DetEDs EDs
(3) where Log is the cross-entropy loss. Alg. demonstrates our full training procedure. To smooth the meta gradient and stabilize the training, we process a batch of episodes before each meta-update.
Since the training domains overlap for the MoE and meta-training, we simulate the test-time out-of- distribution by excluding the corresponding expert model in each episode. To do so, we multiply the features by 0 to mask them out. M' in Lll of Alg. denotes such operation. Therefore, the adaptation is enforced to use the knowledge that is aggregated from other domains.
5

---

4.2 Fully learned knowledge aggregator
Aggregating the knowledge from distinct domains requires capturing the relation among them to ensure the relevant knowledge transfer. Prior works design hand-engineen solutions to combine the knowledge or choose data samples that are closer to the target domain for knowledge transfer [2, 88]. A superior alternative is to replace the hand-designed pipelines with fully learned solutions [19, 9]. Thus we follow the same trend and allow the aggregator A(; Φ) to be fully meta-learned without heavy hand-engineering.
We observe that the self-attention mechanism is quite suitable where interaction among domain knowledge can be computed. Therefore, we use a transformer encoder as the aggregator [22, 73j. The encoder consists of multi-head self-attention and multi-layer perceptron blocks with layernorm [4] applied before each block and residual connection applied after each block. We refer the readers to the appendix for the detailed architecture and computation. We concatenate the output features from the MoE models Concat (x), (x), .... M (x)]
RR Nxd, where d is the feature dimension. The aggregator A(:: o) processes the input tensor to obtain the aggregated feature F E Rd, which is used as supervision signal for test-time adaptation.
4.3 More constrained real-world settings
In this section, we investigate two critical settings for real-world applications that have drawn less attention from the prior works: limitation on computational resources and data privacy.
Constraint on computational cost. real-world deployment environments, the computational power might be highly constrained (e.g,, smartphones). It requires fast inference and compact models. However, the reduction in learning capabilities greatly hinders the generalization as some methods utilize only a single model regardless of the data complexity. On the other hand, when the number of domain data scales up, methods relying on adaptation on every data sample [86] will experience inefficiency. In contrast, our method only needs to perform adaptation once for every unseen domain. Only the final f(: reducing the model size, we experiment with some lightweight network architectures (only f(;; for us) such as MobileNet V2 [61].
(') 0') used for inference. To investigate the impact on generalization caused by
Data privacy. Large-scale training data are normally collected from various venues. However, some venues may have privacy regulations enforced. Their data might not be accessible, but the models that are trained using private data are available. To simulate such an environment, we split the training source domains into two splits: private domains (D and public domains (DsPub). We use D to train MoE models and rd for the subsequent meta teaatttriiinn.... Since ARM and other methods only utilize the data as input, we train them on Pub
We conduct experiments show the superiority of the proposed method in these more constrained real-world settings with computation and data privacy issues. For details on the settings, please refer to the supplementary materials.
5 Experiments
5.1 Datasets and implementation details
Datasets and evaluation metrics. In this work, we mainly evaluate our method on real-world domain shift scenarios. Drastic variation in deployment conditions normally exists in nature, such as a change in illumination, background, and time. It shows a huge domain gap between deployment environments and imposes challenges to the algorithm's 's robustness. Thus, we test our methods on the large-scale distribution shift benchmark WILDS [39], which reflects a diverse range of real-world distribution shifts. Following [86], we mainly perform experiments on five image testbeds, iWildCam [10], Camelyon17 Rx] [69] and FMoW [18] and PovertyMap [83]. In each benchmark dataset, a domain represents a distribution of data that is similar in some way, such as images collected from the same camera trap or satellite images taken in the same location. We follow the same evaluation metrics as in [39] to compute several metrics: accuracy, Macro F1, worst-case (WC) accuracy, Pearson correlation (r), and its worst-case counterpart. We also evaluate our method on popular benchmarks DomainNet [58] and PACS [44] from DomainBed [31] by computing the accuracy.

---

Table Comparison with the state-of-the arts on the WILDS image testbeds and setting. Metric means and standard deviations are reported across replicates. Our proposed method performs well across all problems and achieves the best results on 4 out of datasets.
IWildCam
Camelyon17
RxRx
FMoW
Mav
Method
Acc
Macro Fl
Acc
Acc
WC Acc
Avg. Ace
WC Pearson
Pearson
ERM
(2.5
(1.3)
(6.4)
29.9 (0.4)
(1.25)
53.0 (0.55)
0.45 (0.06)
0.78 (0.04)
CORAT.
32.8 (4.3)
(7.7)
28.4 (0.3)
(1.24)
50.5 (0.36)
0.44 (0.06)
0.78 (0.05)
Group DRO
72.7
22.9
(7.3
23.3
30.8 (0.81)
52.1 (0.5)
0.39 (0.06)
0.75 (0.07)
IRM
(3.7)
(4.1)
(8.1)
(.2
(1.37)
50.8 (0.13)
(0.07)
(0.05)
ARM CML
(0.6)
28.6 (0.1)
(1.4)
(1.8)
(0.38)
45.7 (0.28)
0.37 (0.08)
0.75 (0.04)
ARM-BN
(2.4)
23.7 (2.7)
(0.9
31.2 (0.1)
24.6 (0.04)
42.0 (0.21)
0.49 (0.21)
0.84 (0.05)
ARM-LL LL
(0.6)
(0.8)
(2.6)
24.3 (0.3)
22.1 (0.46)
42.7 (0.71)
0.41 (0.04)
0.76 (0.04)
Ouis (W/O mask)
(0.4)
35.1 (0.9)
(1.3)
29.6 (0.3)
36.8 (1.01)
50.6 (0.20)
0.52 (0.04)
0.80 (0.03)
Ours
(0.3)
34,0 (0.6)
91.4
29.8 (0.4)
35,4 (0.58)
52.5 (0.18)
0.51 (0.04)
0.80 (0.03)
Network architecture. We follow WILDS [39] to use ResNet18 & 50 [32] or DenseNet101 [36] for the expert models { Ji=1 and student network f(.:; 0). Also, we use single-layer transformer encoder ]]oc as the knowledge aggregator A(-: 6). To investigate the resource-constraingg and privacy-sensitive scenarios, we utilize MobileNet V2 [61] with a width multiplier of 0.25. As for DomainNet and PACS, we follow the setting in DomainBed to use ResNet50 for both experts and student networks.
Pre-training domain-specific models. The WILDS benchmark is highly imbalanced in data size, and some domains might contain empty classes. We found that using every single domain train an expert is unstable, and sometimes it cannot converge. Inspired by [52], we propose to cluster the training domains into N super domains and use each super-domain to train the expert models. Specifically, we set N = {10, 5, 3, 4, 3} for iWildCam, Camelyon17, RxRx1, FMoW and Poverty Map, respectively. We use ImageNet [21] pre-trained model as the initialization and separately train the models using Adam optimizer [38] with a learning rate of le and a decay of 0.96 per epoch.
Meta-training and testing. We first pre-train the aggregator and student network [15]. After that, the model is further trained using Alg. 1 for 15 epochs with a fixed learning rate of 3e for a and 3e-5 for B. During meta-testing, we use Line 13 of Alg. 1 to adapt before making prediction for every testing domain. Specifically, we set the number of examples for adaptation at test time- = {24, 64, 75, 64, 64) for iWildCam, Camelyon17, RxRx1, FMoW, and Poverty Map, respectively. For both meta-training and testing, we perform one gradient update for adaptation on the unseen target domain. We refer the readers to the supplementary materials for more detailed information.
5.2 Main results
Comparison on WILDS. We compare the proposed method with prior approaches showing on WILDS Icaderboard [39], including non-adaptive methods: CORAL [67]. ERM [72], IRM [3]. Group DRO [60] and adaptive methods used in ARM [86] (CML, BN and LL). We directly copy the available results from the leaderboard or their corresponding paper. As for the missing ones, we conduct experiments using their provided source code with default hyperparameters. Table I reports the comparison with the tfate-of-the-art Our proposed method performs well across all datasets and increases both worst-case and average accuracy compared to other methods. Our proposed method achieves the best performance on 4 out of 5 benchmark datasets. ARM [86] applies a meta-learning approach to learn how to adapt to unseen domains with unlabeled data. However, their method is greatly bounded by using a single model to exploit knowledge from multiple source domains. Instead, domain settings and meta- trains an aggregator
our proposed method is more fitted to that properly mixes the knowledge from multiple domain-specit outperforms ARM-CMLL BN, and LL by 9.5%, 9.8%, 8.1% for iWildCam, 8.5%, 4.8%, 8.5% for Camelyon 7 and 14.8% 25.0%. 22.9% for FMoW in terms of average accuracy. Furthermore, we also evaluate our method without masking the in-distribution domain in MoE models (Ours w/o mask) during meta-training (Line 10-11 11 of Alg. 1), where the sampled domain is overlapped with MoE. violates the generalization to unseen target domains during testing. As most of the performance dropped, it reflects the importance of aligning the training and evaluation objectives.
experts. As a result, our method
Comparison on DomainNet and PACS. Table and Table report the results on DomainNet and PACS. In DomainNet, our method performs the best all experimental settings and outperforms
7

---

Table Evaluation on DomainNet. Our method performs the best on all experimental settings and outperforms recent SOTA significantly in terms of average accuracy.
Method
clip
infa
quick
real
sketch
avg
[72]
58.1 (0.3)
18.8 (0.3)
46.7 (0.3)
12.2 (0.4)
59.6 (0.1)
49.8 (0.))
40.9
IRM [3]
48.5
(1.0
38.3 (4.3)
10.9 (0.5)
(8.2)
(3.1
33.9
Group DRO |60]
47.2
17.5 (0.4)
(0.5)
9.3 (0.3)
(1.4
40.1 (0.6)
33.3
Mixup [77]
55.7
18.5 (0.5)
44.3
12.5 (0.4)
55.8 (0.3)
(0.5)
39.2
MLDG [43]
39.1 (0.2)
19.1 (0.3)
45.8
(0.3)
(9.6
(0.4)
41.2
CORAL [67]
(0.6)
DANN [66]
53.1 (0.2)
(8.3
(0.2
11.8 (0.1)
55.5 (0.4
(0.6)
38.3
[U]
(7.9.0.5)
18.5
46.0
(0.1)
(0.5
(0.1
40.6
SegNet [55]
57.7 (0.3)
(0.2)
(0.3)
12.7 (0.5)
58.1 (0.5)
48.8 (0.2)
40,3
ARM [86]
49.7 (0.3)
16.3 (0.5)
(1.1)
(0.1)
53.4 (0.4)
43.5 (0.4)
35.5
Ours
63.5 (0.2)
21.4 (0.3)
(0.4
(0.3)
(1.0)
52.4 (0.2)
44.2
Table 3: Evaluation on PACS. Our method outperforms other methods in out of 4 experiments but still achieves the SOTA terms of average accuracy.
Method
art
cartoon
photo
sketch
avg
ERM [72]
84.7 (0.4)
80.8 (0.6)
97.2 (0.3)
79.3
85.5
CORAL [67]
(0.2)
(0.5)
97.5 (0.3)
78.8
Group DRO [60]
83.5 (0.9)
79.1 (0.6)
96.7 (0.3)
(2.3
84.4
IRM [3]
(1.3)
(1.1
96.7 (0.6)
(1.0)
83.5
ARM [86]
(0.6)
(0.5)
97.4
(1.2)
85.1
Ours
86.1 (0.2)
82.5
96.7 (0.4)
82.3 (1.4)
86.9
recent SOTA significantly in terms of the average accuracy (+2.7). [82] has discovered that the lack of large number of meta-training episodes leads to the meta-level weeritiing/memorizatiii
problem. To our task, since PACS has 57 less number of images than DomainNet and 80x less number of domains than iWildCam, the capability of our meta-learning-base diversity of episodes. As result, we outperform other methods in 2 out of 4 experiments still achieve the SOTA in terms of average accuracy.
method is hampered by the less
Visualization of adapted features. To evaluate the capability of adaptation via learning discrimina- tive representations on unseen target domains, we compare the t-SNE [71] feature visualization using the same set of test domains sampled from iWildCam and Camelyon17 datasets. ERM utilizes single model and standard supervised training without adaptation. Therefore, we set it as the baseline. Figure shows the comparison, where each color denotes class, and each point represents data sample. It is clear that our method obtains better clustered and more discriminative features.
(a) Camelyonl7-ERR
(b) Camelyon17-
(c) iWildCam ERM
(d) iWildCam-Our
Figure 2: t-SNE visualization of adapted features at test-time. We directly utilize features adapted to same unseen target domains from ERM and our proposed method in Camelyon17 and iWildCam datasets, respectively. Our resulting features show more discriminative decision boundaries.
5.3 Results under constrained real-world settings
In this section, we mainly conduct experiments on iWildCam dataset under two real-world settings.
Constraint on computational cost. Computational power is always limited in real-world deployment scenarios, such as edge devices. Efficiency and adaptation ability should both be considered. Thus, replace our student model and the models in other methods with MobileNet V2. As reported

---

Table 4: Comparison of WILDS testbeds using MobileNet V2. Reducing the model size hampers the learning capability. Our method shows a better trade-off off as the knowledge is distilled from MoE.
WildCam
Camelyon17
RaRxi
FMo
Paw
Method
Macro F1
Acc
WC Ace
Avg. Acc
WC Pearson
Peasson
ERM
(0.7)
17.5
(8.8
(0.2)
15.7 (0,68)
40.0 (0.11)
0.39 (0.05)
0.77 (0.04)
CORAL
(1.7
17.6 (0.1)
75.9 (6.9)
(216
22.7 (0.76)
31.0 (0.32)
0.44 (0.06)
0.79 (0.04)
ARM-CML
58.8
(0.6)
(4.6)
(1.4
21.1 (0.33)
30.0 (0.
0.41 (0.05)
0.76 (0.03)
ARM-BN
(4.6
13.8 (0.2)
(1.6(
14.9
(1.82
29.0 (0.69)
0.42 (0.05)
0.76 (0.03)
ARM-LL
(7.5
12.6 (0.8)
(1.7)
(0.2)
17.1 (0.22)
(0.54)
(0.07)
0.76 (0.02)
Ours
59.5
19.7 (0.5)
87.1 (2.3)
15.1 (0.4)
(0.67)
37,9
(..44)
0.77 (0.03)
in Table 4, our proposed method still outperforms prior methods. Since the MoE model is only used for knowledge transfer, our method is more flexible in designing the student architecture for different scenarios. We also report multiply-Accumulan operations (MACS) for inference and time complexity on adaptation. As ARM needs to make adaptations before inference on every example, its adaptation cost scales linearly with the number of examples. Our proposed method performs better in accuracy and requires much less computational cost for adaptation, as reported in Table 5,
Constraint on data privacy. On top of compu- tational limitations, privacy-regulatd scenarios are common in the real world. It introduces new challenges as the raw data inaccessible. Our method does not need to access the raw data but the trained models, which greatly mitigates such regulation. Thus, shown in Table 6, our method does not suffer from much performance degradation compared to other methods that re- quire access to the private raw data,
Table 5: Adaptation efficiency evaluated iWild- Cam using MobileNet V2. Our method not only outperforms prior methods but also keeps constant time complexity in test-time adaptation.
Method
Acc/Macro-FF
MACS Complexity
ERM
56.7 17.5
7.18 x10'
N/A
ARM-CML
58.2 15.8
7.73 x10
O(n)
ARM-LL
57.5 12.6
7.18 x10"
O(n)
Ours
59.5 19.7
7.18 x10"
O(1)
5.4 Ablation studies
Table Results on privacy-related regulation set- ting evaluated on iWildCam and FMoW using Mo- bileNet V2. Without privacy considered in the design, prior methods can only exploit public data and thus achieve far worse performance.
In this section, we conduct ablation studies on iWildCam analyze various components of the proposed method. We also seek to answer the two key questions: 1) Does the number of experts affect the capability of capturing knowl- edge from multi-source domains? 2) Is meta- learning superior to standard supervised learning under the knowledge distillation framework?
iWildCam
FMoW
Method
Acc Macro-F1
WC Ace Acc
ERM
51.2
11.2
22.5
35.4
CORAL
50.2
11.1
18.1
25.4
Number of domain-specific experts. We inves- tigate the impact of exploiting multiple experts to store domain-specific knowledge separately. Specifically, we keep the total number of data for experts' pretraining fixed and report the re- sults using a various number of expert models. The experiments in Table 7 validate the benefits of using more domain sppaifc experts.
ARM-CML.
42.77
7.5
16.8
24.1
ARM-BN
46.9
8.7
14.2
22.2
ARM-LL
46.8
9.3
13.7
22.6
Ours
54.7
14.2
24.4
33.8
Table Results on different numbers of domain- specific experts. More experts increase the learning capacity to better explore each source domain, thus, improving generalization.
Training scheme. To verify the effectiveness of meta-learning, we investigate three training schemes: random initialization, pre-train, and meta-train. To pre-train the aggregator, we add a classifier layer to its aggregated output and follow the standard supervised training scheme. For fair comparisons, we use the same testing scheme, including the number of updates and images for adaptation. Table 8 reports the results of different training scheme combinations. We observe that the randomly initialized student model struggles to learn from few-shot data. And the pre-trained aggregator brings weaker adaptation guidance to the student network as the aggregator not learned to distill. In contrast, our bi-level optimization-basee training scheme enforces the aggregator to choose more correlated knowledge
# of experts
2
5
7
10
Accuracy
70.4
74.1
76.4
77.2
Macro-F1
30.6
32.3
33.7
34.0

---

from multiple experts to improve the adaptation of the student model. Therefore, the meta-learned aggregator is more optimal (row vs. row 2). Furthermore, our meta-distillation training process simulates the adaptation in testing scenarios, which aligns with the training objective and evaluation protocol. Hence, for both meta-trained aggregator and student models, gains additional improvement (row 3 vs. row 4).
Aggregator and distillation methods. Table reports the effects of various aggregators, in- cluding two hand-designed operators: Max and Average pooling, and two MLP-based meth- ods: Weighted sum (MLP-WSS DWS) and Projector (MLP-P) (details are provided in the supplemen- tary materials). We found that the fully learned transformer-based aggregator is crucial for mix- ing domain-specific features. Another impor- tant design choice in our proposed framework is the form of knowledge: distilling the teacher model's logits, intermediate features, or both. We show the evaluation results of those three forms of knowledge in Table 10.
Table 8: Evaluation of training schemes. Using both meta-learned aggregator and student model improves generalization as they are learned to- wards test-time adaptation.
Train Scheme
Metrics
Aggregator
Student
Acc
Macro-FI
Pretrain
Random
6.2
0.1
Meta
Random
32.7
0.5
Pretrain
Meta
74.8
32.9
Meta
Meta
77.2
34.0
Table 9: Comparison between different aggregator methods. The transformer explores interconnec- tion, which gives the best result.
6 Discussion
Max Ave. MLP-WS MLP-P (ours)
Acc. 69.2 69.7
70.7
73.7
77.2
We present Meta-DMoE, a framework for adap- tation towards domain shift using unlabeled ex- amples at test-time. We formulate the adap- tation as knowledge distillation process and devise meta-learning algorithm to guide the student network to fast adapt to unseen target domains via transferring the aggregated knowl- edge from multi- malti-source domain domain-specific models. We demonstrate that Meta-DMoF. is state-of the- art on four benchmarks. And it is competitive under two constrained real-world settings, including limited computational budget and data privacy considerations.
M-F1 29.2 25.0
32.8
32.7
34.0
Table 10: Comparison between different distilla- tion methods. Distilling only the feature extractor yields the best generalization.
Logits Logits Feat.
Feat. (Ours)
Accuracy
72.1
73.1
77.2
Marco-F1 26.4
26.9
34,0
Limitations. As discussed in Section 5.4, Meta-DMoE can improve the capacity to capture complex knowledge from multi-source domains by increasing the number of experts. However, to compute the aggregated knowledge from Nomain-sppcciff experts, every expert model needs to have one feed-forward pass. As result, the computational cost of adaptation scales linearly with the number of experts. Furthermore, to add or remove any domain-specific expert, both the aggregator and the student network need to be re-trained. Thus, enabling sparse-gated Meta-DMol to encourage efficiency and scalability could be valuable future direction, where gating module determines sparse combination of Nomain--speciff experts to be used for each target domain.
Social impact. Tackling domain shift problems can have positive social impacts as it helps to elevate the model accuracy in real-world scenarios (e.g., healthcare and self self-driving cars). In healthcare, domain shift occurs when a trained model is applied to patients in different hospitals. In this case, model performance might dramatically decrease, which leads to severe consequences. Tackling domain shifts helps to ensure that models can work well on new data, which can ultimately lead to better patient care. We believe our work a small step toward the goal of adapting to domain shift.
References
[1] Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network of experts for large-scale image categorization. In European Conference on Computer Vision, 2016. 3
[2] Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K. Roy-Chowdhurg Unsupervised multi-soure, domain adaptation without access to source data, In IEEE Conference on Computer Vision and Pattern Recognition, 2021. 4, 6.
10

---

[3] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019. 3, 8, 17
[4] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 06450, 2016. 6, 18
arXiv preprint
[5] Mahsa Baktashmotlagh, Mehrtash Harandi, and Mathieu Salzmann. Distribution-matching embedding for visual domain adaptation. Journal of Machine Learning Research, 17:Article-numberr 2016. 3
[6] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using mefa-regularizztion. In Advances in Neural Information Processing Systems, 2018. 2
[7] Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018. 6
[8] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In JEEE Conference on Computer Vision and Puttern Recognition, 2020. 3
[9] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In European Conference on Artificial Intelligence, 2020. 6
[10] Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset. arXiv preprint arXiv:2004. 10340, 2020. 6
[11] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to new unlabeled sample. Advances in neural information processing systems, 2011.
[12] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In European Conference on Computer Vision, 2020. 2
[13] Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu. Gradient-basse bi-level optimization for deep lcarning: A survey, arXiv preprint arXiv:2207. 2022. 3
[14] Can Sam Chen, Yingxue Zhang, Jie Fu, Xue Liu, and Mark Coates. Bidirectional learning for offline infinite-width model-based optimization. arXiv preprint arXiv:2209.07510 2022.
[15] Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta baseline: exploring simple meta-lcarning for few-shot learning. In IEEE International Conference on Computer Vision, 2021.
[16] Zhixiang Chi, Li Gu, Huan Liu, Yang Wang, Yuanhao Yu, and Jin Tang. Metafscil: A meta-learning approach for few-shot class incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[17] Zhixiang Chi, Yang Wang, Yuanhao Yu, and Jin Tang. Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In IEFE Conference on Computer Vision and Pattern Recognition, 2021. 2, 3
[18] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of world. JEEE Conference on Computer Vision and Pattern Recognition, 2018.
[19] Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arXiv: 1905. 10985, 2019.
[20] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In International Joint Conference on Neural Networks, 2017. 3
[21] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009. 7
[22] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 6, 18
[23] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.0344 2021.
11

---

[24] Basura Fernando. Amaury Habrard, Mare Sebban, and Tinne Tuytelaars. Unsupervised visual domain adaptation using subspace alignment. In IEEE International Conference on Computer Vision. 2013. 1, 17
[25] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learnng for fäst adaptation of deep networks. International Conference on Lachine Learning. 2017. 2, 3,3,4 4.5
[26] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain- adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2330, 2016. 1,
[27] Marta Gamelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, 2018.
[28] Muhammad Ghifary, W Bastiaan Kleijn, Mengjic Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In IEEE International Conference on Computer Vision, 2015. 2, 3
[29] Sam Gross, Mare' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In IEFE Conference on Computer Vision and Pattern Recognition, 2017. 3
[30] Li Gu, Zhixiang Chi, Huan Liu, Yuanhao Yu, and Yang Wang. Improving protonet for few-shot video object recognition: Winner of orbit challenge 2022. arXiv preprint arXiv:2210.0017 2022. 3
[31] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2020. 6, 15
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In IEEE Conference Computer Vision and Pattern Recognition, 2016. 7, 19
[33] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions
and perturbations. In International Conference on Learning Representations, 2019. 3
[34] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. Advances
in Neural Information Processing Systems, 2015. 2,5
[35] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, 2018. 3
[36] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017. 7, 19
[37] Robert A Jacobs. Michael I Jordan. Steven Nowlan, and Geoffree E Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-877 87, 1991.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014. 7
[39] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, 2021. 2, 3, 6, 15, 18. 19, 20
[40] Alex Krizhevsky, 1lya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems, 2012. 1
[41] Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. In IEEE Winter Conference on Applications of Computer Vision, 2021.
[42] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2020. 3
[43] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference artificial intelligence, 2018. 8
[44] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In IEEE International Conference Computer Vision, 2017. 6
12

---

[45] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In European Conference on Computer Vision, 2018.
[46] Yizhuo Li, Miao Hao. Zonglin Di, Nitesh Bharadwaj Gundavarapu, and Xiaolong Wang. Test-time personalization with a transformer for human pose estimation. Advances in Neural Information Processing Systems, 2021. 3
[47] Hanwen Liang, Niamul Quader, Zhixiang Chi, Lizhe Chen, Peng Dai, Juwei Lu, and Yang Wang. Self- supervised spatiotemporal representation learning by exploiting video continuity. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[48] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, 2020.
[49] Huan Liu, Li Gu, Zhixiang Chi, Yang Wang, Yuanhao Yu, Jun Chen, and Jin Tang. Few-shot class- incremental learning via entropy-regularized data-free replay. arXiv preprint arXiv:2207.112155 2022.
[50] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, 2015. 1
[51] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In Advances in Neural Information Processing Systems, 2018 1,3, 17
[52] Daniela Massiceti, Luisa Zintgraf, John Bronskill, Lida Theodorou, Matthew Tobias Harris, Edward Cutrell, Cecily Morrison, Katja Hofmann, and Simone Stumpf. Orbit: A real-world few-shot dataset for teachable object recognition. IEEE International Conference on Computer Vision, 2021.
[53] Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. In AAAI Conference on Artificial Intelligence, 2020. 3, 4
[54] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, 2013. 2
[55] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap via style-agnostic networks. arXiv preprint 1910 16445, 2019.
[56] Jaehoon Oh, Hyungjun Yoo, Changlwan Kim, and Se- Young Yun. Boil: Towards representation change for few-shot learning. In International Conference on Learning Representutions, 2021. 5
[57] Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adaptation. In AAAI Conference on Artificial Intelligence, 2018. 3
[58] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In IEEE International Conference on Computer Vision, 2019. 3,
[59] James Requeima, Jonathan Gordon, John Bronskill, Sehastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 2019. 3.4
[60] Shiori Sagawa, Pang Wei Koh, Tatsumori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. 7,
[61] Mark Sandler, Andrew Howard, Menglong 7hu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenety2: Inverted residuals and linear bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition. 2018. 6,
[62] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- Meta-learnin learning with memory augmented neural networks. In International Conference on Machine Learning, 2016. 3
[63] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In International Conference on Learning Represemiations, 2018. 3
[64] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of experts layer. In International Conference on Learning Representations, 2017. 4
13

---

[65] Jake Snell. Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017. 3,5
[66] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In International Conference Machine Learming, 2020. 4
[67] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European Conference on Computer Vision, 2016. 3,7,8 17
[68] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning, 2020. 2, 3
[69] James Taylor, Berton Earnshaw, Ben Mabey, Mason Victors, and Jason Yosinski. Rxxxl: An image set for cellular morphological variation across many experimental batches. In International Conference on Learning Representations, 2019.
[70] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell, Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 2014. 3
[71] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Rexearch, 911), 2008. 8
[72] Vladimir Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 1999. 7,8 8
[73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017. 6, 7, 18
[74] Oriol Vinyals. Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016. 5
[75] Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In Advances in Neural Information Processing Systems, 2018. 3
[76] Junfeng Wen, Russell Greiner, and Dale Schuurmans. Domain aggregation networks for multi-source domain adaptation. In International Conference Machine Learning, 2020. 3,
[77] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020. 8
[78] Ruijia Xu, Ziliang Chen, Wangmeng Zuo, Junjie Yan, and Liang Lin. Deep cocktail network: Molti-source unsupervised domain adaptation with category shift. In IEFF Conference on Computer Vision and Pattern Recognition, 2018.
[79] Ruijia Xu, Guanbin Li, Jihan Yang, and Liang Lin. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In IEEE International Conference on Computer Vision, 2019.
[80] Luyu Yang, Yogesh Balaji, Ser-Nam Lim, and Abhinav Shrivastava. Curriculum manager for source selection in multi-source domain adaptation. In European Conference on Computer Vision, 2020. 4
[81] Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In IEEE International Conference on Computer Vision, 2021. 2
[82] Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpolation. arXiv preprint arXiv:2106.02600 2021. 8
[83] Christopher Yeh, Anthony Perez, Aune Driscoll, George Azzari, Zhongyi Tang, David Lobell, Stefano Ermnn, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 2020. 6
[84] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
14

---

[85] Chi Zhang, Nan Song, Guosheng Lin, Yun Zheng, Pan Pan, and Yinghui Xu. Few-shot incremental learning with continually evolved classifiers. In IEEE Conference Computer Vision and Pattern Recognition, 2021. 3,5
[86] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: l.carning to adapt domain shift. In Advances in Neural Information Processing Systems, 2021. 1, 2,
[87] Han Zhao, Shanghang Zhang. Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. Adversarial multiple source domain adaptation. Advances in Neural Information Processing Systems, 2018. 4
[88] Sicheng Zhao, Guangzhi Wang, Shanghang Zhang, Yang Gu, Yaxian Li, Zhichao Song, Pengfei Xu, Runbo Hu, Hua Chai, and Kurt Keutzer. Multi-source distilling domain adaptation. AAAI Conference on Artificial Intelligence, 2020. 3,6 6
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplemental material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen-
tal material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See 5.1
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 5.2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See [39] [31]
(b) Did you mention the license of the assets? [Yes] See supplemental material
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "WgywoXryN0H", "reviewer": "Reviewer_RtJB", "summary": "The proposed method addresses test-time adaptation as knowledge distillation from mixture of experts. Each expert is learned within each source domain as teachers. During testing time, a set of unlabeled sample is used to query the knowledge from the teachers. A transformer-based model is used to integrate knowledge from experts and meta-learning is used to encourage the aggregation for useful knowledge and student for good adaptation.", "questions": "See weaknesses.\n\nWhat are the comparison results of the proposed method with recent SOTA on the datasets in [67]?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n\nThe idea is interesting. Distilling knowledge from different sources is not new, but the knowledge distillation and the meta learning seem novel to me. The paper is properly presented and some constrained real-word settings are also considered.\n\nWeaknesses.\n\nThe major weakness is the lack of comparison with prior work on standard datasets. For example [67] (and more recent ones) and the datasets used by [67]. If the authors are willing to validate their method on these standard datasets, more recent works can be compared. Without comparing with those methods, it is hard to evaluate the performance of the method. \n\n\n-------------------------- Post Rebuttal -----------------------------------------\n\nMy concerns regarding to comparison with [67] on standard datasets have been resolved. I raise my score to weak accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vm0_ziIEbs", "reviewer": "Reviewer_cPHa", "summary": "The paper tackles the unsupervised domain adaptation scenario, with multiple labeled source domains and an unlabeled target domain. The work draws inspiration from prior work on test-time adaptation, in particular ARM [72]. The key idea is to train a separate model for each source domain, and leverage the mixture of experts (MoE) to fully benefit from the knowledge from these domains. A meta-learning framework is proposed, where a student network aims to distill knowledge from the teacher MoE model. A transformer-based aggregator network is used to combine and contextualize MoE knowledge. Both the teacher and student models are meta-trained in a bi-level optimization framework. The adapted student network is then used during inference. The paper presents experiments on the popular WILDS benchmark, demonstrating competitive performance vs. state-of-the-art methods on a range of tasks.", "questions": "- Please address the question above about the number of target samples used by the proposed method vs. the baselines.\n\n- Please address the questions above about the experimental details such as number of target domains and val/test splits.\n\n- Section 4.1: It would be good to clarify, is the source domain data used to train the expert model overlapping or is it disjoint from the support/query sets used later in meta-training? (I would expect that these are not seen while training the expert model?)\n\n- The ablation study \"Number of domain-specific experts\" L340 needs a clarification, what does \"increasing the number of experts\" really mean? Is the overall training set fixed and only the \"clusters\" within it are more coarse/fine? Or is each case using more/less data? (I would expect the former?)\n\n- For the aggregator architecture ablation, it would have been interesting to also include another learned yet non-transformer design (in addition to the naive max/mean).\n\n- Table 8 shows that distilling features alone is better than also distilling the logits, do you have any thoughts on the reasons behind it?", "limitations": "The guidelines explicitly state that \"The page limit was increased to ensure that authors have space to address the checklist questions.\" Thus it would have been more appropriate to include the Discussion on Social Impact in the main paper rather than the supplemental.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The proposed approach builds upon related prior work yet appears to be sufficiently novel and also effective.\n- Comparison to SoTA seems comprehensive. The proposed method does not outperform SoTA in all cases but delivers strong performance across the board, which is not true about the other methods.\n- The authors present a detailed and informative ablation study demonstrating that the choices made in the model lead to superior performance.\n- I also appreciate the additional studies concerning the constrained computational cost and data privacy settings.\n\nWeaknesses\n\nMost of the key weaknesses are around presentation/clarity and lack of certain experimental details. (Additional issues are listed under Questions.)\n\n- One of the arguments in favor of Domain Generalization is that \"the data samples from a target domain ... might be unavailable\" L35. This is emphasized by the authors, but later is ignored, since they operate in the setting where unlabeled target data is available. It is indeed a realistic case that _no target domain data is available_, and I was hoping the authors plan to address that, but that was not the case. Perhaps the presentation could be made clearer on that matter.\n\n- Moreover, it would be good to detail, how much target data is indeed used by the proposed method? From the supplemental material Tab 11 it looks like for iWildCam there are 24 images used for adaptation. Would have been good to include these details (for all the datasets) in the main paper! \n- Finally, what about the competing baselines, how many target images were used in those? I.e. is the comparison fair?\n\n- The experimental settings should be detailed more, e.g. we only see the number of source domains in each dataset, but not the number of target domains! \n- Moreover, it is unclear how the validation/test splits are defined, e.g. is validation set part of the target domain? Is it labeled or unlabeled? (Perhaps these are standard settings for WILDS, but would be good to clarify for the readers not closely familiar with WILDS!)\n- Since it is not stated clearly, is it correct that all the labels are shared across the source and target domains in this work?\n\nWriting\n- There are quite a few sentences with incorrect grammar (in the Intro and throughout the paper), I recommend proof-reading.\n- L70 \"Our contributions are manifold\" : I do not think this is the right use of the word \"manifold\".\n- In the Related Work section, ideally each paragraph should include direct contrast of prior works to the proposed method, to make the differences clear.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "71503_bGjwY", "reviewer": "Reviewer_RHgP", "summary": "\nHere, the authors present a novel framework (Meta-DMoE) to alleviate domain shift using unlabelled data during test time. Many techniques were combined such as knowledge distillation and meta-learning in order to address existing limitations in previous methods, including training on multiple source domains with a single model and applying it to all subsequent unseen target domains and its bias toward domain-invariant features. The method was evaluated using five test datasets from WILDS  and compared against existing state-of-the-art methods addressing domain shift.\n", "questions": "As stated earlier, please double-check points in the text that might sound a bit repetitive.", "limitations": "Limitations were properly addressed in the text.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1) Meta-DMoE explores and takes advantage of knowledge distillation and meta-learning to devise a method that can deal with domain shit at test time using unlabelled data. A model with such skill is valuable where training might be expensive and this contribution is important.\n\n2) The authors conducted extensive experiments (WILDS image testbeds and out-of-distribution settings, constraints on computational cost, and privacy regulation ) and Meta-DMoE presented the best results in most cases.\n\n3) Ablation studies were also extensive exploring reasonable conditions.\n\n\nWeaknesses\n\nMinor:\n\n1) I thought the reading was a bit repetitive, e.g.\nLine 70: We name our method as Meta-Distillation of MoE (Meta-DMoE)\nLine 162: We name our approach Meta-distillation from MoE (Meta-DMoE)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept