LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization
Authors: Liang Chen
Yong Zhang
Yibing Song
Zhiqiang Shen
Lingqiao Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Domain generalization
Knowledge distillation
Primary area: machine_vision
TL;DR: A new knowledge distillation paradigm to enable effective learning from multiple experts for domain generalization

Abstract: Domain generalization (DG) methods aim to maintain good performance in an unseen target domain by using training data from multiple source domains. While success on certain occasions are observed, enhancing the baseline across most scenarios remains challenging. This work introduces a simple yet effective framework, dubbed learning from multiple experts (LFME), that aims to make the target model an expert in all source domains to improve DG. Specifically, besides learning the target model used in inference, LFME will also train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. Delving deep into the framework, we reveal that the introduced logit regularization term implicitly provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Extensive experiments on benchmarks from different DG tasks demonstrate that LFME is consistently beneficial to the baseline and can achieve comparable performance to existing arts.  Code is available at https://github.com/liangchen527/LFME.

LFME:
A Simple Framework for Learning from Multiple Experts in Domain Generalization
Liang Chen' Yong Zhang?* Yibing Song Zhiqiang Shen' Lingqiao Liu
MBZUAI Meituan Inc. The University of Adelaide
(1iangchen527, zhangyong201303, yibi sibingsong.oo ev @gmail com zhi zhiqing. Swwnembzuai acc.ae lingqiao, liuladel a de, edu.
Abstract
Domain generalization (DG) methods aim to maintain good performance in an unseen target domain by using training data from multiple source domains. While success on certain occasions are observed, enhancing baseline across most sce- narios remains challenging. This work introduces a simple yet effective framework, dubbed learning from multiple experts (LFME), that aims to make the target model expert in all source domains improve DG. Specifically, besides learning the target model used in inference, LFME will also train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. Delving deep into the framework, reveal that the introduced logit regularization term implicitly provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Extensive experiments benchmarks from different DG tasks demonstrate that LFME is consistently beneficial to the base- line and can achieve comparable performance existing arts. Code is available at hstps: / / thub. com 1 aannchen5227 LFME.
1 Introduction
Deep networks trained with sufficient labeled data are expected to perform well when the training and test domains with similar distributions [74, 4]. However, domains in real-world often exhibit unexpected characteristics, leading to significant performance degradation for the trained model. Such problem referred to as distribution shift and is ubiquitous in common tasks such as image classification [43, 27] and semantic segmentation [19, 40). Various domain generalization (DG) approaches have been proposed address the distribution shift problem lately, such invariant representation learning [53, 69, 57, 62, 29], augmentation [91, 48, 79, 47], adversarial learning [24, 46, 81, 49], meta-learning [44, 2, 22, 45], to name a few. Yet, according [27], most arts perform inferior to the classical Empirical Risk Minimization (ERM) when applied with restricted hyperparameter search and evaluation protocol. Both the experiments in [27] and our findings suggest that existing models are incapable of consistently improving ERM in all evaluated datasets. The consistent improvement for ERM thus becomes our motivation to further explore DG.
Our approach derives from the observation in [10] that some of the data encountered at test time are similar one or more source domains, and in which case, utilizing expert models specialized in the domains might aid the model in making better prediction. The observation can be better interpreted with the example Fig. Given experts trained the "infograph", "real", and "quickdraw" domains, and test samples from the novel "sketch" domain. Due to the large domain shift, it would be better to rely mostly on the expert trained on the similar "quickdraw" domain than others.
Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training
Test
quickdraw
1st Expert
Targes Modil
-|
sketch
Infograph
2nd Expert
3rd
Modell
Target
forward direction
guidance direction
Expert
logits vector
probability vector
contact operation
Figure 1: Pipeline of LFME. Experts and the target model are trained simultancously. To obtain a target model that is an expert on all source domains, we learn multiple experts specialized in corresponding domains to help guide the target model during training. For each sample, the guidance is implemented with a logit regularization term that enforces similarity between the logit of the target model and probability from the corresponding expert. Only the target model is utilized in inference. Please refer to Algorithm for detailed implementations.
However, the test domain information is often unavailable in DG, indicating that we can specif- ically train an expert specialized in particular domain. In light of this, obtaining a target model that an expert on all source domains seems be a practical alternative for handling potential arbitrary test domains. A naive implementation would be training multiple experts on each domain, and dynamically aggregate them to form the target model. So that any encountered test samples can be predicted by corresponding experts who are familiar with their characteristics. Nevertheless, such a practice has two inherent limitations: (1) designing an effective aggregation mechanism is essential and inevitable for the naive model. In fact, our experimental study indicates that using naive aggregating strategies, such as averaging, may deteriorate the performance.
(2) the overall framework requires much more resources for deployment when there are many training domains, since all the experts must be leveraged during the test phase.
This work proposes simple framework for learning from multiple experts (LFME), capable of obtaining an expert specialized in all source domains while avoiding the aforementioned limitations. Specifically, during the training stage, instead of heuristically aggregating different experts, we suggest training universal target model that directly inherits knowledge from all these experts, which is achieved by simple logit regularization term that enforces the logit of the target model to be similar to probability from the corresponding expert. With this approach, the target model expected to leverage professional guidance from multiple experts, evolving into an expert across all source domains. During the test phase, only the target model utilized. As a result, both model aggregations and extra memory and computation resources are not required during the deployment, since we only leverage one model. The overall training and test pipelines are illustrated in Fig. 1.
Our method can be interpreted through the lens of knowledge distillation (KD), where the core idea transferring knowledge by training the student (i.e. the target model) with soft labels from teachers (i.e. experts) [31]. Unlike traditional KD [31] that uses teachers' output probabilities as soft labels in a cross entropy loss, we employ a logit regularization term that uses experts' probabilities to refine the logit of the target model in regression manner, which can be regarded as extending the effectiveness of mean squared error (MSE) loss in classification [37] within the KD realm.
To gain a deeper understanding of the effectiveness of our logit regularization term, we perform in-depth analyses and uncover that its merit over the baseline can be explained in twofold. (1) implicitly regularizes the probability of the target model within a smaller range, enabling it to use more information for prediction and improve DG accordingly. It is noteworthy that the effect is achieved inherently differs from that by label smoothing (LS) [71]. as LFME does not require explicitly calibration for the output probability. Expanding upon this analysis, we find that simple combination of cross entropy and MSE losses achieves comparable performance among existing arts Given its straightforward implementation without unnecessary complexities, this expanding may offer a "free lunch" for DG; (2) It further boosts generalization by helping the target model to focus more on hard samples from the experts, supported by our theoretical finding. Through experiments on different datasets, we find that hard samples from the experts are more beneficial for generalization than those from the model itself. Given that hard sample mining is essential for easing distribution shift [35, 42], this discovery may offer valuable guidance for future research endeavors.

---

Through evaluations on the classification task with the DomainBed benchmark [27] and segmentation task with the synthetic [63, 64] to real [20, 83, 55] setting, we illustrate that LFME is consistently beneficial to the baseline and can obtain favorable performance against current arts (other KD ideas
included). Our method favors extreme simplicity, adding only one hyper-parameter, that can be tuned in rather large range, upon training the baseline ERM.
2 Related Works
General DG methods. Domain generalization (DG), designed to enable a learned model to maintain
robust results in unknown distribution shift, is gaining increasing attention in the research community lately. The problem can be traced back to a decade ago [6], and various approaches and applications have been proposed to push the generalization boundary ever since [53, 26, 46, 32, 81, 49, 2. 16, 11, 13, 82, 12]. The pioneering work [4] theoretically proves that the DG performance is bounded by both the dora-ddmain accuracy and inter-domain differences. Most previous arts focus on reducing the inter-domain differences by learning domain invaariant features with ideas such as kernel methods [53, 26], feature disentanglement [10, 61], and gradient regularization techniques [69, 62]. Same endeavors also include different learning skills: adversarial training is leveraged [24, 81] to enforce representations to be domain agnostic; meta-learning is utilized [2, 45] to simulate distribution shifts during training. Other works aim to improve the intra-domain accuracy: some suggest explicitly mining hard samples or representations with handcraft designs, such as masking out dominant features [36], weighting more on hard samples [42], or both [35]. LFME falls into this category as the target model can also mine hard samples from the experts, beneficial for excelling in all source domains (in Sec. 6.5).
Utilizing experts for DG. Methods with the most relevant motivations with our LFME are perhaps those also involves experts [86, 28, 89, 90]. In DAELDG [90], a shared feature extractor is adopted, which followed by different classifiers (i.e. expert) that correspond to specific domains. Their experts are trained by enforcing the outputs to be similar to the average output from the non-expert classifiers. Different from our work, it uses the average outputs from different experts as the final result. In Meta-DMoE [89], similar to LFME, different experts are trained on their specific domains where a traditional KD idea is adopted: the feature from the target model is enforced to be similar to the transformer-processe
version of their experts' output features. Notably, Meta-DMoE and LFME share very distinct objectives for the expert models. Specifically, Meta-DMoB aims to adapt the trained target model to new domain in test. To facilitate adaptation, their target model is assumed to be capable of identifying comain-specific information (DSI), and is enforced to extract DSI similar to those from domain experts. In their settings, domain experts are expected to thrive in all domains and are used not in their trained domains but rather in an unseen one. Differently, LFME expects its target model to be expert in all source domains, where domain experts provide professional guidance for the target model only in their corresponding domains. Additionally, Meta-DMoE involves meta-learning and test-time training, which is more complicated than the end-to-end training adopted in LFME. Unlike their empirical design, our method is more self-contained, supported by in-depth analysis to explain its efficacy (in Sec. 4). We further show in our experiments (in Sec. 5.1 and E) that these related two methods perform inferior to our design, and the improvements from their basic designs: using average performance (in Sec. 6.4) or enforcing feature similarity between the target model and experts (in Sec. 6.3) are subtle compared to our logit regularization.
DG in semantic segmentation. Different from image classification, semantic segmentation involves classifying each pixel of the image, and the generalizing task often expects a model trained from the synthetic environments to perform well on real-world views. Directly extending general DG ideas to semantic segmentation is not easy. Current solutions mainly consist of domain randomization [34, 85], normalization [19, 56, 73], or using some task-related designs, such as the class memory bank in [40]. Different from some existing DG methods, LFME can be directly extended to ease the distribution shift problem in the semantic segmentation task without requiring any tweaks, and show that it can obtain competitive performance against recent arts specially designed for the task.
3 Methodology
Problem Setting. In the vanilla DG setting, we are given M source domains D, 0D Dz...., Dm). where D, is the i-th source domain containing data-label pairs (x/1))) sampled from different
3

---

probabilities on the joint space X x, the goal is to learn model from D, for making predictions on the data from the unseen target domain D M+1- For either DG or the downstream semantic segmentation task, source and target domains are considered with an identical label space, and we assume that there are a total of K classes.
Learning multiple experts. LFME trains all the experts and the target model simultancously. and the training procedure is illustrated in the upper part of Fig. As each expert corresponds to a specific domain, given the M source domains, a total of M experts have to be trained during this stage, and the training process of each expert is the same as that of the ERM model. Given training batch B E D,- for the i-th expert E,. we only use data from the i-th domain, and the computed loss regarding the data-label pair (wl.y) (B) can be written as,
explec Σÿ exp(ett)
Li H: ,y",, s.t. qe xof)
(1)
where ofr E RK is the output probability computed by applying the softmax function over the output logits z with E = (x'): H denotes the cross-entropy loss: H(q.y) = Σ. -ye yologg
Learning the target model. Data-label pairs (x.y) E B from all domains are used for training the target model T. The main classification loss Lota computed similar Eq. (I): Lela = =H s.t. q softmax(2) 2) = exp(xz) and 2 = T(x). Then, to incorporate professional guidance from the Σラ exp(zz)
experts, we further introduce a logit regularization term Lyuid. to assist 7' to become an expert on all source domains, which is computed by using the probabilities from the experts as label for T:
Lguid = ||2 1|=-9 1=1
where gf is the concatenate probabilities from different experts along the batch dimension 2, |1-11 denotes the L2 norm, and this term only enforced on the target model. Note we use the normalized version of 2 (i.e. q8) for computing Lguidd which can be regarded as extending the effectiveness of MSE loss [37] in the KD realm. Our experimental studies (in Sec. 6.3) find it leads to better performance, and the following contents also elaborate on the motivation. Then, the overall loss Lall for updating the target model can be represented as,
Lall = Leta+ 5 Lquid.
(3)
where a is the weight parameter, the only additional parameter upon ERM. We train the target model and the experts simultancously for simplicity. Please refer to pseudocode in Algorithm for details.
Rationality (comprehension from a KD perspective). Our logit regularization term can be viewed as new KD form, wherein the fundamental principle is to utilize the teachers' (i.e. experts) outputs as soft labels for the student (i.e. target model) in training objective [31]. In the context of classification tasks, the cross entropy loss (qq is widely used in the literature. Based on this objective, an intuitive revision to achieve distillation is by replacing the ground-truth label y with gf in a cross entropy regularization manner (i.e. H f)), which builds the rationality for the pioneering KD art [31]. Nevertheless, a recent study [37] suggests that the MSE loss | (without applying softmax function on 2) performs as well as the cross entropy loss when being applied in the classification task. Correspondingly, distillation scheme motivated by this objective can thus be utilizing the soft label of in regression manner, which comes to our logit-regllarized term: 11a-gf []. From this perspective, the rationality of the introduced logit regularization term aligns with the principle of KD, and can be regarded as extending the applicability of MSE loss in classification to the KD realm. We compare our new KD form with other ideas in Sec 6.3, demonstrating its superior performance against existing KD ideas in the DG task. We delve deep into our method and explain the effectiveness of LFME in the following section.
Computational cost. Inherited from KD, training LFME inevitably requires more resources as both the teacher and student have to be involved during training. However, LFME uses the same test resources as the baseline ERM given only the target model is utilized. Mcanwhile, please also note that the computational cost for LFME is not proportional w.r.t the domain size. Instead, the training cost will always be doubled compared to ERM, as each sample will require two forward passes: one for the target model and the other for the corresponding expert. Please refer to Tab. 11 for training time comparisons between different arts In Sec. F. we show that simply increasing training resources for current arts cannot improve their performances, suggesting it may not be primary factor in DG.
2z can also be regarded concatenated results from different domains: Lguid = E, L L Σ, 1| Eillz'

---

(a) y from ERM
(b) q from LFME (C) z from ERM
(d) from LFME
(e) F and F
Figure 2: Values of probabilities, logits, and rescaling factors(i.e. q, 2, F.F') from the ERM model
and LFME. Models are trained on three source domains from PACS with the same settings.
4 Deeper Analysis: How the Simple Logit Regularization Term Benefits DG?
4.1 Enabling the Target Model to Use More Information
Specifically, for the baseline model, using only the classification loss Leta encourages the probability to be diverse: the ground truth q, to approximate and O for others. Consequently, the corresponding
logits& z, will increase ceaselessly, iee. 2, +00, and vice versa for Zc. i.e. Ze -00, Vc * (depicted
Fig. 2 (a) and (c)), as this is the solution for minimizing - log Eexp(z)) From another point of view, Leia encourages the model to explicitly focus on the dominant and exclusive features that are strongly discriminative but may be biased towards simplistic patterns [25].
exp(2=)
Differently, when Lguid is imposed, the logits 2 will approximate the range of qe (i.e. [0. IJ). Eventually, the final logits will balance these two losses (i.e. z. (q.", +00) and & E (-c0, q. Ve # as shown in Fig. 2 (d)). resulting in smoother distribution of q. where qe (Vc # x) in LFME will be larger than is in ERM (see Fig. (b)). Since both two losses encourage the model to make a good prediction (i.e. is expected to be the largest in both losses), the increase of qe indicates that besides learning the dominant features, the target model will be enforced learn other information that is shared with others. Compared with ERM that prevents the model from learning other features, LFME is more likely to make good predictions when certain types of features are missing while others exist in unseen domains. We provide a "free lunch" inspired by the analysis in Sec. 6, and more justifications (including visual and empirical evidence) to support this analysis in Sec. D.1 and D.2. In Sec. C, we demonstrate that other KD ideas face challenges in achieving the same merit.
The above finding can also be endorsed by the information theory [68]. Specifically, with a smoother distribution of q. the entropy, which measures information, will naturally increases [51], suggesting theoretical basis for utilizing more information in LFME. According to the principle of maximum entropy [38], the improvement for generalization is thus foreseeable [88].
Note this effect is achieved inherently differs from that by label smoothing (LS) [71], as LFME does not involve hand-crafted settings to deliberately calibrate the output probability, which is essential in LS. In Sec. 6.2, we show LS is ineffective in DG compared LFME. Besides the advantage of avoiding problems raised by potential improper heuristic designs, we show in the following that the logit regularization in LFME provides another merit over LS.
4.2 Enabling the Target Model to Mine Hard Samples from the Experts
This effect is realized by example re-weighting, motivated by proposition 2 in [72]. The single sample gradient of Lall with respect to the c-th logit value 2. can be formulated as,
a Lall =c ye talee - 4
(4)
dzc
When the target probability corresponds to the ground-truth, y. = 1, Eq. (4) is reduced into,
Lall
dz. q. +0((0--99))
(5)
In this situation, the rescaling factor F is given by,
T= ô &all
3 0Lela =1 -.
(6)
dz.
az.
1-q-
On the other hand, for Yc # s, summing the gradient values over the finite indexes gives,
Σ dall
Egc+a [ - -..),
Ozc
cR
cR
5

---

Table I: Evaluations in DomainBed with default settings (3 random seeds each with 20 trials). Top5 and score count how often a method achieves the top performance and outperforms ERM. Results with are from the ResNet50 backbone and others with ResNet18. Best and second bests results are highlighted. Results with SWAD are cited from [8], and all others are reevaluated in our device.
PACS
VLCS
OfficeHome
Terralne
DomainNet
Avg.
Top5ç
Score
MMD [46]
81.3 0.8 0.8
74.9 = 0.5
59.9
42.0 + 1.0
7.9 + 6.2
53.2
0
RSC [36]
80.5
75.4
0.6
39.4 1.3
27.9
56.3
0
IRM [1]
8.5
7.1
5.10
0.9
30.4
56.6
0
DANN [24]
0.3
76.3
+.5
0.9
0.1
56.9
0
GroupGRO [66]
80.7
1.04
6..6
4..5
2.1
57.1
0
VREx (42)
0.5
7.6
4.3
2.01
1
CAD [65]
0.3
0.6
6..5
4.
3.8
57.8
0
CondCAD [65]
8.5
7.31
6..0
3.
3.7
57.9
O
MTL [5]
0.8
0.3
0.59
4.
3.0
58.1
0
ERM [75]
79.8
a.2
60.6
1.0
0.1
58.1
0
MixStyle [91]
8.46
0.7
59.6
4.19
3.19
58.4
1
MLDG [44]
0.2
0.3
60.9
4..1
3.0
58.6
0
Mixup [80]
0.9
0.3
6.57
42.1
3..0
58.6
0
MIRO [9]
75.9
7.44
6..1
4
2.11
58.8
3
Fishr [62]
0.3
7.
6..9
4..6
3..2
59.0
U
Meta-DMoE [89]
7..0
6.12
40.0 1.2
3..
SagNet [54]
0.6
7.84
6.35
40.6
3.1
SelfReg [39]
0.3
0.74
6.14
0.3
3.2
59.3
Fish [69]
8.3
7.2
6..0
4.6
3.0
59.3
CORAL [70]
8.0
7.45
6..4
4.
3621
59.4
1
SD [60]
8.3
7.45
6,.9
4..0
3.2
59.7
2
CausEB [17]
8.44
76.5
6.22+0.1
4.3
0.1
59.8
3
ITTA [15]
0..
0.69
62.0
4.5
3.19
60.2
3
RIDG [14]
0.3
0,
6.1
0.5
3.20
60.3
4
0.1
0.1
6.12
4.5
36.1 0.1
ERM [75]
0.9
77.7 0.8
4.95
40.8 0.2
62.8
Fish [69]
0.3
0.1
6.5
4..6
4.2
63.5
CORAL" [70]
8.4
77.9
0.3
1.2
4.0
63.9
SD [60]
0.2
77.6
6.29
4..4
4.2
63.9
Ours
0.5
0.2
6.3
4.9
4.11
64.6
ERM w/ SWAD"
0.1
0.1
7.26
5.3
0.1
66.9
CORAL w/ SWAD*
88.3
0.1
0.1
0.10
4.0
67.3
w/ SWAD
88.7 0.2
0.1
73.1
5..4
4.0
and the rescaling factor if in this situation is,
T Σ &Lall
1 &Lela
1 Ects To of
Oze
Ozc
q+
We observe that both F and F are strictly monotonically increased regarding the value of g Given almost all values of the rescaling factors are negative can be observed in Fig. 2 (e) (except in the few initial steps where and q.º are both small and Lguid barely contributes) with the same logits, a smaller qf in which case the expert is less confident, will lead larger |F| and [F'). This phenomenon indicates that with the logit regularization term, the target model will focus more on the harder samples from the experts. Empirical findings supporting this analysis are in Sec. D.3 and D.4. Note that the segmentation task also utilizes one-hot labels and cross-entropy loss, making applicable to two analyses presented.
Experiments
5.1 Generalization in Image classification
Datasets and Implementation details. We conduct experiments on 5 datasets in DomainBed [27]. namely PACS [43] (9,991 images, 7 classes, 4 domains), VLCS [23] (10,729 images, 5 classes,
We also observe plots of Ff and F' almost overlap in Fig. 2 (e). This is because optimizing || of leads to Y. (&e- c = O, enforcing Σ. Ze x qf = Since there no other constraint for 2' (Lela does not impose a constraint on the summation of logits), we will have Σ. de z and F = F in all cases, this is consistent with the observation in Fig. (d).

---

Table Evaluations on the semantic segmentation task. Results with are directly cited from [19].
others are reevaluated in our device. Best results are colored as red.
Cityscapes (%)
BDD100K (%)
Mapillary (%)
Avg. (%)
mIOU mAce
mIOU
mAcc
mIOU
mAce
mIOU
mAce
Baseline'
35.46
25.09
31.94
30.83
IBN-Net [56]*
35.55
32.18
38.09
35.27
RobustNet [19]*
37.69
34.09
38.49
36.76
Baseline
37.19
48.75
27.95
39.04
32.01
48.88
32.38
45.56
PinMem [40]
41.86
48.30
34.94
44.11
39.41
49.87
38.74
47.43
SD [60]
34.77
46.63
28.00
40.33
31.41
48.18
31.39
45.05
Ours
38.38
48.99
35.70
46.16
41.04
53.71
38.37
49.62
domains), OfficeHome [76] (15,588 images, 65 classes, domains), TerraInc [3] (24,788 images, 10 classes, 4 domains), and DomainNet [59] (586,575 images, 345 classes, 6 domains). We use the ImageNet [21] pretrained ResNet [30] as the backbone for both the experts and the target model. Following the designs in DomainBed, the hyper-paranne
o in Eq. (3) is randomly selected in range of [0.01. 10]. To ensure fair comparisons, all methods are reevaluated using the default settings in DomainBed in the same device with each of them evaluated for 3 x 20 times in different domains. Training- -omainn validation is adopted as the evaluation protocol. Other settings (batch size, learning rate, dropout, etc.) are dynamically selected for each trial according to DomainBed.
We observe that ERM can obtain competitive results among the models compared, which leads more than half of the sophisticated designs, and only 6 methods lead ERM in most datasets (with scores 2 3). We also note that current strategies cannot guarantee improvements for ERM in all situations, given that none of them with a score of 5. Differently, our method can consistently improve ERM in all evaluated datasets and lead others in average accuracy. Specifically, our approach obtains the leading results in 4 out of the 5 datasets, and it also improves ERM by large margin (nearly Spp) in the difficult TerraInc dataset. Compared to methods that explicitly explore hard samples representations (i.e. VREx and RSC) and that use MoE (i.e. Meta- Meta-DMoE), the performances of LFME are superior to them in all cases.
Results with ResNet18. Results are listed Tab. 1.
Results with ResNet50. Because larger networks require more training resources, we only reevaluate some of leading methods (i.e. ERM, Fish, CORAL, and SD). in our device when experimenting with ResNet50. We note that our method surpasses the baseline ERM model in all datasets and leads it by 1.8 in average. Meanwhile, our method can still outperform the second best (i.e. SD) by 0.7 in average. These results indicate that our method can consistently improve the baseline model, and it can perform favorably against existing arts when implemented with a larger ResNet50 backbone.
We also combine LFME with SWAD [8]. Same with the original design, the hyper-parane parammerrrt searching space this setting is smaller than the original DomainBed. We use the reported numbers in [8] for comparisons. As shown, our method can also improve the baseline and obtain competitive results when combined with SWAD. These results further validate the effectiveness of our method.
5.2 Generalization in Semantic Segmentation
Datasets and Implementation details. The training and test processes of the compared algorithms involve 5 different datasets: 2 synthetics for training, where each dataset is considered specific domain, and 3 real datasets for evaluation. Synthetic: GTAV [63] has 24,966 images from 19 categories; Synthia [64] contains 9,400 images of 16 categories. Real: Cityscapes [20] 3,450 finely annotated and 20,000 diverse urban driving scene images: Mapillary [55] includes 25K street view images. Following the design [40], we use the pretrained DeepLabv3+ [18] and ResNet50 as the segmentation backbone for experiments. The maximum iteration step is set to 120K with batch size of 4, and the evaluations are conducted after the last iteration step. The
images collected from 50 cities; Bdd100K [83] contains 8K
_ in Eq. (3) is fixed as 1 in this experiment for simplicity. We use the mean Intersection over Union (mloU) and mean accuracy (mAce) averaged over all classes as criteria to measure the segmentation performance.
Experimental results. Results are shown in Tab. 2. To better justify the effectiveness of our method, we reevaluate the baseline model, which aggregates and trains on all source data, and PinMem in our device. We also implement SD [60], which is a leading method in the image classification
7

---

(a) Input
(b) Bsseline
(c) SD
(d) Pinmem
(c) Ours
(f) Ground Truth
Figure 3: Qualitative comparisons. The compared methods make unsatisfactory predictions about several objects. such clouds with varying shapes, car logo, or people and car the shadow. Please zoom in for a better view.
Table 3: Evaluations of free lunch for DG (i.e. ERM+) and different LS ideas (i.e. LS [71], MbLS [50], and ACLS [58]). We use the suggested settings in their original papers for evaluating.
Target domain in PACS
Target domain in Terralne L38
Model
Avg.
Avg.
Art
Cartoon
Photo
Sketch
L100
L43
L46
ERM
78.0
1.3
73.4
94.1 0.4 73.6 2.2 79
42.1
30.1
122
48,9 0.6
34.0
38.8 1.0
ERM+ 81.9
=
0.7 94,8
73.8
8.4
46.7
37.1
34.8 0.4
4.7
LS
0.3
06
0.2
73.
02
48.1
1.
0.
3..1
0.5
MbLS
8.5
0.
9.
0.
81.7
4.3
39.1
52.
3388
4.5
ACLS
80.8
0.3
1.0
0.
0.6
45.6
36.8
4.
1.
414
LFME
$1.0 76.5
0.9
94.6
0.5
774 824
53.4
0.4
40.7
54.9
04
3674
0.5
task. Results from other methods are directly cited from [19]. We observe SD does not perform as effectively as it does in the classification task, which decreases the baseline model in most situations. In comparison, our LFME can boost the baseline in all datasets, similar to that in the classification task. also performs favorably against existing methods specially designed for the semantic segmentation task, obtaining best results in out of the 3 evaluated datasets in both mIOU and mAcc.
Visualized examples are provided in Fig. 3. We note that when it comes to unseen objects (i.e. clouds with different shapes and a new car logo), objects with an unfamiliar background, such as the person and car hidden in the shadow, due to the large distribution shift between real and synthetic data, compared methods make unsatisfactory predictions. In comparison, LFME can provide reasonable predictions in these objects, demonstrating its effectiveness against current arts regarding generalization to new scenes, These results validate the effectiveness of LFME and its strong applicability in the generalizable semantic segmentation task.
Analysis
Analyses in this section are conducted on the widely-used PACS dataset unless otherwise mentioned. Experimental settings are same as that detailed in Sec. 5.1. Please see the appendix for more analysis.
6.1 A Free Lunch for DG
As stated in Sec, 4.1, when the basis of discrimination not compromised (4. corresponds to larger value in the label and vice versa for qe for Vc # +), the increase of qe encourage the model to learn more information that is shared with other classes, and use them to improve generalization. Based on this analysis, it seems that using the one-hot label to regularize the logit is also reasonable. To validate this hypothesis, we replace qº in Eq. (3) with the ground truth and reformulate Lall into,
Lall = ((softmax()))))++ y) _ ||2 yll?. s.t. ==
(9)
Denoting as ERM+, results listed in the second row Tab. suggest that this idea can improve ERM in all unseen domains. Because the hard sample information is absent in this strategy, we observe it performs inferior to LFME. However, this alternative does not require experts or any other special designs, even the setting of the hyper-purameter cannot be wrong: Eq. (9) will approximate the baseline either with a = or a approximates +00. Thus, we argue this simple modification can serve as free lunch to improve DG. More evaluations of this idea are in our appendix.
8

---

6.2 Compare with Label Smoothing
As detailed in Sec. 4.1, LFME will explicitly constrain the probability of the target model within a smaller range. This effect may resemble the LS technique that aims to implicitly calibrate the output probability. However, compared to LS, LFME has two advantages. First, LFME does not involve heuristic designs of hyper-parameters for determining its probability, which is essentially required in existing LS ideas, such as e in [71] and predefined margin in [50, 58), avoiding the possibility of deteriorating the performance when not choosing them properly; Second, LFME can explicitly mine hard samples from the experts, further ensuring improvements for DG. We evaluate LS methods in both PACS and the difficult Terralnc datasets: (1) the pioneer LS method from [71]); (2) margin-based LS (MbLS) [50] that penalizes logits deviate from the maxima; (3) adaptive and conditional LS (ACLS) that can adaptively determine the degree of smoothing for different classes. Results are illustrated in 3rd-5th columns in Tab. 3. We observe that although these LS methods can improve the baseline, they are all inferior to LFME, validating the effectiveness of LFME against LS,
6.3 Compare with Other Knowledge Distillation Ideas
To test the effectiveness of our logit regularization (i.e. 1|z-9 we compare it with several other KD options, namely different combinations of the logits probabilities from the target model and the experts, including || 1 (which is the basic design Meta-DMoE [89]), l1q =f I|?. and l|q gf Moreover, we also compare it with a common practice in KD that uses the probability from the teacher (i.e. experts) as label for the student (i.e. target model), which reformulates the Eq. (3) into: Lall (1 - 9 (q. y) (q. and denoted as Hqq. in Tab, 4. Following the suggestion in [41]. we gradually increase & over the iteration steps to achieve better performance. Results listed in 2nd-5th columns in Tab. 4 show that not all KD strategies can improve DG and the logit regularization choice achieves the best result in terms of average accuracy. This can be explained by the analysis in Sec. 4.1 that our logit regularization can help using more information. Please also the explanation in Sec. C for details.
6.4 Compare with Naive Aggregation Ideas
Table 4: Out-of-domain evaluations of different KD ideas (using different Lguid in Eq. (2)) and naive aggregations.
To examine the effectiveness of LFME, we compare it with three different vari- ants that employ all the experts during inference: (1) Avg_ Expt, which aver- ages the output from all experts for pre- diction, similar to DAELDG [90]; (2) Model soup (MS) EXPT, which uni- formly combines the weights of differ- experts. This is inspired by the MS idea in [78]; (3) Conf_ Expt that utilizes the output from the most confident pert as the final prediction. Inspired by the finding in [77], the expert with the output of the smallest entropy value is regarded as the most confident one in test sample; (4) byn_Expt, which is a saaming-basee approach that estimates the domain label of each sample and dynamically assigns corresponding weights to the experts via a weighting module. Results are listed in 6th-9th columns in Tab. 4, where both the designs of handcraft (i.e. Avg_ Expt, MS _xppt, and Conf Expt) and learning-based (i.e. fail to improve the baseline model. This because the hand-craft designs and Conf Expt are rather unrealistic in practice as different models may contribute differently in the test phase, we thus cannot use a simple average or select the most confident expert for predicting. Meanwhile, the learning-based Dyn_ will inevitably introduce another generalization problem regarding the weighting module, complicating the setting. In comparison, LFME avoids the nontrivial aggregation design and can improve ERM in all source domains, further validating its effectiveness.
Target domain
Model
Avg.
Art
Cartoon
Photo
Sketch
ERM
78.0=1.3
73.420.8 94.1=0.4
73.6=2.2 79.8=0.4
Performances from different KD ideas
77.8
73.2=0.7
94.1=0.5
74.3
79.9.0.2
11q -
76.4+0.7
75.7=1.3
94.0=0.2
72.4=1.4
79.7=0.5
[1q - 112
81.3+1.
74.++1.4
94.1-0.5
74.0=28
81.1+1.1
H(g.90)
82.1+0.8
73.6=0.2
92.6=0.9 73.4+2.1
80.4=0.4
Performances from naive aggregation idcas
Avg Expt MS. Expt Conf Expt 74.8=0.8
78.4=1.3
65.0=1.6
92,4=0,3
71.8=0.5
76.9=0.1
76.8s2.3
63.0=0.6
93.420.2
72.6a1.5
76.520.6
64.8=1.8
91.9=0.5
72.6=2.6
76.0=0.8
Dyn Expt
68.4=1.8
65.1=1.1
92.320.5
68.1=1.9
73.520.3
Ours 81.0.0.9 76.5.0.9 94.6-0.5
77.4
82.4.0.1
aggregation skills Expt, MS Expt,

---

Table 5: In-domain evaluations of different models.
Source domain PACS
Source domain in Terralnc
Model
Art
Cartoon
Photo
Skelch
Avg.
1100
L38
L43
L46
Avg.
ERM
94.8 # 0.1 96.2 + 0.2 98.5 # 0.2 96.0 # 0.3 36.4.00
0..1 91.1 001 894+02
85.5 0.2
90.3 0.3
Experts
95.4 0.1
96.3
98.7
02
96.
90.
9200=11
90
8.1
91.1
Ours
95.7 + 0.2
96.8
0.3
98.8 + 0.2 96.4 9.1
96.1
+..1
92.6 0.2
90.7
+.7+002 0.2 87.2
91.6
Table Sensitivity analysis of LFME regarding the involved weight parameter / in Eq. (3). LFME degrades to ERM when 5 = O.
Target domain in PACS
Target domain in Terrannc
Model
Avg.
Avg.
Ant
Cartoon
Photo
Sketch
L100
L38
L43
146
=0
78.0
0384
94.1 + 0.4
73.6
2.2
79.8 0.4
42.1
30.1
+
1.2
48.9
0.6
34.0
38.8 1.0
=0.01
82.0 0.4
75.0 0.9
0.1
75.1
0.9
81.8 0.2
2.
40.0
1.2
51.9
11.900..
34,4 0.6
43.3 0.7
10
0.3
75.5
94,5
0.7
818
0,5
37,5
54,4
36.8
45.2
100
80.1
0.8
74.3
93.3
75.7
+
80.9
0.4
49.0
30.5
0.8
52.9
0.6
36.1 +
42.1 0.6
1000
77.1
1.0 72.7
+.9 92.5
0.1
74.7
79.3 0.3
0.1
46.5
+
29.1
19.1+12
1.2
49.6
+ 0.8
34.6
1.0
3 0.2
6.5
Does the Target Model Become an Expert on All Source Domains?
The training process of LFME aggregates all professional knowledge from the experts into one target model, aiming to make the target model expert all source domains. To examine if the goal has been achieved, we conduct in-domain validations for the target model and compare the performances with that of the experts and the ERM model in the PACS and TerraInc datasets. Note in these experiments, ERM and LFME are trained using the same leave-onc-out strategy, and the performances are averaged over the trials on three different target domains. Results are listed in Tab. 5, We observe that ERM performs inferior to the experts in the in-domain setting. The results are not surprising. As stated earlier, when encountering data similar to the source domain, it would be better to rely mostly on the corresponding expert than the model also contaminated with other patterns. In comparison, our method obtains the best results in all source domains because implicitly focuses more on the hard samples from the experts, which is shown to be an effective way to improve the performance many arts [33, 84). These results validate that the proposed strategy can extract professional knowledge from different experts, and enable the target model to become an expert in all source domains.
6.6 Selection of the
Compared with the baseline ERM, our method involves only one additional hyper- parameter (i.e. & in Eq. 3) which is randomly selected the range of [0.01. 10] in DomainBed. To evaluate the sensitiveness of LFME regarding this hyper-parameter, we conduct experiments in the PACS and TerraInc datasets by tuning it in a larger range. As seen in Tab. 6, LFME can obtain relatively better performance with either
0.01 or 10, and it performs on par with ERM even when " = 1000. These results indicate that our method is insensitive w.r.f the hyper-parameter. This is mainly because very large & will enforce the model to only learn from domain experts, given that of is mostly aligned with the label y (as seen in Fig. 2 (a)), the model will perform similarly to another form of the baseline [37] in this case.
7 Conclusion
This paper introduced a simple yet effective method for DG where the professional guidances of experts specialized in specific domains are leveraged. We achieve the guidance by logit regularization term that enforces similarity between logits of the target model and probability of the corresponding expert. After training, the target model is expected to be expert on all source domains, thus thriving in arbitrary test domains. Through deeper analysis, we reveal that the proposed strategy implicitly enables the target model use more information for prediction and mine hard samples from the experts during training. By conducting experiments in related tasks, we show that our method is consistently beneficial to the baseline and performs favorably against existing arts
Acknowwedgement. This work was supported by the Centre for Augmented Reasoning, an initiative by the Department of Education, Australian Government.
10

---

References
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint 19t.x:::977002999 2019.
[2] Yogesh Balaji, Swami Sankaranarayanan. Rama Chellappa. Metareg: Towards domain generalization using meta-regulariiztioon In NeurIPS, 2018.
[3] Sara Beery, Grant Van Hom, and Pictro Perona. Recognition in terra incognita. In ECCV, 2018.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains, Machine learning, 79(1):151-175, 2010.
[5] Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv preprint 17.00900 2017.
[6] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks new unlabeled sample. NeurlPS, 2011.
[7] Fabio Carlucci, Antonio D' Innocente, Silvia Bucci, Barhara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR, 2019.
[8] Junbum Cha, Sanghyuk Chun, Kyungjac Lec, Han-Cheol Cho, Scunghyun Park, Yunsung Lec, and Sungrac Park. Swad: Domain generalization by seeking flat minima. NewrIPS, 2021.
[9] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In ECCV, 2022.
[10] Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Leamingg balance specificity and invariance for in and out of domain generalization. In ECCV, 2020.
[11] Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoging Liu, and Yizhou Yu. Compound domain general- ization via meta-knowledge encoding. In CVPR, 2022.
[12] Chaoqi Chen, Luyao Tang, Yue Huang, Xiaoguang Han, and Yizhou Yu. Coda: generalizing to open and unseen domains with compaction and disambiguation. NeurlPS, 2023.
[13] Chaoqi Chen, Luyao Tang, Feng Liu, Gangming Zhao, Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. In NeurlPS, 2022.
[14] Liang Chen, Yong Zhang, Yibing Song, Anton van den Hengel, and Lingqiao Liu. Domain generalization via rationale invariance. In ICCV, 2023.
[15] Liang Chen, Yong Zhang, Yibing Song. Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR, 2023.
[16] Liang Chen, Yong Zhang, Yibing Song, Jue Wang, and Linggiao Liu. Ost: Improving generalization of deepfake detection via one-shot test-time training. NewrIPS, 2022.
[17] Liang Chen, Yong Zhang, Yibing Song, Zhen Zhang, and Lingqiao Liu. A causal inspired aarly-branching structure for domain generalization. IJCV, pages 1-21, 2024.
[18] Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
[19] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR, 2021.
[20] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson. Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
[22] Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NeurIPS, 2019.
11

---

[23] Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilizaaion of multiple datasets and web images for softening bias. In ICCV, 2013.
[24] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavio- lette, Mario Marchand, and Victor Lempitsky. Domain-adversan training of neural networks. JMLR, 17(1):209- 2030,, 2016.
[25] Robert Geirhos, Jörn-Herrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(16):65-667, 2020.
[26] Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEFF TPAMI, 39(7):1414-1430, 2016.
[27] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. ICLR, 2021.
[28] Jiang Guo, Darsh Shah, and Regina Barzilay, Multi-source domain adaptation with mixture of experts. arXiv preprint 1rXXvv1809002256 2018.
[29] Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Abu-Husscin, Elad Amrani, Roci Herzig, Amit Alfassy, Raja Giryes, Hilde Kuchne, et al. Unsupervised domain generalization by learning a bridge across domains. In CVPR, 2022.
[30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[31] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge neural network. arXiv preprint w 2(7), 2015.
[32] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In UAI, 2020.
[33] Chen Huang, Chen Change Loy, and Xiaoou Tang. Local similarity-awan deep feature embedding. In NewrIPS, 2016.
[34] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In CVPR, 2021.
[35] Zeyi Huang, Haohan Wang, Dong Huang, Yong Jac Lec, and Eric P Xing. The two dimensions of worst-case training and their integrated effect for out-of-domain generalization. In CVPR, 2022.
[36] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. ECCV, 2020.
[37] Like Hui and Mikhail Belkin. Evaluation neural architectures trained with square loss vs cross-entropy in classification tasks. In ICLR, 2021.
[38] Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
[39] Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In /CCV, 2021.
[40] Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learning to generalize semantic segmentation. In CVPR, 2022.
[41] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In ICCV, 2021.
[42] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Priol, and Aaron Courville. Out-of-diitrributio generalization via risk extrapolation (rex). In ICML, 2021.
[43] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In /CCV, 2017.
[44] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-leaminn for domain generalization. In AAAI, 2018
12

---

[45] Da Li. Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, 2019.
[46] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, 2018.
[47] Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In ICCV, 2021.
[48] Xiaotong Li. Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling- Yu Duan. Uncertainty modeling for out-of-distribution generalization. ICLR, 2022.
[49] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018.
[50] Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smoothing for network calibration. In CVPR, 2022.
[51] David JC MacKay, Information theory, inference and learning algorithms. Cambridge university press, 2003.
[52] Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In /CCV, 2017.
[53] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICMI., 2013.
[54] Hyeonseob Nam, HyunJae Lee. Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, 2021.
[55] Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In CVPR, 2017.
[56] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing leamnng and generaliza- tion capacities via ibn-net. In ECCV, 2018
[57] Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization on unseen domains via inference-time label-preserving target projections. In CVPR, 2021.
[58] Hyekang Park, Jongyoun Noh, Youngmin Oh, Donghyeon Back, and Bumsub Ham. Acls: Adaptive and conditional label smoothing for network calibration. In ICCV, 2023.
[59] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Sacnko, and Bo Wang. Moment matching for multi-source domain adaptation. In JCCV, 2019.
[60] Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. NeurlPS, 2021.
[61] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. /CML, 2020.
[62] Alexandre Rame, Corentin Dancette, and Matthieu Cord, Fishr: Invariant gradient variances for out-of- distribution generalization. JCMI,, 2022.
[63] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladien Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
[64] German Ros. Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
[65] Yangjun Ruan, Yann Dubois, and Chris Maddison. Optimal representations for covariate shift. In ICLR, 2022.
[66] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, 2020.
[67] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross- goos-egradeent training. 2018.
13

---

[68] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technicul journal. 27(3):379 1948.
[69] Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In ICLR. 2021.
[70] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
[71] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zhigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[72] Jiaxi Tang. Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03532. 2020.
[73] Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang. Mu Li, and Dimitris N Mctaxas. Crossnorm and selfnorm for generalization under distribution shifts. In ICCV, pages 52-61, 2021.
[74] Vladimir Vapnik. Principles of risk minimization for learning theory. In NeurlPS, 1991.
[75] Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
[76] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
[77] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[78] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
[79] Qinwei Xu, Ruipeng Zhang. Ya Zhang, Yanfeng Wang, and Qi Tian. fourier-based framework for domain generalization. In CVPR, 2021.
[80] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.0066 2020.
[81] Fu-En Yang. Yuan Chia Cheng, Zu Yun Shiau, and Yu Ch-anhnng Frank Wang. Adversarial leacher-studens representation learning for domain generalization. In NeurlPS, 2021.
[82] Huaxiu Yao, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. Leveraging domain relations for domain generalization. arXiv preprint arXiv:2302.02609 2023.
[83] Fisher Yu, Haofeng Chen, Xin Wang, Wengi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, 2020.
[84] Yuhui Yuan, Kuiyuan Yang. and Chao Zhang. Hard-aware deeply cascaded embedding, In ICCV, 2017.
[85] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-rea generalization without access- ing target domain data. In ICCV, 2019.
[86] Seniha Esen Yuksel, Joseph N Wilson, and Paul Gader. Twenty years of mixture of experts. IEEE TNNLS, 217):1177-1193, 2012.
[87] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group distribution shift. arXiv preprint arXiv:2007.02%: 029331 2020.
[88] Guanhua Zheng, Jitao Sang, and Changsheng Xu. Understanding deep lcarning generalization by maximum entropy. In ICLR, 2018
[89] Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang. Meta-dmoc: Adapting to domain shift by meta-distillation from mixture-offexppers. NeurlPS, 2022.
[90] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. JEEE TIP. 30:8008-800 2021.
[91] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR. 2021.
14

---

Appendix
In this section, we provide
Limitations and future works in Section
2. Pseudocode of LFME in Section B.
3. Explanations of why existing KD ideas perform inferior to our LFME in Section C.
More analysis regarding LFME in Section
Compare with Moseb methods and other related methods in Section
6. Existing Ideas with Same Training Resources in Section F.
7. Detailed settings of the learning-basee baaningg-asedd aggregation method Dyn Expt in Sec. 6.4 from the manuscript in Section G;
8. Detailed results of LFME in the unseen domains of the different datasets from DomainBed, and detailed results LFME in the different categories from semantic segmentation task Section
A Limitations and Future Works
Although our method shows competitive results for generalization, there are certain occasions when will encounter setbacks. First, LFME is not applicable for the setting where the domain labels unavailable in the training data, such as those collected from the internet. Since the training of the experts requires data grouped by domain labels. Second. LFME cannot handle the situation when only one source domain is provided, preventing it from performing in more difficult single-source generalization task. How apply LFME to more general setting will be our primary task in future works. Besides, the designs and theoretical supports are built mainly for the classification task, finding proper solution to extend them to the regression tasks also a promising direction in potential future works.
B Pseudo Code of LFME
This section provides the Pytorch-style pseudocode of method. As detailed in Alg. 1, the implementation our method is embarrassingly simple, introducing only one hyper-parameter upon the baseline ERM.
C Compare with Current KD Based on the Analysis in Sec. 4.1
As observed Sec. 6.3, some existing KD ideas, perform inferior to our L.ME. Based on our analysis, we find this is because existing ideas have difficulties achieving the beneficial effect introduced in Sec. 4. As detailed, our logit regularization ensures using more information for the target model by implicitly regularizing the probability in much smaller range. This is hard to achieve by H(q, q and #12, where their probability q will be in a similar range as that in ERM because the soft label gf is still within the range of [0, 1]. enforcing similarity between q and g will not significantly change distribution. Moreover, this effect is also hard to achieve by -|z-z and llq - z 112, which do not provide specific range regularization effect for the probability.
D Further Analysis
This section provides more evidence support the two analyses in Sec. in the manuscript. The experiments are conducted on the widely-used PACS dataset and the difficult TerraInc dataset unless mentioned otherwise.
D.1 More Justification for Enabling Target Model to Use More Information
For the baseline model, we infer that the cross-entropy loss alone will enforce the model only learn discriminative features that are specific to the label category. Because there are no relevant features to
15

---

Algorithm 1: PyTorch-style pseudocode of LFME.
3 M: total domain numbers
a slpha: weight pazameter
and weight desay: selected hyper-parameters
network [None] experts and the target model
4 Initializa:ion:
experts target model
params =
for in (X 1):
network[i] Reskett
paroma.appere "params"; network parametora optimiser - Adam (párams, ir, welgnt_decay)
Training: experts and the target model def train (nniibatitsstthhss))
loss
All samples from the M domains
all_x torch. for y minibatches]) all_y -torch.cat catily fly for x, y in minibatches]) for i in range [M) Training the M experts
training 1-th expert using 1-th domain data xi, yi - sinimebtch[1] [0]] [An] image label pai: from the 1 th. domain z_Bi network(i] (xi) logit from the i-th expert nota the softmax functior is encoded cressent loas pytorch loss i-
Eq, (1) in the manuscript
Concat probabilities qa poftmax(z_Ei) if i==0 clse cst ((ge, poftmax(z_Bi)) 0)
experts
network[ 11 (all_x logit target:
Licie --gii:: Eq. (3) in the manuscript
loss -= crossentropy (softmax all_y) + +lalpha MSRtoss (x, (R.detach
Updating the experts and target model reiimizer.zero... backwarc) cptimizer.stepl
target
test (teest__mmpleess) result - network [-1] (teat_samplea)
interpret the non-label classes in ERM, the confidence for the target class will approach the maximum during training, and vice versa for other non-label classes. Differently, when the output probability has a smoother distribution, the model will also learn information that is shared with other classes to improve the corresponding non-label probabilities. Introducing more information during training will inevitably cause the phenomenon of low confidence for the label category, as probabilities for the non-label classes will increase (their summation increases from nearly Fig. (a) 0.35 in Fig. 2 (b)). Note that low confidence does not mean low accuracy. Because the label category corresponds to the largest logit value in both terms, and both the discriminative and newly introduced information can be used to characterize the label information, exploring more information can also boost classification. This finding further validated by the in-domain results in Tab. 5, where L.FME shown to be able to also improve the in-domain test results compared to ERM, indicating that low confidence our case actually leads to higher accuracy.
Visual examples to support the above analysis are shown in Fig. 4. We observe that compared to the baseline ERM, our method can utilize more regions for prediction. Such as that in the first "horse" examples, our model utilizes both the head, tail. and body for prediction, while the ERM seems to focus only on the head region. These visual samples align with our analysis that compared to the baseline ERM, L.ME tends to use more information for prediction.
D.2 Other Alternatives to Enlarge for Vc #
This section provides more evidence support the first analysis Sec. by conducting experiments on two variants that also enlarge qe for Vc # s, (1) we test model that directly enlarges qe for Ve Specifically, we use the probability from LFME as the label to guide the probability output of new target model, i.e. Lgnid =|1 1ilggg-FMEEp with q and L from the new target and LFME models, respectively. Ideally, this modification (i.e. LFME (__id can also improve the baseline model because it also encourages the model to learn more information shared with other classes
16

---

(a) Input
(b) ERM
(c) Ours
Figure 4: Grad-CAM visualizations of samples from the unseen "cartoon" domain of the PACS benchmark, which is the most challenging domain for ERM and our method according to Tab, Compared to the baseline ERM, highlight regions from our method contain more information related to the label category. These visualizations can further validate our analysis in Sec, that with the proposed strategy, the target model can explore more information for prediction.
Table Further evidences to support our deep analyses in Sec. Evaluations are conducted on the widely-used PACS and difficult TerraIne datasets using settings from the DomainBed benchmark. Here ERM+ is the free lunch introduced in Sec. 6.1 that replaces gf in Eq. (3) with the one-hot label; LFME Guid denotes imposing the guidance from LFME to the ERM model by including Lguid = | " ME where and q² -.FME are probabilities from the ERM model and LFME; Self Guid is the model that replaces probabilities from the experts Eq. (3) with that from itself, where softmax (z) is followed with detach operation; ERM+ w/ expt denotes explicitly focus more on the hard samples from the experts based on ERM+; ERM+ self denotes explicitly focus more on the hard samples from the model itself on the basis of ERM+. All methods are with the same ResNet18 backbone and are examined for 60 trials in each unseen domain.
i.e. Lguid ||| - -softmax
Target domain in PACS
Target domain in Terrainc
Model
Cartoon
Photo
Sketch
Avg.
[00
LA3
146
Avg.
ERM
78.0
03 94.1 0.4 73.6
79.8
0.4
42.
25
30,
4..9
340=
38.8
ERM+
81.9 1.9 0.4 75.1 0.7
94.8 1 73.8 2.2 81.4
46.7
2.6
37.1 1
53.2
34.8
1.3
42.9 1.7 0.7
Performances of other altematives to cnlarge for Vo
LFME Guid
8..6
0.9
0.6 76.1 2 81.3
48.0 2.1 36.8
53.2
35.6204
43.4
Self. Guid
81.5 + 1.0
75.1 0.5 95.1 0.3
81.4
48.0 - 35.9 0.6 53.9
36.7 1.9 43,6 + 0.7
Comparisons between hard samples from different models
HRM+ w/expe
80.8 +0.
74.2
95.0 0.5 76.0 81.3
512-03 370 533
365-11
44.
ERM+ w/ self
82.9 0.4 75.2 = 0.3 94.0 0.4 + 81.5
00 35.7 + 2.0 $11
34.2 0.8
428
LFMB
81.0 2
76.5 = 14.6 77.4 82.42.11
33.4
40.7 1 549
36.40.7
46.3 0.5
without compromising discrimination; (2) we use the output probability from the target model itself to replace that from the experts in Eq. (3), which reformulates Lguid into || (2) (softmax( z) followed with detach operation in updating). According to our analyses in Sec. 4, this alternative (i.e. Self Guid) should also boost generalization because it enforces the target model to learn more information and focus more on the hard sample from itself;
Results are illustrated in Tab. 7. We observe that both the two variants LFME Guid and Self Guid are beneficial the baseline models, leading ERM in all unseen domains. These results further validate our first analysis in Sec. 4, and we can conclude that when the basic discrimination is not compromised, either directly or indirectly enlarging 4e for Ve can help the target model learn more information, and improve generalization accordingly.
17

---

D.3 Hard Samples from the Experts
As analyzed in Sec. 4. LFME not only encourages the target model to learn more information but also implicitly helps it to focus more on the hard samples from the experts. To examine if the classifications of hard samples from the experts are indeed improved, we plot the classification ratio p-miu(p)
P. R max(p)' where p denotes the normalized probability: p max(p)-min(p) - from the target model on these hard samples. We use the ratio because it can quantify the correct predictions regarding the hard samples more precisely than the accuracy: the closer R approaches I, the better the corresponding model performs on the sample. In this setting, 1/3 samples with larger losses in training batch are considered hard samples. We compare R from LFME with that from ERM to highlight the difference. As can be observed in Fig. 5 (a), most R from LFME is larger than that from ERM over the iterations, denoting LFME is better at handling these hard samples than ERM. As a comparison, R from these two models almost overlap in the easy samples as shown in Fig. 5 (b). These results validate our analysis that LFME implicitly helps the target model to focus more on the hard samples compared with ERM.
Meanwhile, to validate if the hard samples from experts can indeed help generalization, we conduct experiments by explicitly putting more weights on the hard samples from the experts the basis of the free launch ERM+ (i.e. ERM+w/ expt). Specifically, the target objective based on Eq. (9) is adjusted into: w wLaul. where is the weight for the training samples and is determined based on the loss of the experts. Basically, the larger the corresponding loss from the experts, the larger the value of w should be, and As can be observed in Tab. 7, mining hard samples from the experts can improve the base model of ERM+, especially in the difficult TerraInc dataset. These results validate that mining hard samples from experts is beneficial for generalization.
the strategy from [35] to implement this hard sample mining process.
D.4 Comparisons Between Hard Samples from Different Models: Why Experts are Required
We note in Tab. 7, the model Self_Guid performs inferior to LFME in both the two datasets, and the improvements are also marginal compared with ERM+ on account of variances. These results indicate the hard samples from the model itself may be less effective for generalization compared with that from the experts. To validate this hypothesis, we conduct experiments by explicitly focusing more on the hard samples from the model itself on the basis of ERM+ (i.e. ERM+ w/ self). The implementation the same as that in ERM+ w/ expt.
Results listed in the 6th row in Tab. 7 validate this hypothesis, where the hard samples from the model itself can hardly improve generalization compared with that from the experts. We presume the main reason is that besides the dominant specific domain information, hard samples from the experts also contain some out-of-the-domain information that makes it challenging for the corresponding expert. Examples are shown in Fig. 6. We observe that compared with hard samples from the model itself (i.e. Fig. (b)), hard samples from the experts (i.e. Fig. 6 (a)) contain more ambiguous data located in the mixed region or the boundary of two domains than, indicating the experts may be more effective revealing samples that with characteristics from different domains. By implicitly emphasizing these oftt-ofthe-sspe tite--pecffc-ddmmae improving generalization accordingly. Because the naive training strategies are fed with data from information is difficult to discover
samples, can the target model learn domain domain-agnostie features, thus
different domains at the same time, the out-of wtwe--secific-dooeee in their framework, explaining why hard samples from other strategies may not contribute. This is also the reason why the experts must be involved in the overall framework.
E
Compare with MoE-Based Ideas and Other Arts
Since our idea involves training multiple experts, one may connect it to the idea of mixture of experts (MoE) [86]. However, it is noteworthy that our LFME is not simple implementation of existing techniques. The introduced logit regularization term new and can be properly explained for its effectiveness. Some similar ideas that use MoE have been explored in other works for improving DG. For example, in DAELDG [90], following shared feature extractor, each domain corresponds to a specific classifier (i.e. expert), whose output is enforced to be similar to the average outputs from the non-expert classifiers. Different from our work, it uses the average outputs from different experts as the final result. In Meta-DMof [89], a meta-learning- based framework is employed to enforce
18

---

1.0
1.00
0.9
0.8
0.90
0.7
0.85
0.6
0.80
0.5
R from LFE
R from LFE R from ERM
R from ERM
0.70
Iteration steps
Iteration steps
(a) R from the hard samples
(h) R from the casy samples
Figure 5: Classification ratio comparisons of ERM and LFME the hard and easy samples from the difficult TerraInc dataset. The closer the ratio R approaches the better the corresponding prediction. Here the hard samples are specified by the experts: the 1/3 samples in a training batch with larger losses from the experts, and the easy samples are the leading 1/3 samples with smaller losses. The two models perform evenly well on the easy samples, while LFME obtains better results than ERM in the hard samples.
carnoo
pooto
skotch
Cartaon
photo
sketch
(a) Hard samples from experts
(b) Hard samples from the model itself
Figure 6: T-SNE visualizations of hard samples (larger dots) from different models. Here the data from different domains are clustered by their styles [91] (i.e. feature statistics from the first layer in the same ResNetl8 8 model), and 10 percents of samples with larger loss are considered hard samples domain. Hard samples from the experts contain more ambiguous data located in the mixed region or the boundary of two different domains.
the feature of the target model to be similar to the aggregated features from the experts. which is different from our logit regularization idea. Sec. D.1 and 6.3, we implement the basic designs of these two ideas with the framework of LFME and show their ineffectiveness. Besides the comparison with Meta-DMoE in Tab. settings in the PACS and OfficeHome datasets with the benchmark provided in [90]. We also compare with several other recent DG methods. Results Tab. show that LFME performs favorably against DAELDG, and outperforms Meta-DMoEE further validating the advantages of the logit regularization term. We also note LFME performs better than other arts with a different benchmark, demonstrating the effectiveness of the proposed approach.
this section, we further compare LFME with them using their original
F Existing Ideas with Same Training Resources
Since LFME requires learning different experts during training. inevitably using more parameters than existing methods, one may wonder if the effectiveness LFME derives from using these extra parameters from the experts. To examine this idea, we conduct experiments by implementing some leading arts (i.e. CORAL, SD) and the baseline ERM with + times of model size (where domain number). Specifically, the feature extractor for the larger method will contain M + branches, each with the same pretrained ResNet backbone. We concate the final outputs from the different branches and use it input for classifier to obtain the final result. Note that in this setting, a sample
19

---

Table 8: Out-of-domain evaluations of other related methods with the benchmark provided in [90].
Target domain in PACS
Target domain in OfficeHome
Model
Avg.
Avg.
Art
Cartoon
Photo
Sketch
Art
Clipart Product
Real
Compared with other recent methods
ERM
77.0
75.9
96.0
69.2
79.3
58.9
49,4
74.3
76.2
64.7
CCSA [52]
80.5
76.9
93.6
66.8
79.4
59,9
49.9
74.1
75.7
64.9
JiGen [7]
79.4
75.3
96.0
71.6
80.5
53.0
47.5
71.5
72.8
61.2
CrossGrad [67]
79.8
76.8
96.0
70.2
58.4
49.4
73.9
75.8
64.4
Epi-FCR [45]
82.1
77.0
93.9
73.0
81.5
DMG [10]
76,9
80.4
93.4
75.2
81,5
Compared
with MoB. based methods
DAELDG [90]
84.6
74.4
95.6
78.9
83.4
59.4
55.1
74.0
66.1
Meta-DMoF [89]
83.2
76.8
95.4
76.6
58.9
55.5
73.6
74.4
65.6
Ours
83.4
78.3
96.8
76.1
83.7
60.4
55.7
74.6
75.1
66.5
Table 9: Existing methods with same resources during training. We expand existing ideas by concatenating M+1 branches to ensure they use the same parameters as LFME during training.
Target domain
Model
Avg.
Art
Cartoon
Photo
Sketch
Performances from original designs
ERM
78.0 1.3
73.4 # 0.8
94.1 + 0.4
73.6 + 2.2
79.8 + 0.4
CORAL
81.5 0.5
75.4 0.7
95.2 + 0.5
74.8 = 0.4
81.7 0.0
SD
83.2 0.6
74.6 # 0.3 94.6 + 0.1
75.1 # 1.6
81.9
Performances from M + model sizes
ERM
76.6 1.1
7
94.5 + 0.2
73.1 1.7
79.8 0.4
CORAL
81.2 0.5
75.82 0.4
95.4 + 0.2
75.4+0.7
81.9 + 0.1
SD
81.6 0.8
75.4 0.5
94.7 0.1
76.9 # 0.8
82.2 + 0.4
Ours 81.0
76.5 # 0.9 94.6 0.5
7..4
0.14
will go through M times of forward passes for both training and inference, which is more than that of LFME design. Results shown in Tab. 9. We note that compared to the results from the original models, when using the same pretrained knowledge, naively expanding model size cannot improve the performance. The reason may be that well-pretrained small backbone already saturate on limited training data (as shown in Table 5, ERM can achieve more than 0.96 acc in the source domains), thus it is unnecessary to use parameters in these datasets.
G Detailed Settings of
Besides the different experts E; where i denotes the domain label, weighting network W estimate the labels of the training data. During the training stage, the experts are trained the same as that in LFME, and the weighting network is trained using the corresponding domain labels by minimizing (W('')),)) ()).. During the test phase, Dyn Expt dynamically combines the outputs from the experts with the corresponding domain probability: for Vx E D M+11. the final result is M. (x).. We use the same backbone for both the experts and the weighting module in this experiment.
also uses an extra
H Detailed Results
This section presents the detailed results on the semantic segmentation task in Tab. 10, and detailed results the unseen domains from the different unseen domains DomainBed benchmark in Tab. 11, 12, 13 14, and 15.
20

---

Table 10: Evaluations the semantic segmentation task with the metrics of Mean IoU (%) and per-class IoU (%). Source data from the synthetic GTAV [63] and Synthia [64] datasets, and the target data is from the real-world Cityscapes [20], BDD100K [83], and the Mapillary [55] datasets. All models are with the same backbone: Deepl.ab V3+ with ResNet50, and results with are from [19].
Silewalk
building
rond
well
Fence
pole
t-light
1-5ggm
tarain
1ider
truck
triin
m-hike
bicycle
Methods
mloU
Baseline'
72.7
36.4
64.9
11.9
2.8
31.0
37.7
20.0
84,9
14.0
71.9
65.3
9.9
84.7
11.6
25.4
0.0
10.6
18.1
35.46
Cityspaces
IBN-Net
29.5
35.55
RobusiNet!
82.6
40.1
73.4
17.4
1.4
34.2
38.6
18.5
84
16.9
81.9
65.2
11.4
84.7
7.2
23.6
0.0
10.4
23.9
37.69
Baseline
65.4 34.1
64.0
20.8
20.7
28.8
41.3
234
83.5
33.7
58.2
6499
13.9
69.6
23.5
14.7
9.8
17.3
19.4
37.19
PinMem
84.6 43.3
79.0
20.3
7.5
38.1
23.4
86.0
69.8
18.9
82.1
25.6
35.7
25.8
21.5
41,86
SD
52.9 33.6
55.2
21.3
19.2
29.1
41.2
22.6
83.7
35.0
54,9
64.8
12.3
61.6
23.6
14.3
9.4
14.4
11.7
34.77
Ours
71.9 32.7
19.4
30.5
84.0
82.8
67.1
15.4
1.9
16.0
38.38
Baseline
44.6 26.1
34.7
6.9
29.5
39.1
20.5
64.9
10.8
51.6
50.6
10.2
63.9
1.1
0.0
5.5
10.1
25.09
BDD100K 100K
Nei
RobustNer
69.5 35.0
36.6
40.5
27.3
71.6
14.0
83.6
17.3
61.9
4.4
0.0
24.3
34.09
Baseline
57.1
27.3
37.3
20.5
29.8
36.8
22.1
59.0
23.2
35.1
48.4
9.5
70.6
15.3
0.0
19.3
8.5
27.95
PinMem
73.4 36.8
56.7
25.1
32.8
37.5
24.1
71.7
23.2
72.1
17.
68.1
9.4
29.1
0.0
16.7
11.6
34.94
SD
56.6 29.6
39.6
2.8
17.4
31.8
37
19.6
60.3
23.4
36.6
49.6
10.5
67.1
15.1
9.3
0.0
19.9
5.5
28.00
Ours
72.2
30.5
35,70
Baseline
62.0 36.3
32.5
9.5
7.7
29.9
40.5
22.5
78.6
40.9
61.0
59.4
6.4
78.3
5.1
5.1
0.1
9.0
21.8
31.94
67.4
10.2
7.6
36.0
40.1
92.1
14.0
10.7
3.5
15.3
38.00
Mapillary
RobustNat!
78.0
41.0
56.6
13.1
6.2
39.4
41.3
36.1
79.5
34.7
90.0
12.0
76.1
10.7
13.1
0.8
16.9
24.8
38.49
Baseline
58.1
314
34.7
9.0
18.0
30.8
38.3
15.1
69.8
30.3
54.3
55.7
866
77.7
22.9
6.0
7.6
18.3
21.8
32.01
PinMem
76.0
40.8
14.5
36.6
77.3
17.6
18.7
39.41
SD
58.3
31.8
37,8
9.0
12.6
31.5
38.1
9.8
68.7
30.6
56.3
55.5
8.4
72.7
23.6
7.5
7.5
17.3
18.0
31.41
Ours
72.3 36.4
15.1
38.5
73.4
91.0
15.8
74.0
34.2
18.6
7.7
21.6
41.04
Table 11: Average accuracies on the PACS [43] datasets using the default hyper- settings in DomainBed [27]. TT denotes the average training time (minutes) for one trial target domain.
art
cartoon
photo
sketch
Average
TT
ERM [75]
78.0 + 1.3
73.4
0.8
94.1 0.4
73.6 * 2.2
79.8
24
IRM [1]
76.9 + 2.6
75.1 + 0.7
94.3 + 0.4
77.4 + 0.4
80.9 + 0.5
18
GroupGRO [66]
77.7 + 2.6
0.3
0,3
74.8 + 1.3
80.7 0.4
24
Mixup [80]
1.1
74.2
0.3
68.3
2.7
79.2
18
MLDG [44]
78.4
75.1 + 0.5
9.4
76.7 0.8
8.2
32
CORAL [70]
81.5
7
0.5
74.8 0.4
81.7
MMD [46]
81.3
+
9.5
74.3
81.3
18
DANN [24]
0.6
72.5
0,5
70.8 +..8
7.3
17
CDANN [49]
0.8
73.7
93.6
7.72
8.5
[TL
7.6
73.4
9..6
7..4
80.1 0.8
18
SagNet [54]
0.4
73.2
9.56
1.8
81.7 +
ARM [87]
7..4
7..0
9.66
73.8 0.6
80.6
24
VREx [42]
7.7
7.40
0,3
7.
80.2 0.5
17
RSC [36]
78.5 1.1
7.9
9.66
1.45
80.5
Meta-DMoE [89]
78.7 0.5
7..2
0.3
7.97
81.0
45
SelfReg [39]
82.5 0.8
7.54
0.5
74.9
81.8 0.3
25
MIRO [9]
79.3
68.1
0.3
60.6
75,9
31
MixStyle [91]
82.6
7..3
0.3
77.5
82.6
Fish [69]
80.9 1.0
7.49
9.4
76.2
820
52
[60]
83.2
7..6
9.1
75.1
81.9
CAD [65]
8.8
7.42
9.
75.0
8
CondCAD [65]
79.7
74.2
94.6
74.8
Fishr [62]
81.2
7.88
9.33
0.6
8.3
17
ITTA [15]
84.7 0.4
78.0
9.45
7.3
838
HRM+
81.9 0.4
7..1
9.7
73.8 2.2
81.4
Ours
81.0 0.9
7.9
9.56
77.4
8.14
21

---

Table 12: Average accuracies the VLCC [23] datasets using the default hyper-parameter settings in DomainBed [27].
Caltech
LabelMe
VOC
Average
ERM [75]
97.7 # 0.3
62.1 = 0.9
7
73.2 + 0.7
73,8 = 0,2
IRM [1]
96.1 0.8
62.5 = 0.3
0.7
72.0
15.100.1
GroupGRO [66]
96.7
61.7 1.5
1.8
0.6
75.4
Mixup [80]
95.6 1.5
6.4
7.3
7.24
0.3
MLDG [44]
95.8 0.5
6.8
6.5
0.8
0.3
CORAL [70]
96.5 0.3
6.1
6.61
73.8
7..5
MMD [46]
96.0 0.8
6.6
6.66
0.1
0.5
DANN [24]
9.1
6.6
7.92
0.2
7.2
CDANN [49]
95.4
6.6
1.3
0.5
76.0 0.5
MTL [5]
2.3
6.6
0.66
1.
0.3
SagNet [54]
0.7
6.7
6.3
0.6
7.84
ARM [87]
96.9
6.4
7.16
0.4
0.3
VREx [42]
96.2
1.3
6.
1.2
0.6
RSC [36]
96.2 0.0
63.6
1.0
0.40
0.3
Meta-DMoF. [89]
96.4
6.0
7.33
1.1
7.6
SelfReg [39]
95.8
6.1
71.1
0.6
7.7
MixStyle [91]
9.3
6.16
7.74
7.
7.7
Fish [69]
9.2
6314
7.,5
7.72
7.2
SD [60]
0.4
6202
69.7
0..6
0.4
CAD [65]
9.9
6.6
7.,4
1.34
0.6
CondCAD [65]
9.8
6..620.4
6.21
7.20
0.3
Fishr [62]
0.6
6.7
7.64
7.80
76.2
ITTA [15]
9..9
6.17
0.3
0.8
76.9 + 0.6
ERM+
96.0 0.3
0.4
7.4
1.2
76.1 # 0.4
Ours
96.4 0.3
62.8
0.84
76.2 0.4
Table 13: Average accuracies the OfficeHome [76] datasets using the default hyper-parameter settings in DomainBed [27].
art
clipart
product
Average
ERM [75]
52.2 0.2
48.7 0.5
69.9
71.7 4 0.5
60.6 = 0,2
IRM [1]
49.7 2 0.2
46.8
0.4
68.1
58.0 2 0.1
GroupGRO [66]
52.6 1.1
0.9
69.9
0.8
60.6 0.3
Mixup [80]
54.0
0.7
7.77
0.36
6..7
MLDG [44]
5.3
0.3
7.7
7.47
60.9
CORAL [70]
55.1
4.9
7.2
0.5
0.44
MMD [46]
50.9 1.0
0.3
6.7
1.3
0.4
DANN [24]
0.5
0.1
6.7
0.7
0.5
CDANN [49]
5.5
0.6
6,5
0.4
0.4
MTL [5]
51.6
0.5
6.31
0.6
0.5
SagNet [54]
5.4
0.2
7.41
0.4
0.3
ARM [87]
5.9
4.4
6..0
0.1
0.
VREx [42]
51.1
0.6
6..0
0.4
5.1
RSC [36]
49.0
1.5
6.78
0.3
0.6
Meta-DMoE [89]
54.7
5.94
7.38
0.1
0.1
SelfReg [39]
55.1
4.6
0.32+
0.
6.14
MixStyle [91]
50.8 0.6
51.4
6.3
0.5
59.6 0.8
Fish [69]
54.6
1.06
7.6
0.2
6.60
SD [60]
5..0
5.5
0.2
0.3
62.9
CAD [65]
5.61
0.5
6.37
7,49
60.5
CondCAD [65]
5.63
4.24
6.,8
7.16
61.0
Fishr [62]
5.
4..6
6.69
7.440.4
6..9
ITTA [15]
5.24
5..3
0.3
71.7
62.0
ERM+
5..1
5..0
0..
0.25
6.12
Ours
56.4
5..1
7.25
0.1
63.2 0.1
22

---

Table 14: Average accuracies the Terralnc [3] datasets using the default hyper- settings in DomainBed [27].
L100
L38
L43
L46
Average
ERM [75]
42.1
2.5
1.2
48.9 0.6
34.0 + 1.1
38.8 : 1.0
IRM [1]
41.8 + 1.8
29.0 2 3.6
2.1
33.1 +: 1.5
38.4 + 0.9
GroupGRO [66]
45.3 4.6
36.1 4.4
5.8
3.9
41.5 2.0
Mixup [80]
49.4 2.0
1.8
5.70
3.9
4.7
MLDG [44]
39.6
2.7
5.5
3.5
4..1
CORAL [70]
4.2
3.3
1.9
3.7
41.4118
MMD [46]
4.2
3.84
50.4
3.5
42.0 1.0
DANN [24]
3.6
1.5
1.0
0.6
0.9
CDANN [49]
3.4
3.2
11.1
3.5
3.3
MTL. [5]
2.6
3160
50.6
34.9
40,4 1.0
SagNet [54]
3.7
2.0
0.6
3.3
40.6 1.5
ARM [87]
4.5
2..
1.,9
1.5
37.4
VREx [42]
48.0
1.5
1158
1.2
RSC [36]
2.4
3,8
49,6
1.2
39.4 1.3
Meta-DMoE [89]
2.0
29.2
47.8
3.88
40.0 1.2
SelfReg [39]
50.6 4..
1.6
4.3
1.5
41.3
MixStyle [91]
2.5
52.1
0.2
40.9 1.1
Fish [69]
4..
1.1
5.27
1.0
40.2 0.6
[60]
1.9
331
5.7
0.8
42.0 1.0
CAD [65]
2.6
3.91
1.6
3.3
40.5
CondCAD [65]
2.9
2.5
1.3
3.5
39.7 19.7 0.4
Fishr [62]
3.3
3.9
4.2
3.3
4..6
ITTA [15]
5..7
0.6
4.69
3.66
4,5
ERM+
2.6
3.3
5..2
3.3
0.7
Ours
5.44
2.4
5..9
0.7
46.3
Table 15: Average accuracies on the DomainNet [59] datasets using the default hyper-parameter settings in DomainBed [27].
clip
info
paint
quick
real
sketch
Average
ERM [75]
50.4 0.2
14.0 0.2
40.3 +
11.7 + 0.2
52.0
43.2 +
35.3
IRM [1]
43.2 0.9
0.3
1.4
9.4
4.0
38.4
30.4
GroupGRO [66]
38.2 0.3
0.3
0.3
8.1
4.5
33.7
27.5
Mixup [80]
13.6
0.5
1.4
4.2
34.0 4.2
MLDG [44]
0.3
0.3
40.7
1.17
0.3
4.2
0.0
CORAL [70]
5.2
0.2
4..0
12.7
0.3
43.4
3.2
MMD [46]
16.6 13.3
0.3 0.0
1
0.0
17.1
0.4
7.9
DANN [24]
45.0 0.2
0.2
3.2
0.34
4.37
3.15 0.3
CDANN [49]
0.2
0.2
3..6
0.4
4.1
0.
31.8
MTL [5]
50.6 0.2
0.4
0.3
0.3
5.11
4.05
+..0
SagNet [54]
1.1
0.1
4.2
1.21
0.3
4.14
3.1
ARM [87]
4.2
11.7
3..6+0.1
9.8
0.3
37.0+03
2.1
VREx [42]
3.62
0.49
3.3
0.22
4.8
3..8
28.1
RSC [36]
3.7
1.,4
30.5
1.2
4.
34.7
27.9
Meta-DMoE [89]
0.9
0.5
4.60
0.49
0.2
36.0 4..0
SelfReg [39]
0.3
0.3
41.2+0.2
11.7+0.3
4.0
43.8
3..7
MixStyle [91]
4.4
1.04
39.3
1.44004
43.7
42.7
3.1
Fish [69]
51.5
0.2
4.34
1.57
5.2
4.11
0.0
SD [60]
51.3
0.1
4.3
12.6+02
0.2
4..0
3.2
CAD [65]
4.0
12.1
1.19
1..2
4.
38.5
3.8
CondCAD [65]
1.0
1.4
36.1
102
4.
38.7
31.9
Fishr [62]
0.7
1..
40.0
11.9
4.72
41.7
0.3
ITTA [15]
0.7
1..9
0.5
1..9
0..
43.5
3..9
ERM+
0.1
0.4
42.3+0.1
1..0
5..4
4.13+
3.14
Ours
50.7
0.07
0.5
0224
51.4
44.8+ 0.2
3.11
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, which matches our theoretical analyses and experimental findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations have been discussed in Sec. A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We provide our theoretical statements and proof in Sec. 4.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide pseudocode in Sec. B, and our code in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Code is provided at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https:/ httppp:://iissp// cc/
publ / guides
des CodeSuumm i ssi assioPolli (yyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss. cc/ cttppcccppccccccccc pubppcpliooooo /wwpudgusssseee/eeeeeeee Code Submi ssissiionnn ceiionnoliinn Piyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Implementation details are described in our experimental sections. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars for our experiments and detailed how they are computed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For all experiments, running time detailed Tab. 11.
Tesla V100 GPU with 32 GB memory, the actual
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /ttp:///////ii....//// ca cf/p
Answer: [Yes]
Justification:
We confirm our work does not violate NeurIPS Code of Ethics, in any respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work aims improve the generalizability of a deep model. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not involve such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Benchmarks used in this data are properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers' raww aw. thcode com/ data setss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "pTvdsPmJi2", "reviewer": "Reviewer_oULp", "summary": "The computer-vision paper introduces a strategy of learning from multiple experts (LFME), which performs knowledge distillation from models specially trained on data from different domains. In particular, the experts are trained jointly with the target model, and a specific form of logit regularization is chosen for knowledge distillation due to empirical verification. The idea is relatively straightforward to understand with strong empirical results on image classification (by Domainbed) and semantic segmentation.", "questions": "1. In Algorithm 1, the target model and the experts are trained simultaneously, which is a more parallel and iterative scheme rather than a sequential one with two stages, i.e., training the experts first, then distilling the knowledge to the target model using logit regularization. Why is such a design choice made? Are there better justifications than just “for simplicity”? Please explain with more details. \n2. Each expert is generally weak as they are only trained using the data from their own domain. Is it necessary and how to ensure the quality of the experts?\n3. In Appendix A, Limitations, the authors discussed that LFME cannot be applied to single-domain tasks. How does the number of domains affect the proposed approach? Will the algorithm benefit from having more domains with less data or more data in fewer domains?", "limitations": "The limitations has been discussed in Appendix A.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The experimental results come in strong when compared to many other baselines. Moreover, the study has extended results for semantic segmentation. \n2. The visualizations are helpful to illustrate the ideas and evidence. \n3. The empirical study on naive aggregation of domain experts shows that naive ensemble does not necessarily lead to better generalization. Then ablation study shows the chosen form of logit regularization outperforms other knowledge distillation alternatives.", "weaknesses": "1. The novelty is relatively limited. The proposed approach is very similar to Meta-DMoE with a few changes such as logit regularization and alternating updates, but these changes are not well justified (see below). The idea of utilizing multiple domains have also been explored by [1]. \n2. There is a lack of coherent argument to understand the proposed approach. In Section 4.1, the paper briefly discussed how the additional logit regularization enables learning more information. In addition, in Section 6.2, the proposed approach LFME is compared with Label Smoothing. However, the discussion is not very convincing. From the perspective of using information from other classes, I do not see the fundamental difference between logit regularization and Label Smoothing (LS). Both logit regularization and LS have hyperparameters to be tuned, so it might not be appropriate to claim its advantage as \"not involve hand-crafted settings” (line 187) and criticize LS has “potential improper heuristic designs” (189). \n3. Moreover, in Section 4.2, the paper justifies the advantage of the proposed approach LFME by “mining hard samples from the experts”. It is a general statement that is true for low-confidence predictions from any model, so it does not constitute a strong argument. Intuitively, the in-domain samples are easier than out-of-domain samples for each expert, and the hard samples aggregated from all experts are not specifically representative of any subpopulations. The argument will be more comprehensive if the paper can show what are these hard samples and why it matters to mine the hard samples from the domain experts (and not an ensemble of experts or some random experts). \n4. In Section 6.5, the result for in-domain evaluation is only provided for one dataset. This is not sufficient to justify a strong claim that the target model is an expert on all source domains. \n5. Code is submitted but it is unclear how the proposed approach can be combined with SWAD. Does it apply to just the target model or also the domain experts? The hyperparameter configurations to obtain the SoTA results are also unclear for reproducibility. \n\n[1] Yao, Huaxiu, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. \"Leveraging domain relations for domain generalization.\" *arXiv preprint arXiv:2302.02609* (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1. Novelty\n\nA1: Differences between Meta-DMoE and LFME are as follow.\n\n1. ### Domain experts in these two works serve different purposes\n\nMeta-DMoE aims to adapt the trained target model to a new domain in test. To facilitate adaptation, their target model should be capable of identifying domain-specific information (DSI). Therefore, the target model is enforced to extract DSI similar to those from domain experts. Notably, their trained experts are expected to thrive in all domains, which are used to extract DSI **not** in their trained domains but rather in an unseen one. In a word, domain experts in Meta-DMoE serve as supervision to ensure that the target model can adapt to a new domain. Differently, LFME expects its target model to be expert in all source domains. In our framework, domain experts provide professional guidance for the target model only in their corresponding domains.\n\n2. ### Different implementations\n\nMeta-DMoE involves meta-learning and test-time training (TTT), where domain experts are used for data from their unseen domains in both training and test. The overall process can be time-consuming due to TTT and the second-order optimization in meta-learning, which may pose efficiency problems compared to the straightforward training in LFME. Meanwhile, their idea is developed on the traditional KD idea that enforces feature similarity between teachers and student, which is shown suboptimal in our analysis.\n\n3. ### Different effectiveness and efficiencies\n\nWhen evaluated with the same strict setting in DomainBed, LFME leads Meta-DMoE in all datasets with an average improvement of 1.8pp. The required running time for LFME is also less than Meta-DMoE: 38' vs. 73' for one trial in PACS (even without the experts training time, Meta-DMoE still requires 45' for one trail).\n\nWe list comparisons with Meta-DMoE in Sec. 2, and are glad that Reviewer xxxE find it comprehensive. Regarding the idea in [1], it shares similar design with DAELDG that uses different classifiers as experts and averages the final outputs of all experts as the result. Similar intuition has been discussed in our analysis. We will discuss this work in the revised version.\n\n> W2: More justifications for argument in Sec. 4.1 and comparisons against LS\n\nA2: We provide more justifications (visual examples included) for further comprehending the analysis in Sec. D.1. More empirical evidence supporting this argument is provided in D.2. Please refer to them for a better understanding.\n\nLS uses a heuristic $\\epsilon$ to adjust output probability and prevent overconfidence in the gt label: $(1-\\epsilon)H(q, y) + \\epsilon H(q, U)$ (given U a uniform distribution). if $\\epsilon$ is improperly set (e.g. close to 1) and the dominant label approximates $U$, LS may fail. Differently, LFME uses labels from both gt and experts' outputs to excel in all source domains: $H(q, y) + \\frac{\\alpha}{2}||z-q^E||^2$, where we do not need to deliberately calibrate the probability. Those two methods differ inherently. Resulsts in Sec. 6.2 also show LFME performs better than latest LS methods. Notably, experiments in Tab. 4 in the general response show that the model is insensitive w.r.t $\\alpha$ as it is on par with ERM even with $\\frac{\\alpha}{2}=1000$, because $q_{\\ast}^E$ is mostly aligned with $y$ (see Fig. 2 (a)), which is a much preciser label than $U$. We thus say that LS bears a con of potential improper heuristic designs against LFME. Meanwhile, LS treats all samples equally, while LFME can specifically focus more on hard samples. That is another pro for LFME. \n\n> W3: What are those hard samples and why use domain experts\n\nA3: The hard samples are domain ambigous data, aligned with your intuition. We find that domain experts can locate more of these data than random models, thus can better help DG. Please see our general response for details.\n\n> W4: More in-domain evaluations\n\nA4: We conduct experiments on other two datasets (i.e. TerraInc and VLCS) and list the results in Tab. 1 in the geneal response. Results are consistent with that in Tab. 5, where the target model can lead the baseline and experts in all cases, validating that it becomes expert on all source domains. \n\n> W5: SWAD and hyper-parameter (HP) settings\n\nA5: We apply SWAD only for the target model. We use the default settings in original implementations of Domainbed and SWAD. For your reference, we list all HP in Tab. 2 in the general response.\n\n> Q1: Why use parallel training\n\nA1: In the parallel scheme, each training sample will go through 2 forward passes (one for expert and another for the target), and we can use a single optimization to update all parameters and a total of 5000 updating steps for a trial in PACS (see Alg. 1). Instead, if using the sequential scheme, each sample will go through 3 forward passes (one for training expert, and two for training the target), and we will require 2x5000 updating steps for a trial. Training times for these two schemes are 38' (parallel) vs. 55' (sequential), we thus use parallel scheme for simiplicity. \n\n> Q2: Is it necessary to ensure the experts' quality\n\nA2: Different from Meta-DMoE, experts in LFME are expected to saturate only on their corresponding domains. Although less data is used, they can still outperform the baseline within their trained domains (see our in-domain experiments). Given that our interest lies primarily in the outputs of these experts within their designated training data, their general proficiency beyond these specific data is of secondary concern. Thus, it may be unnecessary to ensure the quality of the experts across broader domains.\n\n> Q3: How does the number of domains affect the proposed approach, will the method benefit from more domains or more data\n\nA3: We answer this question by merging the source domains to create varying domain numbers. As shown in Tab. 3 in the general response, the model performance is generally positively correlated w.r.t domain numbers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FgDIX2fDm8", "reviewer": "Reviewer_xxxE", "summary": "This paper addresses the problem of domain generalization from the perspective of  ''learning multiple experts''. In particular, they propose to train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. The proposed logit regularization provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Experiments on standard DG benchmarks demonstrate the effectiveness of the proposed framework.", "questions": "- This method needs to train additional expert per domain, what if there are large amount of domains for the training data, any potential solution for this occasion?\n\n- Since multiple expert models are required when training the target model, how would this method be applied to the training process of large foundation models?\n\n- The free lunch idea appears to be applicable beyond just the DG setting. How does it perform on ImageNet? Including such experiments could enhance the significance of this concept.", "limitations": "yes", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The overall manuscript is well-organized and easy to follow.\n\n- The method is simple and effective. Its impressive results in different tasks demonstrate the strong applicability of this idea to potential other tasks.\n\n- The related work section is comprehensive, especially regarding comparisons with Meta-DMoE that also tries to distill knowledge from domain experts.\n\n- The idea of using MSE loss between logit and probability for distillation is new and well-explained, supported by both theoretical and empirical evidence.\n\n- The free lunch idea introduced in sec. 6.1 is interesting and seems to have the potential for more general applications.", "weaknesses": "While adopting the idea of knowledge distillation (KD) and the concept of domain experts for improving DG is not new in the literature, it is appreciated that this method can be implemented in such simple form and be explained with extensive insights. Nevertheless, there are still a certain amount of claims in the paper that requires proper explanations or more justification:\n\n- Performance reported in the original Meta-DMoE paper is higher than that in Table 1. For example, Meta-DMoE reports an average of 86.9 in PACS, surpassing the best result listed in the table. The authors should provide an explanation for this discrepancy. Moreover, I noticed that the training time comparisons between these two methods are provided, it would also be beneficial to highlight the test and training resource differences between these two methods in the related work section.\n\n- The authors mention that their method introduces only one hyper-parameter, which is within a certain range. It would be beneficial to include an ablation study to explain why this specific range was chosen.\n\n- It would be beneficial to evaluate this method with more real-world DG problem, such as using dataset from wilds benchmark.\n\n- The method [1] with similar settings should be compared in Table 1. Additionally, some methods, such as MIRO, report improved performance when combined with SWAD. Including these results in Table 1 would provide a more comprehensive comparison.\n\n- Minor issues. Formats of ”i.e.” in line 145, ”Tab.” in line 578, and ”Fig.” in line 581 are inconsistent with their usages in other places. The authors should standardize the format of similar abbreviations to maintain uniformity.\n\nReference: \n\n[1] Domain Generalization via Rationale Invariance, in ICCV 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: Performance in the Meta-DMoE paper is higher than that in Tab. 1, and highlight the resource usage differences.\n\nA1: Note that we mainly use the ResNet18 model for evaluating Meta-DMoE in Tab. 1, and the experiments are conducted for a total of 3x20 trials using the default hyper-parameter settings that are within large ranges. These settings are different from those in the original paper (where ResNet50 results are reported), which can cause the differences in the reported numbers of Meta-DMoE. We will include this clarification in our revised paper.\n\nWe thank the reviewer for the suggestion. We list the efficiency comparisons in our response regarding w1 for Reviewer oULp, and we will include the differences in the related work section.\n\n> W2: Conducting ablation study to analyse the selected value range for the weight parameter.\n\nA2: We conduct this ablation study in Tab. 4 in our general response, which shows that our method can obtain relatively better performance when the weight parameter is selected wihtin [0.1, 10].\n\n> W3: Experiments in datasets from the Wilds benchmark.\n\nA3: We conduct experiments using three real-world datasets (i.e., iWildCam, RxRx1, and FMoW) using the default settings in Wilds. Results listed in Tab. 5 in the general response show that our method can improve the baseline in all three datasets and obtains favorable performance against existing arts.\n\n> W4: Including RIDG [1] and MIRO+SWAD in Tab. 1.\n\nA4: We include comparisons with the referred methods in Tab. 6 in our general response. The comparisons will be included in Tab. 1 in our revised paper.\n\n> W5: Minor issues.\n\nA5: We thank the reviewer for the notifications, these typos will be corrected in our revised paper.\n\n> Q1: Solution for datasets with a large number of domains.\n\nA1: When encountering datasets containing numerous domains, we can utilize the existing strategy [a] that clusters the training domains into fewer super-domains, thus reducing the expert number. We leave this exploration to potential future works.\n\n[a] Orbit: A real-world few-shot dataset for teachable object recognition, in ICCV'21\n\n> Q2: How to apply the method to the training process of large foundation models?\n\nA2: When applied to the training process of large foundation models, a possible solution for easing the memory constraint problem is to use a sequential training strategy that first learns different experts, and then trains the target model without optimizing these experts. In this way, professional guidance from different experts can still be obtained, and we can avoid the memory constraint for training $M+1$ models at the same time (with $M$ the expert number).\n\n> Q3: Including experiments of the free lunch idea in Imagenet setting.\n\nA3: We conduct experiments with the free lunch idea in Imagenet as suggested. Compared to the baseline, the free lunch can obtain better performance by 0.4pp for the top1 acc (76.9 vs. 76.5) and 0.5pp for the top5 acc (93.7 vs. 93.2). These results show that the free lunch idea is beneficial for large datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ccdCyzrIvU", "reviewer": "Reviewer_uZyo", "summary": "This paper focuses on improving domain generalization by utilizing multiple experts. Particularly, a simple yet effective framework is proposed whereby a target (student) model is learned from multiple expert (teacher) models through logit regularization. After learning, the target model can grasp knowledge from multiple source domains and shows advantages in handling hard samples. The proposed approach demonstrates consistent improvements across different evaluation tasks, outperforming existing SOTAs.", "questions": "1: What is the difference between the proposed approach and foundation models? Why not directly learn a model using all the source domains? How does the performance of the proposed approach compare to Segment Everything?\n\n2: What’s the difference between the proposed approach and the baseline ERM? In lines 77-78, the authors mention ‘only one hyper-parameter’ without further illustration. The difference is clarified only until line 138. Statements near lines 77-78 need to be improved to avoid confusion. More importantly, it can be misleading to say “only one additional parameter upon ERM” since a target model is also involved.\n\n3: How does the value q_*^E relate to the hardness of samples? What causes the sample to be ‘hard’? Low data frequence or image quality?  Without these clarified, the provided theoretical insights are not convincing.", "limitations": "No discussions on limitations and potential negative societal impact of the work are provided. \n\nOne limitation is that the performance of the proposed approach can be upper bounded by the performance of the expert models. This limitation again raises the question of why not directly learn from all source domains?", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper overall is well written. It provides in-depth theoretical insights on its proposed logit regularization and surrogate ablation studies.", "weaknesses": "1: The contribution of the proposed approach is marginal. The performance of the proposed approach does not show much improvement compared to ERM or expert models (Table 5). \n\n2: The baseline models for comparison are not clearly introduced, which creates difficulties in understanding the Tables.\n\n3: The comparison to the approaches that directly learn a foundation model using data from all source domains is not discussed. Hence, the benefits of the proposed approach, which have to use a set of expert models plus a target model, compared to the foundation model are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: The performance does not show much improvements in Tab. 5\n\nA1: We want to clarify that the in-domain comparisons in Tab. 5 are only used to verify our claim that the target model has evolved to be an expert in all source domains. In the DG setting, out-of-domain performance is often regarded as the metric for evaluating different methods, and our method is shown to outperform many sophisticated designs. When compared to ERM and the different expert models, our method also shows large improvements (Tab. 4). We note that the out-of-domain performance of our method is regarded as impressive by other reviewers.\n\n\n> W2: The compared baseline models are not clearly introduced.\n\nA2: The primary baseline models (in both Tab. 1 and 2) are ERM that directly trains the model with data from all source domains. According to the existing benchmark [20], in the DG task, ERM is a strong baseline that can outperform many SOTAs when evaluated with strict evaluation protocols. As for other compared models, we use most of the implementations in the commonly-used DomainBed [20] for evaluation. It would be better to refer to the benchmark for details.\n\n\n> W3: Comparisons with foundation model that uses data from all source domains is not discussed.\n\nA3: We want to clarify that the referred foundation model is ERM, and we include comparisons with it in all our experiments (i.e. Baseline in Tab. 2 and ERM in all other tables), where our design can improve the performance in almost all evaluated datasets.\n\n> Q1 (part 1): Difference between the proposed approach and foundation models, why not directly learn a model using all the source domains?\n\nA1 (part1): Compared to the foundation model, the target model is trained with an additional logit regularization term $\\mathcal{L}_{guid}$ (without it, the target model degrades to the foundation model), which is designed to help the target model to be expert in all source domains. As explained in Sec. 4, we reveal that the logit regularization term can help the target model to use more information and mine hard samples from the experts, which are both beneficial for improving generalization.\n\n> Q1 (part 2): Why not compare with the Segment Anything Model (SAM).\n\nA1 (part2): Our experiment settings in the generalizable semantic segmentation task are the same as previous mainstreams [32, 48, 12], which train the target model on synthetic data and test it in unseen real-world scenes. We do not compare with SAM mainly because (1) SAM and mainstreams use different network structures: SAM uses ViT and CLIP text encoder in their model, which has a total of 636M parameters. Differently, the mainstreams use DeepLabv3+ ResNet50 as the backbone, consisting of 39M parameters, which is only 6 percent the size of SAM; (2) SAM is trained over 1 billion data, while the synthetic data used for training in our setting is only 34,366. We thus do not compare with SAM. \n\nOur work focus the generalization task, which can be applied to many downstream vision tasks, semantic segmentation included. Unlike SAM that aims to solve segmentation with large models and more data, this work can be applied to situations when there is only limited training data from certain domains, and the task is to generalize the model to new domains, for example in particular medical situations. We leave such exploration to potential future works.\n\n> Q2 (part 1): What’s the difference between the proposed approach and the baseline ERM.\n\nA2 (part 1): ERM is the referred foundation model that is trained with all data (by using only $\\mathcal{L}_{cla}$ in Eq. (3)). Please refer to our response for Q1 (part 1) for their differences.\n\n> Q2 (part 2): Improper to say “only one additional parameter upon ERM” since a target model is involved.\n\nA2 (part2): We appreciate the suggestion. This sentence is to underline that except for the only one weight parameter, our model does not involve more heuristic hyper-parameter settings compared to ERM. We will clarify it in our revised version to avoid confusion.\n\n> Q3: How does $q_{\\ast}^E$ relates to hard samples and what are those hard samples?\n\nA3: As the classification loss is computed as $-\\sum y \\log q$, smaller $q$ indicates larger loss, which corresponds to a more difficult sample. This is is consistent with the previous work [27]. In Sec. D.4 in the appendix, we find that hard samples from the domain experts contain more domain ambigous data than that from the model fed with all data, explaning why domain experts must be involved. Please also refer to our general response for details.  \n\n> L1: No discussions on limitations and potential negative societal impact of the work are provided.\n\nA1: Please note that we discuss the limitation of this work in Sec. A in the appendix. For the potential negative societal impact, this work designs a new idea for the task of domain generalization that holds significant potential for a wide array of real-world applications. It may bear potential societal consequences similar to other machine learning systems. We will include this in our revised paper.\n\n> L2: The performance of the proposed approach can be upper bounded by the performance of the expert models, why not directly learn from all source domains?\n\nA2: Our method outperforms domain experts in both out-of-domain (i.e., Tab. 4) and in-domain (i.e., Tab. 5) settings, indicating that the performance is not bounded by domain experts. This is because the target model uses outputs from all experts and gt as training labels (see Eq. (3)), thus the experts can not bound the performance of the target model. When compared to the referred foundation model that directly trains with all data, our method also shows large improvement (i.e., Tab. 1 and 2). Please refer to our response for Q1 (part 1) for the explanations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OzEgpvRL4G", "reviewer": "Reviewer_h8LM", "summary": "This paper proposes a simple framework called LFME for learning from multiple experts in domain generalization. LFME introduces a logarithmic regularization term to enforce similarity between the target model and expert models, allowing the target model to acquire expertise from all source domains and perform well in any testing domain. Experimental results demonstrate that LFME outperforms existing techniques not only in classification and segmentation tasks but also reveals through in-depth analysis its ability to implicitly use more information for predictions and to discover difficult samples from experts, thereby enhancing generalization capability.", "questions": "In Section 4.2, the authors claim that their method improves DG by explicitly mining hard samples from the experts. However, Tables 3 and 6 show that specifically mining hard samples from the experts improves performance of ERM only in the TerraInc dataset, but not in the PACS dataset (with an average improvement of only 0.1 pp). Could the authors provide an explanation for this phenomenon?", "limitations": "Authors have adequately addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a novel idea on the basis of a new knowledge distillation paradigm for DG. The motivation of using multiple experts for improving generalization is clear and reasonable. Meanwhile, I also found the paper well-presented and easy to understand.\n\nAfter checking the pseudocode and the provided code, I found the implementation extremely simple. Given the impressive results computed with rigorous evaluation protocols, I believe this paper can offer valuable contributions to the literature by providing an accessible and effective solution. \n\nThe provided deep analyses that explains why their method works are appreciated and, in my opinion, crucial for a paper, especially in the era of deep learning. Coupled with the empirical evidence in their supplementary material, I think most of claims made in their paper can be well supported.", "weaknesses": "Regarding the proposed method, although large improvements over the baseline ERM are observed, their method involves higher training costs. I noticed the authors provide the training time comparisons with different methods in their supplementary material, which shows that their LFME requires 1.5 times more training time than ERM, and it is also one of the most time-consuming methods compared. While I agree with the authors that requiring more training time is inevitable for methods designed based on KD, the authors should at least list this as a major drawback in their limitation section.\n\nAs in recent times, Vision Transformers (ViTs) have demonstrated substantial improvements in the classification and semantic segmentation tasks. Although the authors conducted experiments using different ResNet models, they should also consider adopting a ViT backbone for their experiments. This could provide valuable insights into the performance of their method when applied to more advanced architectures, potentially highlighting further benefits or limitations. Including such experiments would enhance the comprehensiveness of the study and align it with current trends in the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: List more computational cost as a limitation.\n\nA1: We thank the reviewer for the suggestion, we will include this limitation in our revised paper.\n\n> W2: Experiments with ViTs.\n\nA2: We conducted further experiments by evaluating our method, the baseline ERM, and some leading methods in Tab. 1 with a ViT-Base model using the same setting detailed in the manuscript. Results in Tab. 7 in the general response show that our method can still obtain favorable performance against existing arts despite using different networks.\n\n> Q1: Why hard sample mining can improve performance in TerraInc dataset but not the PACS dataset.\n\nA1: We infer this is mainly because compared to the PACS dataset, TerraInc is more difficult for the corresponding expert, which can provide more hard samples to boost generalization (briefly mentioned in Line 623 in our paper). As shown in Tab. 5 in the manuscript, the experts can achieve an average accuracy of 96.7% when tested with the source data in the PACS dataset, indicating that there are very few samples in PACS that can be regarded as hard for the experts. On the other hand, the in-domain results for the experts in TerraInc are 91.1% (Tab. 1 in the general response), suggesting that the dataset can provide more hard samples to guide the target model than PACS, thus improving the performance. We will include this analysis in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their hard work and insightful suggestions. We are inspired that Reviewer xxxE and h8LM find our work simple and Reviewer xxxE, h8LM, and oULp think the performance is strong. We are also glad that our in-depth theoretical insights are appreciated by Reviewer xxxE, h8LM, and uZyo. 

For the general question raised by Reviewer oULp and uZyo, we answer them below:

>Q: How does $q_{\ast}^E$ relates to hard samples, what are those hard samples, and why mine hard samples from the domain experts?

A: A smaller $q_{\ast}^E$ is associated with larger cross-entropy loss for the corresponding expert, indicating that the sample is more difficult for the expert than others with smaller loss (i.e. larger $q_{\ast}^E$). This is consistent with the previous work that specifically aims to mine hard samples [27].

We conduct analyses in D.4 to study what are those hard samples and the differences between hard samples from domain experts and other models. Specifically, compared to hard samples from the model that trains with all data, we find that hard samples from domain experts contain more ambiguous data that locates in the mixed region or the boundary of different domains (shown in Fig. 6), indicating that they may encode more out-of-domain information. This is consistent with the assumption from Reviewer oULp.

By conducting experiments (results are listed in Tab. 6), we find that hard samples from domain experts is more beneficial for DG than hard samples from the model fed with all data. This is because emphasizing these domain ambiguous data can assist the target model in exploring more domain-invariant features that is consistent across different domains, which is crucial for improving model robustness [17]. This experiment justifies our design by involving domain experts (instead of some random models) in our framework. Please see D.4 for more details.

We conduct the following experiments and list the results in the uploaded file:

1. More in-domain evaluations from different models (Tab. 1);
2. Hyper-parameters used for reproducing results in the DomainBed benchmark (Tab. 2);
3. Effect of the number of domains on model performance (Tab. 3);
4. Sensitivity analysis regarding the weight parameter $\alpha$ (Tab. 4);
5. Evaluation results in datasets from the Wilds benchmark (Tab. 5);
6. Comparisons with more methods in DomainBed (Tab. 6);
7. Evaluating our method with vision transformer (Tab. 7).

All experiments in the rebuttal are conducted with the same experimental settings introduced in our paper where the DomainBed benchmark is utilized. These evaluations will be included in our revised manuscript.

We answer questions raised by each reviewer in the following.

Meta-review: The paper proposes a framework that leverages multiple expert models to enhance domain generalization through logit regularization. While reviewers commended the clarity of the writing and the empirical results showing improvements over baselines, several concerns were noted: the approach's novelty is limited, there is a lack of sufficient justification for key methodological choices, and the evaluation does not cover a diverse range of tasks and datasets. the authors' rebuttal provided additional experiments and clarifications and addressed most of the concerns. Therefore, the recommendation is to accept the paper.
Decision: Accept (poster)