Title: LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Domain generalization

Knowledge distillation
Abstract: Domain generalization (DG) methods aim to maintain good performance in an unseen target domain by using training data from multiple source domains. While success on certain occasions are observed, enhancing the baseline across most scenarios remains challenging. This work introduces a simple yet effective framework, dubbed learning from multiple experts (LFME), that aims to make the target model an expert in all source domains to improve DG. Specifically, besides learning the target model used in inference, LFME will also train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. Delving deep into the framework, we reveal that the introduced logit regularization term implicitly provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Extensive experiments on benchmarks from different DG tasks demonstrate that LFME is consistently beneficial to the baseline and can achieve comparable performance to existing arts.  Code is available at https://github.com/liangchen527/LFME.

LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization

Abstract

Domain generalization (DG) methods aim to maintain good performance in an unseen target domain by using training data from multiple source domains. While success on certain occasions are observed, enhancing the baseline across most scenarios remains challenging. This work introduces a simple yet effective framework, dubbed learning from multiple experts (LFME), that aims to make the target model an expert in all source domains to improve DG. Specifically, besides learning the target model used in inference, LFME will also train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. Delving deep into the framework, we reveal that the introduced logit regularization term implicitly provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Extensive experiments on benchmarks from different DG tasks demonstrate that LFME is consistently beneficial to the baseline and can achieve comparable performance to existing arts. Code is available at https://github.com/liangchen527/LFME .

1 Introduction

Deep networks trained with sufficient labeled data are expected to perform well when the training and test domains with similar distributions [74, 4]. However, test domains in real-world often exhibit unexpected characteristics, leading to significant performance degradation for the trained model. Such a problem is referred to as distribution shift and is ubiquitous in common tasks such as image classification [43, 27] and semantic segmentation [19, 40]. Various domain generalization (DG) approaches have been proposed to address the distribution shift problem lately, such as invariant representation learning [53, 69, 57, 62, 29], augmentation [91, 48, 79, 47], adversarial learning [24, 46, 81, 49], meta-learning [44, 2, 22, 45], to name a few. Yet, according to [27], most arts perform inferior to the classical Empirical Risk Minimization (ERM) when applied with restricted hyperparameter search and evaluation protocol. Both the experiments in [27] and our findings suggest that existing models are incapable of consistently improving ERM in all evaluated datasets. The consistent improvement for ERM thus becomes our motivation to further explore DG.

Our approach derives from the observation in [10] that some of the data encountered at test time are similar to one or more source domains, and in which case, utilizing expert models specialized in the domains might aid the model in making a better prediction. The observation can be better interpreted with the example in Fig. 1. Given experts trained in the 'infograph", 'real", and 'quickdraw" domains, and test samples from the novel 'sketch" domain. Due to the large domain shift, it would be better to rely mostly on the expert trained on the similar 'quickdraw" domain than others.

quickdraw

Training

Test

infograph

real

1st

Expert

2nd

Expert

Target

Model

C

3rd

Expert

sketch

Target

Model

forward

direction

guidance

direction

logits  vector

probability  vector

C

contact  operation

However, the test domain information is often unavailable in DG, indicating that we can not specifically train an expert specialized in a particular domain. In light of this, obtaining a target model that is an expert on all source domains seems to be a practical alternative for handling potential arbitrary test domains. A naive implementation would be training multiple experts on each domain, and dynamically aggregate them to form the target model. So that any encountered test samples can be predicted by corresponding experts who are familiar with their characteristics. Nevertheless, such a practice has two inherent limitations: (1) designing an effective aggregation mechanism is essential and inevitable for the naive model. In fact, our experimental study indicates that using naive aggregating strategies, such as averaging, may deteriorate the performance. (2) the overall framework requires much more resources for deployment when there are many training domains, since all the experts must be leveraged during the test phase.

This work proposes a simple framework for learning from multiple experts (LFME), capable of obtaining an expert specialized in all source domains while avoiding the aforementioned limitations. Specifically, during the training stage, instead of heuristically aggregating different experts, we suggest training a universal target model that directly inherits knowledge from all these experts, which is achieved by a simple logit regularization term that enforces the logit of the target model to be similar to probability from the corresponding expert. With this approach, the target model is expected to leverage professional guidance from multiple experts, evolving into an expert across all source domains. During the test phase, only the target model is utilized. As a result, both model aggregations and extra memory and computation resources are not required during the deployment, since we only leverage one model. The overall training and test pipelines are illustrated in Fig. 1.

Our method can be interpreted through the lens of knowledge distillation (KD), where the core idea is transferring knowledge by training the student ( i.e . the target model) with soft labels from teachers ( i.e . experts) [31]. Unlike traditional KD [31] that uses teachers' output probabilities as soft labels in a cross entropy loss, we employ a logit regularization term that uses experts' probabilities to refine the logit of the target model in a regression manner, which can be regarded as extending the effectiveness of mean squared error (MSE) loss in classification [37] within the KD realm.

To gain a deeper understanding of the effectiveness of our logit regularization term, we perform in-depth analyses and uncover that its merit over the baseline can be explained in twofold. (1) It implicitly regularizes the probability of the target model within a smaller range, enabling it to use more information for prediction and improve DG accordingly. It is noteworthy that the effect is achieved inherently differs from that by label smoothing (LS) [71], as LFME does not require explicitly calibration for the output probability. Expanding upon this analysis, we find that a simple combination of cross entropy and MSE losses achieves comparable performance among existing arts. Given its straightforward implementation without unnecessary complexities, this expanding may offer a "free lunch" for DG; (2) It further boosts generalization by helping the target model to focus more on hard samples from the experts, supported by our theoretical finding. Through experiments on different datasets, we find that hard samples from the experts are more beneficial for generalization than those from the model itself. Given that hard sample mining is essential for easing distribution shift [35, 42], this discovery may offer valuable guidance for future research endeavors.

Through evaluations on the classification task with the DomainBed benchmark [27] and segmentation task with the synthetic [63, 64] to real [20, 83, 55] setting, we illustrate that LFME is consistently beneficial to the baseline and can obtain favorable performance against current arts (other KD ideas included). Our method favors extreme simplicity, adding only one hyper-parameter, that can be tuned in a rather large range, upon training the baseline ERM.

2 Related Works

General DG methods. Domain generalization (DG), designed to enable a learned model to maintain robust results in unknown distribution shift, is gaining increasing attention in the research community lately. The problem can be traced back to a decade ago [6], and various approaches and applications have been proposed to push the generalization boundary ever since [53, 26, 46, 32, 81, 49, 2, 16, 11, 13, 82, 12]. The pioneering work [4] theoretically proves that the DG performance is bounded by both the intra-domain accuracy and inter-domain differences. Most previous arts focus on reducing the inter-domain differences by learning domain-invariant features with ideas such as kernel methods [53, 26], feature disentanglement [10, 61], and gradient regularization techniques [69, 62]. Same endeavors also include different learning skills: adversarial training is leveraged [24, 81] to enforce representations to be domain agnostic; meta-learning is utilized [2, 45] to simulate distribution shifts during training. Other works aim to improve the intra-domain accuracy: some suggest explicitly mining hard samples or representations with handcraft designs, such as masking out dominant features [36], weighting more on hard samples [42], or both [35]. LFME falls into this category as the target model can also mine hard samples from the experts, beneficial for excelling in all source domains (in Sec. 6.5).

Utilizing experts for DG. Methods with the most relevant motivations with our LFME are perhaps those also involves experts [86, 28, 89, 90]. In DAELDG [90], a shared feature extractor is adopted, which is followed by different classifiers ( i.e . expert) that correspond to specific domains. Their experts are trained by enforcing the outputs to be similar to the average output from the non-expert classifiers. Different from our work, it uses the average outputs from different experts as the final result. In Meta-DMoE [89], similar to LFME, different experts are trained on their specific domains where a traditional KD idea is adopted: the feature from the target model is enforced to be similar to the transformer-processed version of their experts' output features. Notably, Meta-DMoE and LFME share very distinct objectives for the expert models. Specifically, Meta-DMoE aims to adapt the trained target model to a new domain in test. To facilitate adaptation, their target model is assumed to be capable of identifying domain-specific information (DSI), and is enforced to extract DSI similar to those from domain experts. In their settings, domain experts are expected to thrive in all domains and are used not in their trained domains but rather in an unseen one. Differently, LFME expects its target model to be expert in all source domains, where domain experts provide professional guidance for the target model only in their corresponding domains. Additionally, Meta-DMoE involves meta-learning and test-time training, which is more complicated than the end-to-end training adopted in LFME. Unlike their empirical design, our method is more self-contained, supported by in-depth analysis to explain its efficacy (in Sec. 4). We further show in our experiments (in Sec. 5.1 and E) that these related two methods perform inferior to our design, and the improvements from their basic designs: using average performance (in Sec. 6.4) or enforcing feature similarity between the target model and experts (in Sec. 6.3) are subtle compared to our logit regularization.

DG in semantic segmentation. Different from image classification, semantic segmentation involves classifying each pixel of the image, and the generalizing task often expects a model trained from the synthetic environments to perform well on real-world views. Directly extending general DG ideas to semantic segmentation is not easy. Current solutions mainly consist of domain randomization [34, 85], normalization [19, 56, 73], or using some task-related designs, such as the class memory bank in [40]. Different from some existing DG methods, LFME can be directly extended to ease the distribution shift problem in the semantic segmentation task without requiring any tweaks, and we show that it can obtain competitive performance against recent arts specially designed for the task.

3 Methodology

Problem Setting. In the vanilla DG setting, we are given 𝑀 source domains D 𝑠 = {D 1 , D 2 , ..., D 𝑀 } , where D 𝑖 is the 𝑖 -th source domain containing data-label pairs ( 𝑥 𝑖 𝑛 , 𝑦 𝑖 𝑛 ) sampled from different

probabilities on the joint space X×Y , the goal is to learn a model from D 𝑠 for making predictions on the data from the unseen target domain D 𝑀 + 1 . For either DG or the downstream semantic segmentation task, source and target domains are considered with an identical label space, and we assume that there are a total of 𝐾 classes.

Learning multiple experts . LFME trains all the experts and the target model simultaneously, and the training procedure is illustrated in the upper part of Fig. 1. As each expert corresponds to a specific domain, given the 𝑀 source domains, a total of 𝑀 experts have to be trained during this stage, and the training process of each expert is the same as that of the ERM model. Given a training batch B ∈ D 𝑠 , for the 𝑖 -th expert 𝐸 𝑖 , we only use data from the 𝑖 -th domain, and the computed loss regarding the data-label pair ( 𝑥 𝑖 , 𝑦 𝑖 ) ∈ (B ∩ D 𝑖 ) can be written as,

where 𝑞 𝐸 𝑖 ∈ R 𝐾 is the output probability computed by applying the softmax function over the output logits 𝑧 𝐸 𝑖 with 𝑧 𝐸 𝑖 = 𝐸 𝑖 ( 𝑥 𝑖 ) ; H denotes the cross-entropy loss: H( 𝑞, 𝑦 ) = ˝ 𝐾 𝑐 -𝑦 𝑐 log 𝑞 𝑐 .

Learning the target model . Data-label pairs ( 𝑥, 𝑦 ) ∈ B from all domains are used for training the target model 𝑇 . The main classification loss L 𝑐𝑙𝑎 is computed similar to Eq. (1): L 𝑐𝑙𝑎 = H( 𝑞, 𝑦 ) , s.t. 𝑞 = softmax ( 𝑧 ) = 𝑒𝑥𝑝 ( 𝑧 𝑐 ) ˝ 𝐾 𝑗 𝑒𝑥𝑝 ( 𝑧 𝑗 ) and 𝑧 = 𝑇 ( 𝑥 ) . Then, to incorporate professional guidance from the experts, we further introduce a logit regularization term L 𝑔𝑢𝑖𝑑 , to assist 𝑇 to become an expert on all source domains, which is computed by using the probabilities from the experts as a label for 𝑇 :

where 𝑞 𝐸 is the concatenate probabilities from different experts along the batch dimension 2 , ∥ · ∥ denotes the 𝐿 2 norm, and this term is only enforced on the target model. Note we use the normalized version of 𝑧 𝐸 ( i.e . 𝑞 𝐸 ) for computing L 𝑔𝑢𝑖𝑑 , which can be regarded as extending the effectiveness of MSE loss [37] in the KD realm. Our experimental studies (in Sec. 6.3) find it leads to better performance, and the following contents also elaborate on the motivation. Then, the overall loss L 𝑎𝑙𝑙 for updating the target model can be represented as,

where 𝛼 is the weight parameter, the only additional parameter upon ERM. We train the target model and the experts simultaneously for simplicity. Please refer to pseudocode in Algorithm 1 for details.

Rationality (comprehension from a KD perspective). Our logit regularization term can be viewed as a new KD form, wherein the fundamental principle is to utilize the teachers' ( i.e . experts) outputs as soft labels for the student ( i.e . target model) in a training objective [31]. In the context of classification tasks, the cross entropy loss H( 𝑞, 𝑦 ) is widely used in the literature. Based on this objective, an intuitive revision to achieve distillation is by replacing the ground-truth label 𝑦 with 𝑞 𝐸 in a cross entropy regularization manner ( i.e . H( 𝑞, 𝑞 𝐸 ) ), which builds the rationality for the pioneering KD art [31]. Nevertheless, a recent study [37] suggests that the MSE loss ∥ 𝑧 -𝑦 ∥ (without applying softmax function on 𝑧 ) performs as well as the cross entropy loss when being applied in the classification task. Correspondingly, a distillation scheme motivated by this objective can thus be utilizing the soft label 𝑞 𝐸 in a regression manner, which comes to our logit-regularized term: ∥ 𝑧 -𝑞 𝐸 ∥ . From this perspective, the rationality of the introduced logit regularization term aligns with the principle of KD, and it can be regarded as extending the applicability of MSE loss in classification to the KD realm. We compare our new KD form with other ideas in Sec 6.3, demonstrating its superior performance against existing KD ideas in the DG task. We delve deep into our method and explain the effectiveness of LFME in the following section.

Computational cost. Inherited from KD, training LFME inevitably requires more resources as both the teacher and student have to be involved during training. However, LFME uses the same test resources as the baseline ERM given only the target model is utilized. Meanwhile, please also note that the computational cost for LFME is not proportional w.r.t the domain size. Instead, the training cost will always be doubled compared to ERM, as each sample will require two forward passes: one for the target model and the other for the corresponding expert. Please refer to Tab. 11 for training time comparisons between different arts. In Sec. F, we show that simply increasing training resources for current arts cannot improve their performances, suggesting it may not be a primary factor in DG.

0

1000

2000

3000

4000

5000

Iteration steps

0.0

0.2

0.4

0.6

0.8

1.0

Average values of probability

q∗

K

∑

c ≠ ∗

qc

0

1000

2000

3000

4000

5000

Iteration steps

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Average values of probability

q∗

K

∑

c ≠ ∗

qc

0

1000

2000

3000

4000

5000

Iteration steps

-20

-10

0

10

20

Average values of logits

z∗

K

∑

c ≠ ∗

zc

0

1000

2000

3000

4000

5000

Iteration steps

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

Average values of logits

z∗

K

∑

c ≠ ∗

zc

K

∑

c

zc

0

1000

2000

3000

4000

5000

Iteration steps

-8

-6

-4

-2

0

Average rescaling factors



 `

(a)

𝑞

from ERM

(b)

𝑞

from LFME

(c)

𝑧

from ERM

(d)

𝑧

from LFME

(e)

F

and

F ′

4 Deeper Analysis: How the Simple Logit Regularization Term Benefits DG?

4.1 Enabling the Target Model to Use More Information

Specifically, for the baseline model, using only the classification loss L 𝑐𝑙𝑎 encourages the probability 𝑞 to be diverse: the ground truth 𝑞 ∗ to approximate 1 and 0 for others. Consequently, the corresponding logits 𝑧 ∗ will increase ceaselessly, i.e . 𝑧 ∗→+∞ , and vice versa for 𝑧 𝑐 , i.e . 𝑧 𝑐 →-∞ , ∀ 𝑐 ≠ ∗ (depicted in Fig. 2 (a) and (c)), as this is the solution for minimizing -log 𝑒𝑥𝑝 ( 𝑧 ∗ ) ˝ 𝑒𝑥𝑝 ( 𝑧 𝑐 ) . From another point of view, L 𝑐𝑙𝑎 encourages the model to explicitly focus on the dominant and exclusive features that are strongly discriminative but may be biased towards simplistic patterns [25].

Differently, when L 𝑔𝑢𝑖𝑑 is imposed, the logits 𝑧 will approximate the range of 𝑞 𝐸 ( i.e . [ 0 , 1 ] ). Eventually, the final logits will balance these two losses ( i.e . 𝑧 ∗ ∈ ( 𝑞 𝐸 ∗ , +∞) and 𝑧 𝑐 ∈ (-∞ , 𝑞 𝐸 𝑐 )∀ 𝑐 ≠ ∗ as shown in Fig. 2 (d)), resulting in a smoother distribution of 𝑞 , where 𝑞 𝑐 ( ∀ 𝑐 ≠ ∗ ) in LFME will be larger than it is in ERM (see Fig. 2 (b)). Since both two losses encourage the model to make a good prediction ( i.e . 𝑧 ∗ is expected to be the largest in both losses), the increase of 𝑞 𝑐 indicates that besides learning the dominant features, the target model will be enforced to learn other information that is shared with others. Compared with ERM that prevents the model from learning other features, LFME is more likely to make good predictions when certain types of features are missing while others exist in unseen domains. We provide a 'free lunch" inspired by the analysis in Sec. 6.1, and more justifications (including visual and empirical evidence) to support this analysis in Sec. D.1 and D.2. In Sec. C, we demonstrate that other KD ideas face challenges in achieving the same merit.

The above finding can also be endorsed by the information theory [68]. Specifically, with a smoother distribution of 𝑞 , the entropy, which measures information, will naturally increases [51], suggesting a theoretical basis for utilizing more information in LFME. According to the principle of maximum entropy [38], the improvement for generalization is thus foreseeable [88].

Note this effect is achieved inherently differs from that by label smoothing (LS) [71], as LFME does not involve hand-crafted settings to deliberately calibrate the output probability, which is essential in LS. In Sec. 6.2, we show LS is ineffective in DG compared to LFME. Besides the advantage of avoiding problems raised by potential improper heuristic designs, we show in the following that the logit regularization in LFME provides another merit over LS.

4.2 Enabling the Target Model to Mine Hard Samples from the Experts

This effect is realized by example re-weighting, motivated by proposition 2 in [72]. The single sample gradient of L 𝑎𝑙𝑙 with respect to the 𝑐 -th logit value 𝑧 𝑐 can be formulated as,

When the target probability corresponds to the ground-truth 𝑦 ∗ = 1 , Eq. (4) is reduced into,

In this situation, the rescaling factor F is given by,

On the other hand, for ∀ 𝑐 ≠ ∗ , summing the gradient values over the finite indexes gives,

and the rescaling factor F ′ in this situation is,

We observe that both F and F ′ are strictly monotonically increased regarding the value of 𝑞 𝐸 ∗ . Given almost all values of the rescaling factors are negative as can be observed in Fig. 2 (e) (except in the few initial steps where 𝑧 ∗ and 𝑞 𝐸 ∗ are both small and L 𝑔𝑢𝑖𝑑 barely contributes) 3 , with the same logits, a smaller 𝑞 𝐸 ∗ , in which case the expert is less confident, will lead to larger |F | and |F ′ | . This phenomenon indicates that with the logit regularization term, the target model will focus more on the harder samples from the experts. Empirical findings supporting this analysis are in Sec. D.3 and D.4. Note that the segmentation task also utilizes one-hot labels and cross-entropy loss, making it applicable to the two analyses presented.

5 Experiments

5.1 Generalization in Image classification

Datasets and Implementation details. We conduct experiments on 5 datasets in DomainBed [27], namely PACS [43] (9,991 images, 7 classes, 4 domains), VLCS [23] (10,729 images, 5 classes, 4

domains), OfficeHome [76] (15,588 images, 65 classes, 4 domains), TerraInc [3] (24,788 images, 10 classes, 4 domains), and DomainNet [59] (586,575 images, 345 classes, 6 domains). We use the ImageNet [21] pretrained ResNet [30] as the backbone for both the experts and the target model. Following the designs in DomainBed, the hyper-parameter 𝛼 2 in Eq. (3) is randomly selected in a range of [ 0 . 01 , 10 ] . To ensure fair comparisons, all methods are reevaluated using the default settings in DomainBed in the same device with each of them evaluated for 3 × 20 times in different domains. Training-domain validation is adopted as the evaluation protocol. Other settings (batch size, learning rate, dropout, etc.) are dynamically selected for each trial according to DomainBed.

Results with ResNet18. Results are listed in Tab. 1. We observe that ERM can obtain competitive results among the models compared, which leads more than half of the sophisticated designs, and only 6 methods lead ERM in most datasets (with scores ≥ 3 ). We also note that current strategies cannot guarantee improvements for ERM in all situations, given that none of them with a score of 5. Differently, our method can consistently improve ERM in all evaluated datasets and lead others in average accuracy. Specifically, our approach obtains the leading results in 4 out of the 5 datasets, and it also improves ERM by a large margin (nearly 8pp) in the difficult TerraInc dataset. Compared to methods that explicitly explore hard samples or representations ( i.e . VREx and RSC) and that use MoE ( i.e . Meta-DMoE), the performances of LFME are superior to them in all cases.

Results with ResNet50. Because larger networks require more training resources, we only reevaluate some of leading methods ( i.e . ERM, Fish, CORAL, and SD). in our device when experimenting with ResNet50. We note that our method surpasses the baseline ERM model in all datasets and leads it by 1.8 in average. Meanwhile, our method can still outperform the second best ( i.e . SD) by 0.7 in average. These results indicate that our method can consistently improve the baseline model, and it can perform favorably against existing arts when implemented with a larger ResNet50 backbone.

We also combine LFME with SWAD [8]. Same with the original design, the hyper-parameter searching space in this setting is smaller than the original DomainBed. We use the reported numbers in [8] for comparisons. As shown, our method can also improve the baseline and obtain competitive results when combined with SWAD. These results further validate the effectiveness of our method.

5.2 Generalization in Semantic Segmentation

Datasets and Implementation details. The training and test processes of the compared algorithms involve 5 different datasets: 2 synthetics for training, where each dataset is considered a specific domain, and 3 real datasets for evaluation. Synthetic: GTAV [63] has 24,966 images from 19 categories; Synthia [64] contains 9,400 images of 16 categories. Real: Cityscapes [20] 3,450 finely annotated and 20,000 coarsely-annotated images collected from 50 cities; Bdd100K [83] contains 8K diverse urban driving scene images; Mapillary [55] includes 25K street view images. Following the design [40], we use the pretrained DeepLabv3+ [18] and ResNet50 as the segmentation backbone for experiments. The maximum iteration step is set to 120K with a batch size of 4, and the evaluations are conducted after the last iteration step. The hyper-parameter 𝛼 2 in Eq. (3) is fixed as 1 in this experiment for simplicity. We use the mean Intersection over Union (mIoU) and mean accuracy (mAcc) averaged over all classes as criteria to measure the segmentation performance.

Experimental results. Results are shown in Tab. 2. To better justify the effectiveness of our method, we reevaluate the baseline model, which aggregates and trains on all source data, and PinMem in our device. We also implement SD [60], which is a leading method in the image classification

(a) Input

(b) Bsseline

(c) SD

(d) Pinmem

(e) Ours

主

task. Results from other methods are directly cited from [19]. We observe SD does not perform as effectively as it does in the classification task, which decreases the baseline model in most situations. In comparison, our LFME can boost the baseline in all datasets, similar to that in the classification task. It also performs favorably against existing methods specially designed for the semantic segmentation task, obtaining best results in 2 out of the 3 evaluated datasets in both mIOU and mAcc.

Visualized examples are provided in Fig. 3. We note that when it comes to unseen objects ( i.e . clouds with different shapes and a new car logo), or objects with an unfamiliar background, such as the person and car hidden in the shadow, due to the large distribution shift between real and synthetic data, compared methods make unsatisfactory predictions. In comparison, LFME can provide reasonable predictions in these objects, demonstrating its effectiveness against current arts regarding generalization to new scenes. These results validate the effectiveness of LFME and its strong applicability in the generalizable semantic segmentation task.

6 Analysis

Analyses in this section are conducted on the widely-used PACS dataset unless otherwise mentioned. Experimental settings are same as that detailed in Sec. 5.1. Please see the appendix for more analysis.

6.1 A Free Lunch for DG

As stated in Sec. 4.1, when the basis of discrimination is not compromised ( 𝑞 ∗ corresponds to a larger value in the label and vice versa for 𝑞 𝑐 for ∀ 𝑐 ≠ ∗ ), the increase of 𝑞 𝑐 can encourage the model to learn more information that is shared with other classes, and use them to improve generalization. Based on this analysis, it seems that using the one-hot label to regularize the logit is also reasonable. To validate this hypothesis, we replace 𝑞 𝐸 in Eq. (3) with the ground truth and reformulate L 𝑎𝑙𝑙 into,

Denoting as ERM+, results listed in the second row in Tab. 3 suggest that this idea can improve ERM in all unseen domains. Because the hard sample information is absent in this strategy, we observe it performs inferior to LFME. However, this alternative does not require experts or any other special designs, even the setting of the hyper-parameter 𝛼 cannot be wrong: Eq. (9) will approximate the baseline either with 𝛼 = 0 or 𝛼 approximates +∞ . Thus, we argue this simple modification can serve as a free lunch to improve DG. More evaluations of this idea are in our appendix.

6.2 Compare with Label Smoothing

As detailed in Sec. 4.1, LFME will explicitly constrain the probability of the target model within a smaller range. This effect may resemble the LS technique that aims to implicitly calibrate the output probability. However, compared to LS, LFME has two advantages. First , LFME does not involve heuristic designs of hyper-parameters for determining its probability, which is essentially required in existing LS ideas, such as 𝜖 in [71] and predefined margin in [50, 58], avoiding the possibility of deteriorating the performance when not choosing them properly; Second , LFME can explicitly mine hard samples from the experts, further ensuring improvements for DG. We evaluate 3 LS methods in both PACS and the difficult TerraInc datasets: (1) the pioneer LS method from [71]); (2) margin-based LS (MbLS) [50] that penalizes logits deviate from the maxima; (3) adaptive and conditional LS (ACLS) that can adaptively determine the degree of smoothing for different classes. Results are illustrated in 3rd-5th columns in Tab. 3. We observe that although these LS methods can improve the baseline, they are all inferior to LFME, validating the effectiveness of LFME against LS.

6.3 Compare with Other Knowledge Distillation Ideas

To test the effectiveness of our logit regularization ( i.e . ∥ 𝑧 -𝑞 𝐸 ∥ 2 ), we compare it with several other KD options, namely different combinations of the logits or probabilities from the target model and the experts, including ∥ 𝑧 -𝑧 𝐸 ∥ 2 (which is the basic design in Meta-DMoE [89]), ∥ 𝑞 -𝑧 𝐸 ∥ 2 , and ∥ 𝑞 -𝑞 𝐸 ∥ 2 . Moreover, we also compare it with a common practice in KD that uses the probability from the teacher ( i.e . experts) as a label for the student ( i.e . target model), which reformulates the Eq. (3) into: L 𝑎𝑙𝑙 = ( 1 -𝛼 2 )H( 𝑞, 𝑦 ) + 𝛼 2 H( 𝑞, 𝑞 𝐸 ) , and denoted as H( 𝑞, 𝑞 𝐸 ) in Tab. 4. Following the suggestion in [41], we gradually increase 𝛼 2 over the iteration steps to achieve better performance. Results listed in 2nd-5th columns in Tab. 4 show that not all KD strategies can improve DG and the logit regularization choice achieves the best result in terms of average accuracy. This can be explained by the analysis in Sec. 4.1 that our logit regularization can help using more information. Please also the explanation in Sec. C for details.

6.4 Compare with Naive Aggregation Ideas

To examine the effectiveness of LFME, we compare it with three different variants that employ all the experts during inference: (1) Avg_Expt, which averages the output from all experts for prediction, similar to DAELDG [90]; (2) Model soup (MS)_EXPT, which uniformly combines the weights of different experts. This is inspired by the MS idea in [78]; (3) Conf_Expt that utilizes the output from the most confident expert as the final prediction. Inspired by the finding in [77], the expert with the output of the smallest entropy value is regarded as the most confident one in a test sample; (4) Dyn_Expt, which is a learning-based approach that estimates

the domain label of each sample and dynamically assigns corresponding weights to the experts via a weighting module. Results are listed in 6th-9th columns in Tab. 4, where both the designs of handcraft ( i.e . Avg_Expt, MS_Expt, and Conf_Expt) and learning-based ( i.e . Dyn_Expt) aggregation skills fail to improve the baseline model. This is because the hand-craft designs in Avg_Expt, MS_Expt, and Conf_Expt are rather unrealistic in practice as different models may contribute differently in the test phase, we thus cannot use a simple average or select the most confident expert for predicting. Meanwhile, the learning-based Dyn_Expt will inevitably introduce another generalization problem regarding the weighting module, complicating the setting. In comparison, LFME avoids the nontrivial aggregation design and can improve ERM in all source domains, further validating its effectiveness.

6.5 Does the Target Model Become an Expert on All Source Domains?

The training process of LFME aggregates all professional knowledge from the experts into one target model, aiming to make the target model an expert on all source domains. To examine if the goal has been achieved, we conduct in-domain validations for the target model and compare the performances with that of the experts and the ERM model in the PACS and TerraInc datasets. Note in these experiments, ERM and LFME are trained using the same leave-one-out strategy, and the performances are averaged over the trials on three different target domains. Results are listed in Tab. 5. We observe that ERM performs inferior to the experts in the in-domain setting. The results are not surprising. As stated earlier, when encountering data similar to the source domain, it would be better to rely mostly on the corresponding expert than the model also contaminated with other patterns. In comparison, our method obtains the best results in all source domains because it implicitly focuses more on the hard samples from the experts, which is shown to be an effective way to improve the performance in many arts [33, 84]. These results validate that the proposed strategy can extract professional knowledge from different experts, and enable the target model to become an expert in all source domains.

6.6 Selection of the Hyper-Parameter

Compared with the baseline ERM, our method involves only one additional hyper-parameter ( i.e . 𝛼 2 in Eq. 3) which is randomly selected in the range of [ 0 . 01 , 10 ] in DomainBed. To evaluate the sensitiveness of LFME regarding this hyper-parameter, we conduct experiments in the PACS and TerraInc datasets by tuning it in a larger range. As seen in Tab. 6, LFME can obtain relatively better performance with either 𝛼 2 = 0 . 01 or 10, and it performs on par with ERM even when 𝛼 2 = 1000 . These results indicate that our method is insensitive w.r.t the hyper-parameter. This is mainly because a very large 𝛼 2 will enforce the model to only learn from domain experts, given that 𝑞 𝐸 ∗ is mostly aligned with the label 𝑦 (as seen in Fig. 2 (a)), the model will perform similarly to another form of the baseline [37] in this case.

7 Conclusion

This paper introduced a simple yet effective method for DG where the professional guidances of experts specialized in specific domains are leveraged. We achieve the guidance by a logit regularization term that enforces similarity between logits of the target model and probability of the corresponding expert. After training, the target model is expected to be an expert on all source domains, thus thriving in arbitrary test domains. Through deeper analysis, we reveal that the proposed strategy implicitly enables the target model to use more information for prediction and mine hard samples from the experts during training. By conducting experiments in related tasks, we show that our method is consistently beneficial to the baseline and performs favorably against existing arts.

Acknowledgement. This work was supported by the Centre for Augmented Reasoning, an initiative by the Department of Education, Australian Government.

References

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularization. In NeurIPS , 2018.

Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In ECCV , 2018.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79(1):151-175, 2010.

Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. arXiv preprint arXiv:1711.07910 , 2017.

Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classification tasks to a new unlabeled sample. In NeurIPS , 2011.

Fabio M Carlucci, Antonio D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In CVPR , 2019.

Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. NeurIPS , 2021.

Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutualinformation regularization with pre-trained models. In ECCV , 2022.

Prithvijit Chattopadhyay, Yogesh Balaji, and Judy Hoffman. Learning to balance specificity and invariance for in and out of domain generalization. In ECCV , 2020.

Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoqing Liu, and Yizhou Yu. Compound domain generalization via meta-knowledge encoding. In CVPR , 2022.

Chaoqi Chen, Luyao Tang, Yue Huang, Xiaoguang Han, and Yizhou Yu. Coda: generalizing to open and unseen domains with compaction and disambiguation. NeurIPS , 2023.

Chaoqi Chen, Luyao Tang, Feng Liu, Gangming Zhao, Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. In NeurIPS , 2022.

Liang Chen, Yong Zhang, Yibing Song, Anton van den Hengel, and Lingqiao Liu. Domain generalization via rationale invariance. In ICCV , 2023.

Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR , 2023.

Liang Chen, Yong Zhang, Yibing Song, Jue Wang, and Lingqiao Liu. Ost: Improving generalization of deepfake detection via one-shot test-time training. NeurIPS , 2022.

Liang Chen, Yong Zhang, Yibing Song, Zhen Zhang, and Lingqiao Liu. A causal inspired early-branching structure for domain generalization. IJCV , pages 1-21, 2024.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV , 2018.

Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In CVPR , 2021.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, UweFranke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , 2016.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

Qi Dou, Daniel Coelho de Castro, Konstantinos Kamnitsas, and Ben Glocker. Domain generalization via model-agnostic learning of semantic features. In NeurIPS , 2019.

Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV , 2013.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR , 17(1):2096-2030, 2016.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Scatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE TPAMI , 39(7):1414-1430, 2016.

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR , 2021.

Jiang Guo, Darsh J Shah, and Regina Barzilay. Multi-source domain adaptation with mixture of experts. arXiv preprint arXiv:1809.02256 , 2018.

Sivan Harary, Eli Schwartz, Assaf Arbelle, Peter Staar, Shady Abu-Hussein, Elad Amrani, Roei Herzig, Amit Alfassy, Raja Giryes, Hilde Kuehne, et al. Unsupervised domain generalization by learning a bridge across domains. In CVPR , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , 2016.

Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2(7), 2015.

Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In UAI , 2020.

Chen Huang, Chen Change Loy, and Xiaoou Tang. Local similarity-aware deep feature embedding. In NeurIPS , 2016.

Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Fsdr: Frequency space domain randomization for domain generalization. In CVPR , 2021.

Zeyi Huang, Haohan Wang, Dong Huang, Yong Jae Lee, and Eric P Xing. The two dimensions of worst-case training and their integrated effect for out-of-domain generalization. In CVPR , 2022.

Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV , 2020.

Like Hui and Mikhail Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks. In ICLR , 2021.

Edwin T Jaynes. Information theory and statistical mechanics. Physical review , 106(4):620, 1957.

Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In ICCV , 2021.

Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learning to generalize semantic segmentation. In CVPR , 2022.

Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets. In ICCV , 2021.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In ICML , 2021.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV , 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , 2018.

Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV , 2019.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR , 2018.

Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In ICCV , 2021.

Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling-Yu Duan. Uncertainty modeling for out-of-distribution generalization. In ICLR , 2022.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV , 2018.

Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin-based label smoothing for network calibration. In CVPR , 2022.

David JC MacKay. Information theory, inference and learning algorithms . Cambridge university press, 2003.

Saeid Motiian, Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep supervised domain adaptation and generalization. In ICCV , 2017.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML , 2013.

Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR , 2021.

Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In CVPR , 2017.

Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and generalization capacities via ibn-net. In ECCV , 2018.

Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization on unseen domains via inference-time label-preserving target projections. In CVPR , 2021.

Hyekang Park, Jongyoun Noh, Youngmin Oh, Donghyeon Baek, and Bumsub Ham. Acls: Adaptive and conditional label smoothing for network calibration. In ICCV , 2023.

Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In ICCV , 2019.

Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. In NeurIPS , 2021.

Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via commonspecific low-rank decomposition. In ICML , 2020.

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-ofdistribution generalization. In ICML , 2022.

Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV , 2016.

German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR , 2016.

Yangjun Ruan, Yann Dubois, and Chris J Maddison. Optimal representations for covariate shift. In ICLR , 2022.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR , 2020.

Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. 2018.

Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal , 27(3):379-423, 1948.

Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. In ICLR , 2021.

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV , 2016.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR , 2016.

Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03532 , 2020.

Zhiqiang Tang, Yunhe Gao, Yi Zhu, Zhi Zhang, Mu Li, and Dimitris N Metaxas. Crossnorm and selfnorm for generalization under distribution shifts. In ICCV , pages 52-61, 2021.

Vladimir Vapnik. Principles of risk minimization for learning theory. In NeurIPS , 1991.

Vladimir Vapnik. The nature of statistical learning theory . Springer science & business media, 1999.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , 2017.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR , 2021.

Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML , 2022.

Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In CVPR , 2021.

Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 , 2020.

Fu-En Yang, Yuan-Chia Cheng, Zu-Yun Shiau, and Yu-Chiang Frank Wang. Adversarial teacher-student representation learning for domain generalization. In NeurIPS , 2021.

Huaxiu Yao, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. Leveraging domain relations for domain generalization. arXiv preprint arXiv:2302.02609 , 2023.

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR , 2020.

Yuhui Yuan, Kuiyuan Yang, and Chao Zhang. Hard-aware deeply cascaded embedding. In ICCV , 2017.

Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. In ICCV , 2019.

Seniha Esen Yuksel, Joseph N Wilson, and Paul D Gader. Twenty years of mixture of experts. IEEE TNNLS , 23(8):1177-1193, 2012.

Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: A meta-learning approach for tackling group distribution shift. arXiv preprint arXiv:2007.02931 , 2020.

Guanhua Zheng, Jitao Sang, and Changsheng Xu. Understanding deep learning generalization by maximum entropy. In ICLR , 2018.

Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang. Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-experts. NeurIPS , 2022.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. IEEE TIP , 30:8008-8018, 2021.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR , 2021.

Appendix

In this section, we provide

Limitations and future works in Section A

Pseudocode of LFME in Section B.

Explanations of why existing KD ideas perform inferior to our LFME in Section C.

More analysis regarding LFME in Section D;

Compare with MoE-based methods and other related methods in Section E.

Existing Ideas with Same Training Resources in Section F.

Detailed settings of the learning-based aggregation method Dyn_Expt in Sec. 6.4 from the manuscript in Section G;

8. Detailed results of LFME in the unseen domains of the different datasets from DomainBed, and detailed results of LFME in the different categories from the semantic segmentation task in Section H.

A Limitations and Future Works

Although our method shows competitive results for generalization, there are certain occasions when it will encounter setbacks. First, LFME is not applicable for the setting where the domain labels are unavailable in the training data, such as those collected from the internet. Since the training of the experts requires data grouped by domain labels. Second, LFME cannot handle the situation when only one source domain is provided, preventing it from performing in a more difficult single-source generalization task. How to apply LFME to a more general setting will be our primary task in future works. Besides, as the designs and theoretical supports are built mainly for the classification task, finding a proper solution to extend them to the regression tasks is also a promising direction in potential future works.

B Pseudo Code of LFME

This section provides the Pytorch-style pseudocode of our method. As detailed in Alg. 1, the implementation of our method is embarrassingly simple, introducing only one hyper-parameter upon the baseline ERM.

C Compare with Current KD Based on the Analysis in Sec. 4.1

As observed in Sec. 6.3, some existing KD ideas, perform inferior to our LFME. Based on our analysis, we find this is because existing ideas have difficulties achieving the beneficial effect introduced in Sec. 4.1. As detailed, our logit regularization ensures using more information for the target model by implicitly regularizing the probability in a much smaller range. This is hard to achieve by H( 𝑞, 𝑞 𝐸 ) and ∥ 𝑞 -𝑞 𝐸 ∥ 2 , where their probability 𝑞 will be in a similar range as that in ERM because the soft label 𝑞 𝐸 is still within the range of [ 0 , 1 ] , enforcing similarity between 𝑞 and 𝑞 𝐸 will not significantly change its distribution. Moreover, this effect is also hard to achieve by ∥ 𝑧 -𝑧 𝐸 ∥ and ∥ 𝑞 -𝑧 𝐸 ∥ 2 , which do not provide a specific range regularization effect for the probability.

D Further Analysis

This section provides more evidence to support the two analyses in Sec. 4 in the manuscript. The experiments are conducted on the widely-used PACS dataset and the difficult TerraInc dataset unless mentioned otherwise.

D.1 More Justification for Enabling Target Model to Use More Information

For the baseline model, we infer that the cross-entropy loss alone will enforce the model to only learn discriminative features that are specific to the label category. Because there are no relevant features to

Algorithm 1: PyTorch-style pseudocode of LFME.

interpret the non-label classes in ERM, the confidence for the target class will approach the maximum during training, and vice versa for other non-label classes. Differently, when the output probability has a smoother distribution, the model will also learn information that is shared with other classes to improve the corresponding non-label probabilities. Introducing more information during training will inevitably cause the phenomenon of low confidence for the label category, as probabilities for the non-label classes will increase (their summation increases from nearly 0 in Fig. 2 (a) to 0.35 in Fig. 2 (b)). Note that low confidence does not mean low accuracy. Because the label category corresponds to the largest logit value in both terms, and both the discriminative and newly introduced information can be used to characterize the label information, exploring more information can also boost classification. This finding is further validated by the in-domain results in Tab. 5, where LFME is shown to be able to also improve the in-domain test results compared to ERM, indicating that low confidence in our case actually leads to higher accuracy.

Visual examples to support the above analysis are shown in Fig. 4. We observe that compared to the baseline ERM, our method can utilize more regions for prediction. Such as that in the first 'horse" examples, our model utilizes both the head, tail, and body for prediction, while the ERM seems to focus only on the head region. These visual samples align with our analysis that compared to the baseline ERM, LFME tends to use more information for prediction.

D.2 Other Alternatives to Enlarge 𝑞 𝑐 for ∀ 𝑐 ≠ ∗

This section provides more evidence to support the first analysis in Sec. 4 by conducting experiments on two variants that also enlarge 𝑞 𝑐 for ∀ 𝑐 ≠ ∗ . (1) we test a model that directly enlarges 𝑞 𝑐 for ∀ 𝑐 ≠ ∗ . Specifically, we use the probability from LFME as the label to guide the probability output of a new target model, i.e . L 𝑔𝑢𝑖𝑑 = ∥ 𝑞 -𝑞 𝐿𝐹𝑀𝐸 ∥ 2 with 𝑞 and 𝑞 𝐿𝐹𝑀𝐸 from the new target and LFME models, respectively. Ideally, this modification ( i.e . LFME_Guid) can also improve the baseline model because it also encourages the model to learn more information shared with other classes

(a) Input

(b) ERM

(c) Ours

without compromising discrimination; (2) we use the output probability from the target model itself to replace that from the experts in Eq. (3), which reformulates L 𝑔𝑢𝑖𝑑 into ∥ 𝑧 -softmax ( 𝑧 )∥ 2 (softmax ( 𝑧 ) is followed with a detach operation in updating). According to our analyses in Sec. 4, this alternative ( i.e . Self_Guid) should also boost generalization because it enforces the target model to learn more information and focus more on the hard sample from itself;

Results are illustrated in Tab. 7. We observe that both the two variants LFME_Guid and Self_Guid are beneficial to the baseline models, leading ERM in all unseen domains. These results further validate our first analysis in Sec. 4, and we can conclude that when the basic discrimination is not compromised, either directly or indirectly enlarging 𝑞 𝑐 for ∀ 𝑐 ≠ ∗ can help the target model learn more information, and improve generalization accordingly.

D.3 Hard Samples from the Experts

As analyzed in Sec. 4, LFME not only encourages the target model to learn more information but also implicitly helps it to focus more on the hard samples from the experts. To examine if the classifications of hard samples from the experts are indeed improved, we plot the classification ratio R = 𝑝 ∗ max ( 𝑝 ) , where 𝑝 denotes the normalized probability: 𝑝 = 𝑝 -min ( 𝑝 ) max ( 𝑝 ) -min ( 𝑝 ) , from the target model on these hard samples. We use the ratio because it can quantify the correct predictions regarding the hard samples more precisely than the accuracy: the closer R approaches 1 , the better the corresponding model performs on the sample. In this setting, 1 / 3 samples with larger losses in a training batch are considered hard samples. We compare R from LFME with that from ERM to highlight the difference. As can be observed in Fig. 5 (a), most R from LFME is larger than that from ERM over the iterations, denoting LFME is better at handling these hard samples than ERM. As a comparison, R from these two models almost overlap in the easy samples as shown in Fig. 5 (b). These results validate our analysis that LFME implicitly helps the target model to focus more on the hard samples compared with ERM.

Meanwhile, to validate if the hard samples from experts can indeed help generalization, we conduct experiments by explicitly putting more weights on the hard samples from the experts on the basis of the free launch ERM+ ( i.e . ERM+w/ expt). Specifically, the target objective based on Eq. (9) is adjusted into: 𝑤 L 𝑎𝑙𝑙 , where 𝑤 is the weight for the training samples and is determined based on the loss of the experts. Basically, the larger the corresponding loss from the experts, the larger the value of 𝑤 should be, and we use the strategy from [35] to implement this hard sample mining process. As can be observed in Tab. 7, mining hard samples from the experts can improve the base model of ERM+, especially in the difficult TerraInc dataset. These results validate that mining hard samples from experts is beneficial for generalization.

D.4 Comparisons Between Hard Samples from Different Models: Why Experts are Required

We note in Tab. 7, the model Self_Guid performs inferior to LFME in both the two datasets, and the improvements are also marginal compared with ERM+ on account of variances. These results indicate the hard samples from the model itself may be less effective for generalization compared with that from the experts. To validate this hypothesis, we conduct experiments by explicitly focusing more on the hard samples from the model itself on the basis of ERM+ ( i.e . ERM+ w/ self). The implementation is the same as that in ERM+ w/ expt.

Results listed in the 6th row in Tab. 7 validate this hypothesis, where the hard samples from the model itself can hardly improve generalization compared with that from the experts. We presume the main reason is that besides the dominant specific domain information, hard samples from the experts also contain some out-of-the-domain information that makes it challenging for the corresponding expert. Examples are shown in Fig. 6. We observe that compared with hard samples from the model itself ( i.e . Fig. 6 (b)), hard samples from the experts ( i.e . Fig. 6 (a)) contain more ambiguous data located in the mixed region or the boundary of two domains than, indicating the experts may be more effective at revealing samples that with characteristics from different domains. By implicitly emphasizing these out-of-the-specific-domain samples, can the target model learn domain-agnostic features, thus improving generalization accordingly. Because the naive training strategies are fed with data from different domains at the same time, the out-of-the-specific-domain information is difficult to discover in their framework, explaining why hard samples from other strategies may not contribute. This is also the reason why the experts must be involved in the overall framework.

E Compare with MoE-Based Ideas and Other Arts

Since our idea involves training multiple experts, one may connect it to the idea of mixture of experts (MoE) [86]. However, it is noteworthy that our LFME is not a simple implementation of existing techniques. The introduced logit regularization term is new and can be properly explained for its effectiveness. Some similar ideas that use MoE have been explored in other works for improving DG. For example, in DAELDG [90], following a shared feature extractor, each domain corresponds to a specific classifier ( i.e . expert), whose output is enforced to be similar to the average outputs from the non-expert classifiers. Different from our work, it uses the average outputs from different experts as the final result. In Meta-DMoE [89], a meta-learning-based framework is employed to enforce

0

1000

2000

3000

4000

5000

Iteration steps

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

 from LFE

 from ERM

(a)

R

from the hard samples

0

1000

2000

3000

4000

5000

Iteration steps

0.70

0.75

0.80

0.85

0.90

0.95

1.00

 from LFE

 from ERM

(b) R from the easy samples

cartoon

photo

sketch

(a) Hard samples from experts

cartoon

photo

sketch

(b) Hard samples from the model itself

the feature of the target model to be similar to the aggregated features from the experts, which is different from our logit regularization idea. In Sec. D.1 and 6.3, we implement the basic designs of these two ideas with the framework of LFME and show their ineffectiveness. Besides the comparison with Meta-DMoE in Tab. 1, in this section, we further compare LFME with them using their original settings in the PACS and OfficeHome datasets with the benchmark provided in [90]. We also compare with several other recent DG methods. Results in Tab. 8 show that LFME performs favorably against DAELDG, and outperforms Meta-DMoE, further validating the advantages of the logit regularization term. We also note LFME performs better than other arts with a different benchmark, demonstrating the effectiveness of the proposed approach.

F Existing Ideas with Same Training Resources

Since LFME requires learning different experts during training, inevitably using more parameters than existing methods, one may wonder if the effectiveness of LFME derives from using these extra parameters from the experts. To examine this idea, we conduct experiments by implementing some leading arts ( i.e . CORAL, SD) and the baseline ERM with 𝑀 + 1 times of model size (where 𝑀 is the domain number). Specifically, the feature extractor for the larger method will contain 𝑀 + 1 branches, each with the same pretrained ResNet backbone. We concate the final outputs from the different branches and use it as input for a classifier to obtain the final result. Note that in this setting, a sample

will go through 𝑀 + 1 times of forward passes for both training and inference, which is more than that of LFME design. Results are shown in Tab. 9. We note that compared to the results from the original models, when using the same pretrained knowledge, naively expanding model size cannot improve the performance. The reason may be that a well-pretrained small backbone can already saturate on limited training data (as shown in Table 5, ERM can achieve more than 0.96 acc in the source domains), thus it is unnecessary to use more parameters in these datasets.

G Detailed Settings of Dyn_Expt

Besides the different experts 𝐸 𝑖 where 𝑖 denotes the domain label, Dyn_Expt also uses an extra weighting network 𝑊 to estimate the labels of the training data. During the training stage, the experts are trained the same as that in LFME, and the weighting network is trained using the corresponding domain labels by minimizing H( 𝑊 ( 𝑥 𝑖 ) , 𝑖 ) . During the test phase, Dyn_Expt dynamically combines the outputs from the experts with the corresponding domain probability: for ∀ 𝑥 ∈ D 𝑀 + 1 , the final result is ˝ 𝑀 𝑖 𝑊 ( 𝑥 ) 𝑖 𝐸 𝑖 ( 𝑥 ) . We use the same backbone for both the experts and the weighting module in this experiment.

H Detailed Results

This section presents the detailed results on the semantic segmentation task in Tab. 10, and detailed results in the unseen domains from the different unseen domains of DomainBed benchmark in Tab. 11, 12, 13 14, and 15.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Main claims made in the abstract and introduction accurately reflect the paper's contributions and scope, which matches our theoretical analyses and experimental findings.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations have been discussed in Sec. A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide our theoretical statements and proof in Sec. 4.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide pseudocode in Sec. B, and our code in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided at https://github.com/liangchen527/LFME . Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Implementation details are described in our experimental sections.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars for our experiments and detailed how they are computed. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For all experiments, we use a Tesla V100 GPU with 32 GB memory, the actual running time is detailed in Tab. 11.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm our work does not violate NeurIPS Code of Ethics, in any respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work aims to improve the generalizability of a deep model. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not involve such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Benchmarks used in this data are properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple framework called LFME for learning from multiple experts in domain generalization. LFME introduces a logarithmic regularization term to enforce similarity between the target model and expert models, allowing the target model to acquire expertise from all source domains and perform well in any testing domain. Experimental results demonstrate that LFME outperforms existing techniques not only in classification and segmentation tasks but also reveals through in-depth analysis its ability to implicitly use more information for predictions and to discover difficult samples from experts, thereby enhancing generalization capability.\n\nSTRENGTHS:\nThis paper proposes a novel idea on the basis of a new knowledge distillation paradigm for DG. The motivation of using multiple experts for improving generalization is clear and reasonable. Meanwhile, I also found the paper well-presented and easy to understand.\n\nAfter checking the pseudocode and the provided code, I found the implementation extremely simple. Given the impressive results computed with rigorous evaluation protocols, I believe this paper can offer valuable contributions to the literature by providing an accessible and effective solution. \n\nThe provided deep analyses that explains why their method works are appreciated and, in my opinion, crucial for a paper, especially in the era of deep learning. Coupled with the empirical evidence in their supplementary material, I think most of claims made in their paper can be well supported.\n\nWEAKNESSES:\nRegarding the proposed method, although large improvements over the baseline ERM are observed, their method involves higher training costs. I noticed the authors provide the training time comparisons with different methods in their supplementary material, which shows that their LFME requires 1.5 times more training time than ERM, and it is also one of the most time-consuming methods compared. While I agree with the authors that requiring more training time is inevitable for methods designed based on KD, the authors should at least list this as a major drawback in their limitation section.\n\nAs in recent times, Vision Transformers (ViTs) have demonstrated substantial improvements in the classification and semantic segmentation tasks. Although the authors conducted experiments using different ResNet models, they should also consider adopting a ViT backbone for their experiments. This could provide valuable insights into the performance of their method when applied to more advanced architectures, potentially highlighting further benefits or limitations. Including such experiments would enhance the comprehensiveness of the study and align it with current trends in the field.\n\nQUESTIONS:\nIn Section 4.2, the authors claim that their method improves DG by explicitly mining hard samples from the experts. However, Tables 3 and 6 show that specifically mining hard samples from the experts improves performance of ERM only in the TerraInc dataset, but not in the PACS dataset (with an average improvement of only 0.1 pp). Could the authors provide an explanation for this phenomenon?\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving domain generalization by utilizing multiple experts. Particularly, a simple yet effective framework is proposed whereby a target (student) model is learned from multiple expert (teacher) models through logit regularization. After learning, the target model can grasp knowledge from multiple source domains and shows advantages in handling hard samples. The proposed approach demonstrates consistent improvements across different evaluation tasks, outperforming existing SOTAs.\n\nSTRENGTHS:\nThis paper overall is well written. It provides in-depth theoretical insights on its proposed logit regularization and surrogate ablation studies.\n\nWEAKNESSES:\n1: The contribution of the proposed approach is marginal. The performance of the proposed approach does not show much improvement compared to ERM or expert models (Table 5). \n\n2: The baseline models for comparison are not clearly introduced, which creates difficulties in understanding the Tables.\n\n3: The comparison to the approaches that directly learn a foundation model using data from all source domains is not discussed. Hence, the benefits of the proposed approach, which have to use a set of expert models plus a target model, compared to the foundation model are not clear.\n\nQUESTIONS:\n1: What is the difference between the proposed approach and foundation models? Why not directly learn a model using all the source domains? How does the performance of the proposed approach compare to Segment Everything?\n\n2: What’s the difference between the proposed approach and the baseline ERM? In lines 77-78, the authors mention ‘only one hyper-parameter’ without further illustration. The difference is clarified only until line 138. Statements near lines 77-78 need to be improved to avoid confusion. More importantly, it can be misleading to say “only one additional parameter upon ERM” since a target model is also involved.\n\n3: How does the value q_*^E relate to the hardness of samples? What causes the sample to be ‘hard’? Low data frequence or image quality?  Without these clarified, the provided theoretical insights are not convincing.\n\nLIMITATIONS:\nNo discussions on limitations and potential negative societal impact of the work are provided. \n\nOne limitation is that the performance of the proposed approach can be upper bounded by the performance of the expert models. This limitation again raises the question of why not directly learn from all source domains?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of domain generalization from the perspective of  ''learning multiple experts''. In particular, they propose to train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. The proposed logit regularization provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Experiments on standard DG benchmarks demonstrate the effectiveness of the proposed framework.\n\nSTRENGTHS:\n- The overall manuscript is well-organized and easy to follow.\n\n- The method is simple and effective. Its impressive results in different tasks demonstrate the strong applicability of this idea to potential other tasks.\n\n- The related work section is comprehensive, especially regarding comparisons with Meta-DMoE that also tries to distill knowledge from domain experts.\n\n- The idea of using MSE loss between logit and probability for distillation is new and well-explained, supported by both theoretical and empirical evidence.\n\n- The free lunch idea introduced in sec. 6.1 is interesting and seems to have the potential for more general applications.\n\nWEAKNESSES:\nWhile adopting the idea of knowledge distillation (KD) and the concept of domain experts for improving DG is not new in the literature, it is appreciated that this method can be implemented in such simple form and be explained with extensive insights. Nevertheless, there are still a certain amount of claims in the paper that requires proper explanations or more justification:\n\n- Performance reported in the original Meta-DMoE paper is higher than that in Table 1. For example, Meta-DMoE reports an average of 86.9 in PACS, surpassing the best result listed in the table. The authors should provide an explanation for this discrepancy. Moreover, I noticed that the training time comparisons between these two methods are provided, it would also be beneficial to highlight the test and training resource differences between these two methods in the related work section.\n\n- The authors mention that their method introduces only one hyper-parameter, which is within a certain range. It would be beneficial to include an ablation study to explain why this specific range was chosen.\n\n- It would be beneficial to evaluate this method with more real-world DG problem, such as using dataset from wilds benchmark.\n\n- The method [1] with similar settings should be compared in Table 1. Additionally, some methods, such as MIRO, report improved performance when combined with SWAD. Including these results in Table 1 would provide a more comprehensive comparison.\n\n- Minor issues. Formats of ”i.e.” in line 145, ”Tab.” in line 578, and ”Fig.” in line 581 are inconsistent with their usages in other places. The authors should standardize the format of similar abbreviations to maintain uniformity.\n\nReference: \n\n[1] Domain Generalization via Rationale Invariance, in ICCV 2023.\n\nQUESTIONS:\n- This method needs to train additional expert per domain, what if there are large amount of domains for the training data, any potential solution for this occasion?\n\n- Since multiple expert models are required when training the target model, how would this method be applied to the training process of large foundation models?\n\n- The free lunch idea appears to be applicable beyond just the DG setting. How does it perform on ImageNet? Including such experiments could enhance the significance of this concept.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe computer-vision paper introduces a strategy of learning from multiple experts (LFME), which performs knowledge distillation from models specially trained on data from different domains. In particular, the experts are trained jointly with the target model, and a specific form of logit regularization is chosen for knowledge distillation due to empirical verification. The idea is relatively straightforward to understand with strong empirical results on image classification (by Domainbed) and semantic segmentation.\n\nSTRENGTHS:\n1. The experimental results come in strong when compared to many other baselines. Moreover, the study has extended results for semantic segmentation. \n2. The visualizations are helpful to illustrate the ideas and evidence. \n3. The empirical study on naive aggregation of domain experts shows that naive ensemble does not necessarily lead to better generalization. Then ablation study shows the chosen form of logit regularization outperforms other knowledge distillation alternatives.\n\nWEAKNESSES:\n1. The novelty is relatively limited. The proposed approach is very similar to Meta-DMoE with a few changes such as logit regularization and alternating updates, but these changes are not well justified (see below). The idea of utilizing multiple domains have also been explored by [1]. \n2. There is a lack of coherent argument to understand the proposed approach. In Section 4.1, the paper briefly discussed how the additional logit regularization enables learning more information. In addition, in Section 6.2, the proposed approach LFME is compared with Label Smoothing. However, the discussion is not very convincing. From the perspective of using information from other classes, I do not see the fundamental difference between logit regularization and Label Smoothing (LS). Both logit regularization and LS have hyperparameters to be tuned, so it might not be appropriate to claim its advantage as \"not involve hand-crafted settings” (line 187) and criticize LS has “potential improper heuristic designs” (189). \n3. Moreover, in Section 4.2, the paper justifies the advantage of the proposed approach LFME by “mining hard samples from the experts”. It is a general statement that is true for low-confidence predictions from any model, so it does not constitute a strong argument. Intuitively, the in-domain samples are easier than out-of-domain samples for each expert, and the hard samples aggregated from all experts are not specifically representative of any subpopulations. The argument will be more comprehensive if the paper can show what are these hard samples and why it matters to mine the hard samples from the domain experts (and not an ensemble of experts or some random experts). \n4. In Section 6.5, the result for in-domain evaluation is only provided for one dataset. This is not sufficient to justify a strong claim that the target model is an expert on all source domains. \n5. Code is submitted but it is unclear how the proposed approach can be combined with SWAD. Does it apply to just the target model or also the domain experts? The hyperparameter configurations to obtain the SoTA results are also unclear for reproducibility. \n\n[1] Yao, Huaxiu, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. \"Leveraging domain relations for domain generalization.\" *arXiv preprint arXiv:2302.02609* (2023).\n\nQUESTIONS:\n1. In Algorithm 1, the target model and the experts are trained simultaneously, which is a more parallel and iterative scheme rather than a sequential one with two stages, i.e., training the experts first, then distilling the knowledge to the target model using logit regularization. Why is such a design choice made? Are there better justifications than just “for simplicity”? Please explain with more details. \n2. Each expert is generally weak as they are only trained using the data from their own domain. Is it necessary and how to ensure the quality of the experts?\n3. In Appendix A, Limitations, the authors discussed that LFME cannot be applied to single-domain tasks. How does the number of domains affect the proposed approach? Will the algorithm benefit from having more domains with less data or more data in fewer domains?\n\nLIMITATIONS:\nThe limitations has been discussed in Appendix A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}