LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization

Authors: liang  chen
yong  zhang
yibing  song
zhiqiang  shen
lingqiao  liu
  mbzuai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:59:59

Abstract: Domain generalization (DG) methods aim to maintain good performance in an unseen target domain by using training data from multiple source domains. While success on certain occasions are observed, enhancing the baseline across most scenarios remains challenging. This work introduces a simple yet effective framework, dubbed learning from multiple experts (LFME), that aims to make the target model an expert in all source domains to improve DG. Specifically, besides learning the target model used in inference, LFME will also train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. Delving deep into the framework, we reveal that the introduced logit regularization term implicitly provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Extensive experiments on benchmarks from different DG tasks demonstrate that LFME is consistently beneficial to the baseline and can achieve comparable performance to existing arts.  Code is available at https://github.com/liangchen527/LFME.

Main paper: 
 introduction: 
 Deep networks trained with sufficient labeled data are expected to perform well when the training and test domains with similar distributions [74, 4] . However, test domains in real-world often exhibit unexpected characteristics, leading to significant performance degradation for the trained model. Such a problem is referred to as distribution shift and is ubiquitous in common tasks such as image classification [43, 27] and semantic segmentation [19, 40] . Various domain generalization (DG) approaches have been proposed to address the distribution shift problem lately, such as invariant representation learning [53, 69, 57, 62, 29] , augmentation [91, 48, 79, 47] , adversarial learning [24, 46, 81, 49] , meta-learning [44, 2, 22, 45] , to name a few. Yet, according to [27] , most arts perform inferior to the classical Empirical Risk Minimization (ERM) when applied with restricted hyperparameter search and evaluation protocol. Both the experiments in [27] and our findings suggest that existing models are incapable of consistently improving ERM in all evaluated datasets. The consistent improvement for ERM thus becomes our motivation to further explore DG.Our approach derives from the observation in [10] that some of the data encountered at test time are similar to one or more source domains, and in which case, utilizing expert models specialized in the domains might aid the model in making a better prediction. The observation can be better interpreted with the example in Fig. 1 . Given experts trained in the "infograph", "real", and "quickdraw" domains, and test samples from the novel "sketch" domain. Due to the large domain shift, it would be better to rely mostly on the expert trained on the similar "quickdraw" domain than others. To obtain a target model that is an expert on all source domains, we learn multiple experts specialized in corresponding domains to help guide the target model during training. For each sample, the guidance is implemented with a logit regularization term that enforces similarity between the logit of the target model and probability from the corresponding expert. Only the target model is utilized in inference.Please refer to Algorithm 1 for detailed implementations.However, the test domain information is often unavailable in DG, indicating that we can not specifically train an expert specialized in a particular domain. In light of this, obtaining a target model that is an expert on all source domains seems to be a practical alternative for handling potential arbitrary test domains. A naive implementation would be training multiple experts on each domain, and dynamically aggregate them to form the target model. So that any encountered test samples can be predicted by corresponding experts who are familiar with their characteristics. Nevertheless, such a practice has two inherent limitations: (1) designing an effective aggregation mechanism is essential and inevitable for the naive model. In fact, our experimental study indicates that using naive aggregating strategies, such as averaging, may deteriorate the performance. (2) the overall framework requires much more resources for deployment when there are many training domains, since all the experts must be leveraged during the test phase.This work proposes a simple framework for learning from multiple experts (LFME), capable of obtaining an expert specialized in all source domains while avoiding the aforementioned limitations. Specifically, during the training stage, instead of heuristically aggregating different experts, we suggest training a universal target model that directly inherits knowledge from all these experts, which is achieved by a simple logit regularization term that enforces the logit of the target model to be similar to probability from the corresponding expert. With this approach, the target model is expected to leverage professional guidance from multiple experts, evolving into an expert across all source domains. During the test phase, only the target model is utilized. As a result, both model aggregations and extra memory and computation resources are not required during the deployment, since we only leverage one model. The overall training and test pipelines are illustrated in Fig. 1 .Our method can be interpreted through the lens of knowledge distillation (KD), where the core idea is transferring knowledge by training the student (i.e. the target model) with soft labels from teachers (i.e. experts) [31] . Unlike traditional KD [31] that uses teachers' output probabilities as soft labels in a cross entropy loss, we employ a logit regularization term that uses experts' probabilities to refine the logit of the target model in a regression manner, which can be regarded as extending the effectiveness of mean squared error (MSE) loss in classification [37] within the KD realm.To gain a deeper understanding of the effectiveness of our logit regularization term, we perform in-depth analyses and uncover that its merit over the baseline can be explained in twofold. (1) It implicitly regularizes the probability of the target model within a smaller range, enabling it to use more information for prediction and improve DG accordingly. It is noteworthy that the effect is achieved inherently differs from that by label smoothing (LS) [71] , as LFME does not require explicitly calibration for the output probability. Expanding upon this analysis, we find that a simple combination of cross entropy and MSE losses achieves comparable performance among existing arts. Given its straightforward implementation without unnecessary complexities, this expanding may offer a "free lunch" for DG; (2) It further boosts generalization by helping the target model to focus more on hard samples from the experts, supported by our theoretical finding. Through experiments on different datasets, we find that hard samples from the experts are more beneficial for generalization than those from the model itself. Given that hard sample mining is essential for easing distribution shift [35, 42] , this discovery may offer valuable guidance for future research endeavors.Through evaluations on the classification task with the DomainBed benchmark [27] and segmentation task with the synthetic [63, 64] to real [20, 83, 55] setting, we illustrate that LFME is consistently beneficial to the baseline and can obtain favorable performance against current arts (other KD ideas included). Our method favors extreme simplicity, adding only one hyper-parameter, that can be tuned in a rather large range, upon training the baseline ERM. 
related works: 
 General DG methods. Domain generalization (DG), designed to enable a learned model to maintain robust results in unknown distribution shift, is gaining increasing attention in the research community lately. The problem can be traced back to a decade ago [6] , and various approaches and applications have been proposed to push the generalization boundary ever since [53, 26, 46, 32, 81, 49, 2, 16, 11, 13, 82, 12] . The pioneering work [4] theoretically proves that the DG performance is bounded by both the intra-domain accuracy and inter-domain differences. Most previous arts focus on reducing the inter-domain differences by learning domain-invariant features with ideas such as kernel methods [53, 26] , feature disentanglement [10, 61] , and gradient regularization techniques [69, 62] . Same endeavors also include different learning skills: adversarial training is leveraged [24, 81] to enforce representations to be domain agnostic; meta-learning is utilized [2, 45] to simulate distribution shifts during training. Other works aim to improve the intra-domain accuracy: some suggest explicitly mining hard samples or representations with handcraft designs, such as masking out dominant features [36] , weighting more on hard samples [42] , or both [35] . LFME falls into this category as the target model can also mine hard samples from the experts, beneficial for excelling in all source domains (in Sec. 6.5).Utilizing experts for DG. Methods with the most relevant motivations with our LFME are perhaps those also involves experts [86, 28, 89, 90] . In DAELDG [90] , a shared feature extractor is adopted, which is followed by different classifiers (i.e. expert) that correspond to specific domains. Their experts are trained by enforcing the outputs to be similar to the average output from the non-expert classifiers. Different from our work, it uses the average outputs from different experts as the final result. In Meta-DMoE [89] , similar to LFME, different experts are trained on their specific domains where a traditional KD idea is adopted: the feature from the target model is enforced to be similar to the transformer-processed version of their experts' output features. Notably, Meta-DMoE and LFME share very distinct objectives for the expert models. Specifically, Meta-DMoE aims to adapt the trained target model to a new domain in test. To facilitate adaptation, their target model is assumed to be capable of identifying domain-specific information (DSI), and is enforced to extract DSI similar to those from domain experts. In their settings, domain experts are expected to thrive in all domains and are used not in their trained domains but rather in an unseen one. Differently, LFME expects its target model to be expert in all source domains, where domain experts provide professional guidance for the target model only in their corresponding domains. Additionally, Meta-DMoE involves meta-learning and test-time training, which is more complicated than the end-to-end training adopted in LFME.Unlike their empirical design, our method is more self-contained, supported by in-depth analysis to explain its efficacy (in Sec. 4). We further show in our experiments (in Sec. 5.1 and E) that these related two methods perform inferior to our design, and the improvements from their basic designs: using average performance (in Sec. 6.4) or enforcing feature similarity between the target model and experts (in Sec. 6.3) are subtle compared to our logit regularization.DG in semantic segmentation. Different from image classification, semantic segmentation involves classifying each pixel of the image, and the generalizing task often expects a model trained from the synthetic environments to perform well on real-world views. Directly extending general DG ideas to semantic segmentation is not easy. Current solutions mainly consist of domain randomization [34, 85] , normalization [19, 56, 73] , or using some task-related designs, such as the class memory bank in [40] . Different from some existing DG methods, LFME can be directly extended to ease the distribution shift problem in the semantic segmentation task without requiring any tweaks, and we show that it can obtain competitive performance against recent arts specially designed for the task. 
methodology: 
 Problem Setting. In the vanilla DG setting, we are given 𝑀 source domainsD 𝑠 = {D 1 , D 2 , ..., D 𝑀 },where D 𝑖 is the 𝑖-th source domain containing data-label pairs (𝑥 𝑖 𝑛 , 𝑦 𝑖 𝑛 ) sampled from different probabilities on the joint space X × Y, the goal is to learn a model from D 𝑠 for making predictions on the data from the unseen target domain D 𝑀+1 . For either DG or the downstream semantic segmentation task, source and target domains are considered with an identical label space, and we assume that there are a total of 𝐾 classes.Learning multiple experts. LFME trains all the experts and the target model simultaneously, and the training procedure is illustrated in the upper part of Fig. 1 . As each expert corresponds to a specific domain, given the 𝑀 source domains, a total of 𝑀 experts have to be trained during this stage, and the training process of each expert is the same as that of the ERM model. Given a training batch B ∈ D 𝑠 , for the 𝑖-th expert 𝐸 𝑖 , we only use data from the 𝑖-th domain, and the computed loss regarding the data-label pair (𝑥 𝑖 , 𝑦 𝑖 ) ∈ (B ∩ D 𝑖 ) can be written as,EQUATIONwhere 𝑞 𝐸 𝑖 ∈ R 𝐾 is the output probability computed by applying the softmax function over the output logits 𝑧 𝐸 𝑖 with 𝑧 𝐸 𝑖 = 𝐸 𝑖 (𝑥 𝑖 ); H denotes the cross-entropy loss: H (𝑞, 𝑦) = 𝐾 𝑐 -𝑦 𝑐 log 𝑞 𝑐 . Learning the target model. Data-label pairs (𝑥, 𝑦) ∈ B from all domains are used for training the target model 𝑇. The main classification loss L 𝑐𝑙𝑎 is computed similar to Eq. ( 1): L 𝑐𝑙𝑎 = H (𝑞, 𝑦), s.t. 𝑞 = softmax(𝑧) = 𝑒𝑥 𝑝 (𝑧 𝑐 ) 𝐾 𝑗 𝑒𝑥 𝑝 (𝑧 𝑗 ) and 𝑧 = 𝑇 (𝑥). Then, to incorporate professional guidance from the experts, we further introduce a logit regularization term L 𝑔𝑢𝑖𝑑 , to assist 𝑇 to become an expert on all source domains, which is computed by using the probabilities from the experts as a label for 𝑇:EQUATIONwhere 𝑞 𝐸 is the concatenate probabilities from different experts along the batch dimension2 , ∥ • ∥ denotes the 𝐿 2 norm, and this term is only enforced on the target model. Note we use the normalized version of 𝑧 𝐸 (i.e. 𝑞 𝐸 ) for computing L 𝑔𝑢𝑖𝑑 , which can be regarded as extending the effectiveness of MSE loss [37] in the KD realm. Our experimental studies (in Sec. 6 .3) find it leads to better performance, and the following contents also elaborate on the motivation. Then, the overall loss L 𝑎𝑙𝑙 for updating the target model can be represented as,EQUATIONwhere 𝛼 is the weight parameter, the only additional parameter upon ERM. We train the target model and the experts simultaneously for simplicity. Please refer to pseudocode in Algorithm 1 for details.Rationality (comprehension from a KD perspective). Our logit regularization term can be viewed as a new KD form, wherein the fundamental principle is to utilize the teachers' (i.e. experts) outputs as soft labels for the student (i.e. target model) in a training objective [31] . In the context of classification tasks, the cross entropy loss H (𝑞, 𝑦) is widely used in the literature. Based on this objective, an intuitive revision to achieve distillation is by replacing the ground-truth label 𝑦 with 𝑞 𝐸 in a cross entropy regularization manner (i.e. H (𝑞, 𝑞 𝐸 )), which builds the rationality for the pioneering KD art [31] . Nevertheless, a recent study [37] suggests that the MSE loss ∥𝑧 -𝑦∥ (without applying softmax function on 𝑧) performs as well as the cross entropy loss when being applied in the classification task. Correspondingly, a distillation scheme motivated by this objective can thus be utilizing the soft label 𝑞 𝐸 in a regression manner, which comes to our logit-regularized term: ∥𝑧 -𝑞 𝐸 ∥. From this perspective, the rationality of the introduced logit regularization term aligns with the principle of KD, and it can be regarded as extending the applicability of MSE loss in classification to the KD realm. We compare our new KD form with other ideas in Sec 6.3, demonstrating its superior performance against existing KD ideas in the DG task. We delve deep into our method and explain the effectiveness of LFME in the following section.Computational cost. Inherited from KD, training LFME inevitably requires more resources as both the teacher and student have to be involved during training. However, LFME uses the same test resources as the baseline ERM given only the target model is utilized. Meanwhile, please also note that the computational cost for LFME is not proportional w.r.t the domain size. Instead, the training cost will always be doubled compared to ERM, as each sample will require two forward passes: one for the target model and the other for the corresponding expert. Please refer to Tab. 11 for training time comparisons between different arts. In Sec. F, we show that simply increasing training resources for current arts cannot improve their performances, suggesting it may not be a primary factor in DG. Specifically, for the baseline model, using only the classification loss L 𝑐𝑙𝑎 encourages the probability 𝑞 to be diverse: the ground truth 𝑞 * to approximate 1 and 0 for others. Consequently, the corresponding logits 𝑧 * will increase ceaselessly, i.e. 𝑧 * → +∞, and vice versa for 𝑧 𝑐 , i.e. 𝑧 𝑐 → -∞, ∀𝑐 ≠ * (depicted in Fig. 2 (a) and (c)), as this is the solution for minimizinglog 𝑒𝑥 𝑝 (𝑧 * ) 𝑒𝑥 𝑝 (𝑧 𝑐 ) . From another point of view, L 𝑐𝑙𝑎 encourages the model to explicitly focus on the dominant and exclusive features that are strongly discriminative but may be biased towards simplistic patterns [25] .Differently, when L 𝑔𝑢𝑖𝑑 is imposed, the logits 𝑧 will approximate the range of 𝑞 𝐸 (i.e. [0, 1]). Eventually, the final logits will balance these two losses (i.e. 𝑧 * ∈ (𝑞 𝐸 * , +∞) and 𝑧 𝑐 ∈ (-∞, 𝑞 𝐸 𝑐 )∀𝑐 ≠ * as shown in Fig. 2 (d) ), resulting in a smoother distribution of 𝑞, where 𝑞 𝑐 (∀𝑐 ≠ * ) in LFME will be larger than it is in ERM (see Fig. 2 (b) ). Since both two losses encourage the model to make a good prediction (i.e. 𝑧 * is expected to be the largest in both losses), the increase of 𝑞 𝑐 indicates that besides learning the dominant features, the target model will be enforced to learn other information that is shared with others. Compared with ERM that prevents the model from learning other features, LFME is more likely to make good predictions when certain types of features are missing while others exist in unseen domains. We provide a "free lunch" inspired by the analysis in Sec. 6.1, and more justifications (including visual and empirical evidence) to support this analysis in Sec. D.1 and D.2. In Sec. C, we demonstrate that other KD ideas face challenges in achieving the same merit.The above finding can also be endorsed by the information theory [68] . Specifically, with a smoother distribution of 𝑞, the entropy, which measures information, will naturally increases [51] , suggesting a theoretical basis for utilizing more information in LFME. According to the principle of maximum entropy [38] , the improvement for generalization is thus foreseeable [88] .Note this effect is achieved inherently differs from that by label smoothing (LS) [71] , as LFME does not involve hand-crafted settings to deliberately calibrate the output probability, which is essential in LS. In Sec. 6.2, we show LS is ineffective in DG compared to LFME. Besides the advantage of avoiding problems raised by potential improper heuristic designs, we show in the following that the logit regularization in LFME provides another merit over LS. 
enabling the target model to mine hard samples from the experts: 
 This effect is realized by example re-weighting, motivated by proposition 2 in [72] . The single sample gradient of L 𝑎𝑙𝑙 with respect to the 𝑐-th logit value 𝑧 𝑐 can be formulated as,EQUATIONWhen the target probability corresponds to the ground-truth 𝑦 * = 1, Eq. ( 4) is reduced into,EQUATIONIn this situation, the rescaling factor F is given by,EQUATIONOn the other hand, for ∀𝑐 ≠ * , summing the gradient values over the finite indexes gives, and the rescaling factor F ′ in this situation is, ). We use the ImageNet [21] pretrained ResNet [30] as the backbone for both the experts and the target model. Following the designs in DomainBed, the hyper-parameter 𝛼 2 in Eq. ( 3) is randomly selected in a range of [0.01, 10]. To ensure fair comparisons, all methods are reevaluated using the default settings in DomainBed in the same device with each of them evaluated for 3 × 20 times in different domains. Training-domain validation is adopted as the evaluation protocol. Other settings (batch size, learning rate, dropout, etc.) are dynamically selected for each trial according to DomainBed.EQUATIONF ′ = ∑︁ 𝑐≠ * 𝜕L 𝑎𝑙𝑙 𝜕𝑧 𝑐 / ∑︁ 𝑐≠ * 𝜕L 𝑐𝑙𝑎 𝜕𝑧 𝑐 = 1 -𝛼 1 -𝑐≠ * 𝑧 𝑐 -𝑞 𝐸Results with ResNet18. Results are listed in Tab. 1. We observe that ERM can obtain competitive results among the models compared, which leads more than half of the sophisticated designs, and only 6 methods lead ERM in most datasets (with scores ≥ 3). We also note that current strategies cannot guarantee improvements for ERM in all situations, given that none of them with a score of 5. Differently, our method can consistently improve ERM in all evaluated datasets and lead others in average accuracy. Specifically, our approach obtains the leading results in 4 out of the 5 datasets, and it also improves ERM by a large margin (nearly 8pp) in the difficult TerraInc dataset. Compared to methods that explicitly explore hard samples or representations (i.e. VREx and RSC) and that use MoE (i.e. Meta-DMoE), the performances of LFME are superior to them in all cases.Results with ResNet50. Because larger networks require more training resources, we only reevaluate some of leading methods (i.e. ERM, Fish, CORAL, and SD). in our device when experimenting with ResNet50. We note that our method surpasses the baseline ERM model in all datasets and leads it by 1.8 in average. Meanwhile, our method can still outperform the second best (i.e. SD) by 0.7 in average. These results indicate that our method can consistently improve the baseline model, and it can perform favorably against existing arts when implemented with a larger ResNet50 backbone.We also combine LFME with SWAD [8] . Same with the original design, the hyper-parameter searching space in this setting is smaller than the original DomainBed. We use the reported numbers in [8] for comparisons. As shown, our method can also improve the baseline and obtain competitive results when combined with SWAD. These results further validate the effectiveness of our method. 
generalization in semantic segmentation: 
 Datasets and Implementation details. The training and test processes of the compared algorithms involve 5 different datasets: 2 synthetics for training, where each dataset is considered a specific domain, and 3 real datasets for evaluation. Synthetic: GTAV [63] has 24,966 images from 19 categories; Synthia [64] contains 9,400 images of 16 categories. Real: Cityscapes [20] 3,450 finely annotated and 20,000 coarsely-annotated images collected from 50 cities; Bdd100K [83] contains 8K diverse urban driving scene images; Mapillary [55] includes 25K street view images. Following the design [40] , we use the pretrained DeepLabv3+ [18] and ResNet50 as the segmentation backbone for experiments. The maximum iteration step is set to 120K with a batch size of 4, and the evaluations are conducted after the last iteration step. The hyper-parameter 𝛼 2 in Eq. ( 3) is fixed as 1 in this experiment for simplicity. We use the mean Intersection over Union (mIoU) and mean accuracy (mAcc) averaged over all classes as criteria to measure the segmentation performance.Experimental results. Results are shown in Tab. 2. To better justify the effectiveness of our method, we reevaluate the baseline model, which aggregates and trains on all source data, and PinMem in our device. We also implement SD [60] , which is a leading method in the image classification Table 3 : Evaluations of free lunch for DG (i.e. ERM+) and different LS ideas (i.e. LS [71] , MbLS [50] , and ACLS [58] ). We use the suggested settings in their original papers for evaluating. task. Results from other methods are directly cited from [19] . We observe SD does not perform as effectively as it does in the classification task, which decreases the baseline model in most situations. In comparison, our LFME can boost the baseline in all datasets, similar to that in the classification task. It also performs favorably against existing methods specially designed for the semantic segmentation task, obtaining best results in 2 out of the 3 evaluated datasets in both mIOU and mAcc.Visualized examples are provided in Fig. 3 . We note that when it comes to unseen objects (i.e. clouds with different shapes and a new car logo), or objects with an unfamiliar background, such as the person and car hidden in the shadow, due to the large distribution shift between real and synthetic data, compared methods make unsatisfactory predictions. In comparison, LFME can provide reasonable predictions in these objects, demonstrating its effectiveness against current arts regarding generalization to new scenes. These results validate the effectiveness of LFME and its strong applicability in the generalizable semantic segmentation task. 
analysis: 
 Analyses in this section are conducted on the widely-used PACS dataset unless otherwise mentioned. Experimental settings are same as that detailed in Sec. 5.1. Please see the appendix for more analysis.As stated in Sec. 4.1, when the basis of discrimination is not compromised (𝑞 * corresponds to a larger value in the label and vice versa for 𝑞 𝑐 for ∀𝑐 ≠ * ), the increase of 𝑞 𝑐 can encourage the model to learn more information that is shared with other classes, and use them to improve generalization. Based on this analysis, it seems that using the one-hot label to regularize the logit is also reasonable. To validate this hypothesis, we replace 𝑞 𝐸 in Eq. ( 3) with the ground truth and reformulate L 𝑎𝑙𝑙 into,EQUATIONDenoting as ERM+, results listed in the second row in Tab. 3 suggest that this idea can improve ERM in all unseen domains. Because the hard sample information is absent in this strategy, we observe it performs inferior to LFME. However, this alternative does not require experts or any other special designs, even the setting of the hyper-parameter 𝛼 cannot be wrong: Eq. (9) will approximate the baseline either with 𝛼 = 0 or 𝛼 approximates +∞. Thus, we argue this simple modification can serve as a free lunch to improve DG. More evaluations of this idea are in our appendix.As detailed in Sec. 4.1, LFME will explicitly constrain the probability of the target model within a smaller range. This effect may resemble the LS technique that aims to implicitly calibrate the output probability. However, compared to LS, LFME has two advantages. First, LFME does not involve heuristic designs of hyper-parameters for determining its probability, which is essentially required in existing LS ideas, such as 𝜖 in [71] and predefined margin in [50, 58] , avoiding the possibility of deteriorating the performance when not choosing them properly; Second, LFME can explicitly mine hard samples from the experts, further ensuring improvements for DG. We evaluate 3 LS methods in both PACS and the difficult TerraInc datasets: (1) the pioneer LS method from [71] );(2) margin-based LS (MbLS) [50] that penalizes logits deviate from the maxima; (3) adaptive and conditional LS (ACLS) that can adaptively determine the degree of smoothing for different classes.Results are illustrated in 3rd-5th columns in Tab. 3. We observe that although these LS methods can improve the baseline, they are all inferior to LFME, validating the effectiveness of LFME against LS.To test the effectiveness of our logit regularization (i.e. ∥𝑧 -𝑞 𝐸 ∥ 2 ), we compare it with several other KD options, namely different combinations of the logits or probabilities from the target model and the experts, including ∥𝑧 -𝑧 𝐸 ∥ 2 (which is the basic design in Meta-DMoE [89] ), ∥𝑞 -𝑧 𝐸 ∥ 2 , and ∥𝑞 -𝑞 𝐸 ∥ 2 . Moreover, we also compare it with a common practice in KD that uses the probability from the teacher (i.e. experts) as a label for the student (i.e. target model), which reformulates the Eq. ( 3) into:L 𝑎𝑙𝑙 = (1 -𝛼 2 )H (𝑞, 𝑦) + 𝛼 2 H (𝑞, 𝑞 𝐸 ), and denoted as H (𝑞, 𝑞 𝐸 ) in Tab. 4. Following the suggestion in [41] , we gradually increase 𝛼 2 over the iteration steps to achieve better performance. Results listed in 2nd-5th columns in Tab. 4 show that not all KD strategies can improve DG and the logit regularization choice achieves the best result in terms of average accuracy. This can be explained by the analysis in Sec. 4.1 that our logit regularization can help using more information. Please also the explanation in Sec. C for details. The training process of LFME aggregates all professional knowledge from the experts into one target model, aiming to make the target model an expert on all source domains. To examine if the goal has been achieved, we conduct in-domain validations for the target model and compare the performances with that of the experts and the ERM model in the PACS and TerraInc datasets. Note in these experiments, ERM and LFME are trained using the same leave-one-out strategy, and the performances are averaged over the trials on three different target domains. Results are listed in Tab. 5. We observe that ERM performs inferior to the experts in the in-domain setting. The results are not surprising. As stated earlier, when encountering data similar to the source domain, it would be better to rely mostly on the corresponding expert than the model also contaminated with other patterns. In comparison, our method obtains the best results in all source domains because it implicitly focuses more on the hard samples from the experts, which is shown to be an effective way to improve the performance in many arts [33, 84] . These results validate that the proposed strategy can extract professional knowledge from different experts, and enable the target model to become an expert in all source domains.Compared with the baseline ERM, our method involves only one additional hyper-parameter (i.e.𝛼 2 in Eq. 3) which is randomly selected in the range of [0.01, 10] in DomainBed. To evaluate the sensitiveness of LFME regarding this hyper-parameter, we conduct experiments in the PACS and TerraInc datasets by tuning it in a larger range. As seen in Tab. 6, LFME can obtain relatively better performance with either 𝛼 2 = 0.01 or 10, and it performs on par with ERM even when 𝛼 2 = 1000. These results indicate that our method is insensitive w.r.t the hyper-parameter. This is mainly because a very large 𝛼 2 will enforce the model to only learn from domain experts, given that 𝑞 𝐸 * is mostly aligned with the label 𝑦 (as seen in Fig. 2 (a) ), the model will perform similarly to another form of the baseline [37] in this case. 
conclusion: 
 This paper introduced a simple yet effective method for DG where the professional guidances of experts specialized in specific domains are leveraged. We achieve the guidance by a logit regularization term that enforces similarity between logits of the target model and probability of the corresponding expert. After training, the target model is expected to be an expert on all source domains, thus thriving in arbitrary test domains. Through deeper analysis, we reveal that the proposed strategy implicitly enables the target model to use more information for prediction and mine hard samples from the experts during training. By conducting experiments in related tasks, we show that our method is consistently beneficial to the baseline and performs favorably against existing arts. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work aims to improve the generalizability of a deep model. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not involve such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Benchmarks used in this data are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: In this section, we provide 1. Limitations and future works in Section A 2. Pseudocode of LFME in Section B.3. Explanations of why existing KD ideas perform inferior to our LFME in Section C. Although our method shows competitive results for generalization, there are certain occasions when it will encounter setbacks. First, LFME is not applicable for the setting where the domain labels are unavailable in the training data, such as those collected from the internet. Since the training of the experts requires data grouped by domain labels. Second, LFME cannot handle the situation when only one source domain is provided, preventing it from performing in a more difficult single-source generalization task. How to apply LFME to a more general setting will be our primary task in future works. Besides, as the designs and theoretical supports are built mainly for the classification task, finding a proper solution to extend them to the regression tasks is also a promising direction in potential future works.This section provides the Pytorch-style pseudocode of our method. As detailed in Alg. 1, the implementation of our method is embarrassingly simple, introducing only one hyper-parameter upon the baseline ERM.C Compare with Current KD Based on the Analysis in Sec. 4 As observed in Sec. 6.3, some existing KD ideas, perform inferior to our LFME. Based on our analysis, we find this is because existing ideas have difficulties achieving the beneficial effect introduced in Sec. 4.1. As detailed, our logit regularization ensures using more information for the target model by implicitly regularizing the probability in a much smaller range. This is hard to achieve by H (𝑞, 𝑞 𝐸 ) and ∥𝑞 -𝑞 𝐸 ∥ 2 , where their probability 𝑞 will be in a similar range as that in ERM because the soft label 𝑞 𝐸 is still within the range of [0, 1], enforcing similarity between 𝑞 and 𝑞 𝐸 will not significantly change its distribution. Moreover, this effect is also hard to achieve by ∥𝑧 -𝑧 𝐸 ∥ and ∥𝑞 -𝑧 𝐸 ∥ 2 , which do not provide a specific range regularization effect for the probability.This section provides more evidence to support the two analyses in Sec. 4 in the manuscript. The experiments are conducted on the widely-used PACS dataset and the difficult TerraInc dataset unless mentioned otherwise.For the baseline model, we infer that the cross-entropy loss alone will enforce the model to only learn discriminative features that are specific to the label category. Because there are no relevant features to interpret the non-label classes in ERM, the confidence for the target class will approach the maximum during training, and vice versa for other non-label classes. Differently, when the output probability has a smoother distribution, the model will also learn information that is shared with other classes to improve the corresponding non-label probabilities. Introducing more information during training will inevitably cause the phenomenon of low confidence for the label category, as probabilities for the non-label classes will increase (their summation increases from nearly 0 in Fig. 2 (a) to 0.35 in Fig. 2 (b) ). Note that low confidence does not mean low accuracy. Because the label category corresponds to the largest logit value in both terms, and both the discriminative and newly introduced information can be used to characterize the label information, exploring more information can also boost classification. This finding is further validated by the in-domain results in Tab. 5, where LFME is shown to be able to also improve the in-domain test results compared to ERM, indicating that low confidence in our case actually leads to higher accuracy.Visual examples to support the above analysis are shown in Fig. 4 . We observe that compared to the baseline ERM, our method can utilize more regions for prediction. Such as that in the first "horse" examples, our model utilizes both the head, tail, and body for prediction, while the ERM seems to focus only on the head region. These visual samples align with our analysis that compared to the baseline ERM, LFME tends to use more information for prediction. Figure 4 : Grad-CAM visualizations of samples from the unseen "cartoon" domain of the PACS benchmark, which is the most challenging domain for ERM and our method according to Tab. 3. Compared to the baseline ERM, highlight regions from our method contain more information related to the label category. These visualizations can further validate our analysis in Sec. 4 that with the proposed strategy, the target model can explore more information for prediction.Table 7 : Further evidences to support our deep analyses in Sec. 4. Evaluations are conducted on the widely-used PACS and difficult TerraInc datasets using settings from the DomainBed benchmark.Here ERM+ is the free lunch introduced in Sec. 6.1 that replaces 𝑞 𝐸 in Eq. ( 3) with the one-hot label; LFME_Guid denotes imposing the guidance from LFME to the ERM model by including L 𝑔𝑢𝑖𝑑 = ∥𝑞 -𝑞 𝐿𝐹 𝑀 𝐸 ∥ 2 where 𝑞 and 𝑞 𝐿𝐹 𝑀 𝐸 are probabilities from the ERM model and LFME; Self_Guid is the model that replaces probabilities from the experts in Eq. ( 3) with that from itself, i.e. L 𝑔𝑢𝑖𝑑 = ∥𝑧softmax(𝑧) ∥ 2 where softmax(𝑧) is followed with a detach operation; ERM+ w/ expt denotes explicitly focus more on the hard samples from the experts based on ERM+; ERM+ w/ self denotes explicitly focus more on the hard samples from the model itself on the basis of ERM+. All methods are with the same ResNet18 backbone and are examined for 60 trials in each unseen domain. (2) we use the output probability from the target model itself to replace that from the experts in Eq. ( 3), which reformulates L 𝑔𝑢𝑖𝑑 into ∥𝑧softmax(𝑧) ∥ 2 (softmax(𝑧) is followed with a detach operation in updating). According to our analyses in Sec. 4, this alternative (i.e. Self_Guid) should also boost generalization because it enforces the target model to learn more information and focus more on the hard sample from itself;Results are illustrated in Tab. 7. We observe that both the two variants LFME_Guid and Self_Guid are beneficial to the baseline models, leading ERM in all unseen domains. These results further validate our first analysis in Sec. 4, and we can conclude that when the basic discrimination is not compromised, either directly or indirectly enlarging 𝑞 𝑐 for ∀𝑐 ≠ * can help the target model learn more information, and improve generalization accordingly.As analyzed in Sec. 4, LFME not only encourages the target model to learn more information but also implicitly helps it to focus more on the hard samples from the experts. To examine if the classifications of hard samples from the experts are indeed improved, we plot the classification ratio R = 𝑝 * max( 𝑝) , where 𝑝 denotes the normalized probability: 𝑝 = 𝑝-min( 𝑝) max( 𝑝) -min( 𝑝) , from the target model on these hard samples. We use the ratio because it can quantify the correct predictions regarding the hard samples more precisely than the accuracy: the closer R approaches 1, the better the corresponding model performs on the sample. In this setting, 1/3 samples with larger losses in a training batch are considered hard samples. We compare R from LFME with that from ERM to highlight the difference. As can be observed in Fig. 5 (a), most R from LFME is larger than that from ERM over the iterations, denoting LFME is better at handling these hard samples than ERM. As a comparison, R from these two models almost overlap in the easy samples as shown in Fig. 5 (b ). These results validate our analysis that LFME implicitly helps the target model to focus more on the hard samples compared with ERM.Meanwhile, to validate if the hard samples from experts can indeed help generalization, we conduct experiments by explicitly putting more weights on the hard samples from the experts on the basis of the free launch ERM+ (i.e. ERM+w/ expt). Specifically, the target objective based on Eq. ( 9) is adjusted into: 𝑤L 𝑎𝑙𝑙 , where 𝑤 is the weight for the training samples and is determined based on the loss of the experts. Basically, the larger the corresponding loss from the experts, the larger the value of 𝑤 should be, and we use the strategy from [35] to implement this hard sample mining process. As can be observed in Tab. 7, mining hard samples from the experts can improve the base model of ERM+, especially in the difficult TerraInc dataset. These results validate that mining hard samples from experts is beneficial for generalization.We note in Tab. 7, the model Self_Guid performs inferior to LFME in both the two datasets, and the improvements are also marginal compared with ERM+ on account of variances. These results indicate the hard samples from the model itself may be less effective for generalization compared with that from the experts. To validate this hypothesis, we conduct experiments by explicitly focusing more on the hard samples from the model itself on the basis of ERM+ (i.e. ERM+ w/ self). The implementation is the same as that in ERM+ w/ expt.Results listed in the 6th row in Tab. 7 validate this hypothesis, where the hard samples from the model itself can hardly improve generalization compared with that from the experts. We presume the main reason is that besides the dominant specific domain information, hard samples from the experts also contain some out-of-the-domain information that makes it challenging for the corresponding expert. Examples are shown in Fig. 6 . We observe that compared with hard samples from the model itself (i.e. Fig. 6 (b )), hard samples from the experts (i.e. Fig. 6 (a) ) contain more ambiguous data located in the mixed region or the boundary of two domains than, indicating the experts may be more effective at revealing samples that with characteristics from different domains. By implicitly emphasizing these out-of-the-specific-domain samples, can the target model learn domain-agnostic features, thus improving generalization accordingly. Because the naive training strategies are fed with data from different domains at the same time, the out-of-the-specific-domain information is difficult to discover in their framework, explaining why hard samples from other strategies may not contribute. This is also the reason why the experts must be involved in the overall framework.Since our idea involves training multiple experts, one may connect it to the idea of mixture of experts (MoE) [86] . However, it is noteworthy that our LFME is not a simple implementation of existing techniques. The introduced logit regularization term is new and can be properly explained for its effectiveness. Some similar ideas that use MoE have been explored in other works for improving DG. For example, in DAELDG [90] , following a shared feature extractor, each domain corresponds to a specific classifier (i.e. expert), whose output is enforced to be similar to the average outputs from the non-expert classifiers. Different from our work, it uses the average outputs from different experts as the final result. In Meta-DMoE [89] , a meta-learning-based framework is employed to enforce Here the hard samples are specified by the experts: the 1/3 samples in a training batch with larger losses from the experts, and the easy samples are the leading 1/3 samples with smaller losses. The two models perform evenly well on the easy samples, while LFME obtains better results than ERM in the hard samples. the feature of the target model to be similar to the aggregated features from the experts, which is different from our logit regularization idea. In Sec. D.1 and 6.3, we implement the basic designs of these two ideas with the framework of LFME and show their ineffectiveness. Besides the comparison with Meta-DMoE in Tab. 1, in this section, we further compare LFME with them using their original settings in the PACS and OfficeHome datasets with the benchmark provided in [90] . We also compare with several other recent DG methods. Results in Tab. 8 show that LFME performs favorably against DAELDG, and outperforms Meta-DMoE, further validating the advantages of the logit regularization term. We also note LFME performs better than other arts with a different benchmark, demonstrating the effectiveness of the proposed approach.Since LFME requires learning different experts during training, inevitably using more parameters than existing methods, one may wonder if the effectiveness of LFME derives from using these extra parameters from the experts. To examine this idea, we conduct experiments by implementing some leading arts (i.e. CORAL, SD) and the baseline ERM with 𝑀 + 1 times of model size (where 𝑀 is the domain number). Specifically, the feature extractor for the larger method will contain 𝑀 + 1 branches, each with the same pretrained ResNet backbone. We concate the final outputs from the different branches and use it as input for a classifier to obtain the final result. Note that in this setting, a sample 81.0 ± 0.9 76.5 ± 0.9 94.6 ± 0.5 77.4 ± 0.2 82.4 ± 0.1 will go through 𝑀 + 1 times of forward passes for both training and inference, which is more than that of LFME design. Results are shown in Tab. 9. We note that compared to the results from the original models, when using the same pretrained knowledge, naively expanding model size cannot improve the performance. The reason may be that a well-pretrained small backbone can already saturate on limited training data (as shown in Table 5 , ERM can achieve more than 0.96 acc in the source domains), thus it is unnecessary to use more parameters in these datasets.Besides the different experts 𝐸 𝑖 where 𝑖 denotes the domain label, Dyn_Expt also uses an extra weighting network 𝑊 to estimate the labels of the training data. During the training stage, the experts are trained the same as that in LFME, and the weighting network is trained using the corresponding domain labels by minimizing H (𝑊 (𝑥 𝑖 ), 𝑖). During the test phase, Dyn_Expt dynamically combines the outputs from the experts with the corresponding domain probability: for ∀𝑥 ∈ D 𝑀+1 , the final result is 𝑀 𝑖 𝑊 (𝑥) 𝑖 𝐸 𝑖 (𝑥). We use the same backbone for both the experts and the weighting module in this experiment.This section presents the detailed results on the semantic segmentation task in Tab. 10, and detailed results in the unseen domains from the different unseen domains of DomainBed benchmark in Tab. 11, 12, 13 14, and 15.Table 10 : Evaluations on the semantic segmentation task with the metrics of Mean IoU (%) and per-class IoU (%). Source data is from the synthetic GTAV [63] and Synthia [64] datasets, and the target data is from the real-world Cityscapes [20] , BDD100K [83] , and the Mapillary [55] datasets. All models are with the same backbone: DeepLabV3+ with ResNet50, and results with † are from [19] . Justification: For all experiments, we use a Tesla V100 GPU with 32 GB memory, the actual running time is detailed in Tab. 11.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).𝑧 can also be regarded as concatenated results from different domains:L 𝑔𝑢𝑖𝑑 = 𝑖 L 𝑖 𝑔𝑢𝑖𝑑 = 𝑖 ∥𝑧 𝑖 -𝑞 𝐸 𝑖 ∥ 2 .We also observe plots of F and F ′ almost overlap in Fig.2 (e). This is because optimizing ∥𝑧 -𝑞 𝐸 * ∥ 2 leads to 𝑐 (𝑧 𝑐 -𝑞 𝐸 𝑐 ) = 0, enforcing 𝑐 𝑧 𝑐 ≈ 𝑞 𝐸 𝑐 = 1. Since there is no other constraint for 𝑐 𝑧 𝑐 (L 𝑐𝑙𝑎 does not impose a constraint on the summation of logits), we will have 𝑐 𝑧 𝑐 ≈ 1 and F ≈ F ′ in all cases, this is consistent with the observation in Fig.2 (d). 


Decision: The paper proposes a framework that leverages multiple expert models to enhance domain generalization through logit regularization. While reviewers commended the clarity of the writing and the empirical results showing improvements over baselines, several concerns were noted: the approach's novelty is limited, there is a lack of sufficient justification for key methodological choices, and the evaluation does not cover a diverse range of tasks and datasets. the authors' rebuttal provided additional experiments and clarifications and addressed most of the concerns. Therefore, the recommendation is to accept the paper.

Reviews: review: paper_summary: paper_summary: The computer-vision paper introduces a strategy of learning from multiple experts (LFME), which performs knowledge distillation from models specially trained on data from different domains. In particular, the experts are trained jointly with the target model, and a specific form of logit regularization is chosen for knowledge distillation due to empirical verification. The idea is relatively straightforward to understand with strong empirical results on image classification (by Domainbed) and semantic segmentation.
strength_weakness: strength_weakness: 1. The novelty is relatively limited. The proposed approach is very similar to Meta-DMoE with a few changes such as logit regularization and alternating updates, but these changes are not well justified (see below). The idea of utilizing multiple domains have also been explored by [1]. 
2. There is a lack of coherent argument to understand the proposed approach. In Section 4.1, the paper briefly discussed how the additional logit regularization enables learning more information. In addition, in Section 6.2, the proposed approach LFME is compared with Label Smoothing. However, the discussion is not very convincing. From the perspective of using information from other classes, I do not see the fundamental difference between logit regularization and Label Smoothing (LS). Both logit regularization and LS have hyperparameters to be tuned, so it might not be appropriate to claim its advantage as "not involve hand-crafted settings” (line 187) and criticize LS has “potential improper heuristic designs” (189). 
3. Moreover, in Section 4.2, the paper justifies the advantage of the proposed approach LFME by “mining hard samples from the experts”. It is a general statement that is true for low-confidence predictions from any model, so it does not constitute a strong argument. Intuitively, the in-domain samples are easier than out-of-domain samples for each expert, and the hard samples aggregated from all experts are not specifically representative of any subpopulations. The argument will be more comprehensive if the paper can show what are these hard samples and why it matters to mine the hard samples from the domain experts (and not an ensemble of experts or some random experts). 
4. In Section 6.5, the result for in-domain evaluation is only provided for one dataset. This is not sufficient to justify a strong claim that the target model is an expert on all source domains. 
5. Code is submitted but it is unclear how the proposed approach can be combined with SWAD. Does it apply to just the target model or also the domain experts? The hyperparameter configurations to obtain the SoTA results are also unclear for reproducibility. 

[1] Yao, Huaxiu, Xinyu Yang, Xinyi Pan, Shengchao Liu, Pang Wei Koh, and Chelsea Finn. "Leveraging domain relations for domain generalization." *arXiv preprint arXiv:2302.02609* (2023).
questions: questions: 1. In Algorithm 1, the target model and the experts are trained simultaneously, which is a more parallel and iterative scheme rather than a sequential one with two stages, i.e., training the experts first, then distilling the knowledge to the target model using logit regularization. Why is such a design choice made? Are there better justifications than just “for simplicity”? Please explain with more details. 
2. Each expert is generally weak as they are only trained using the data from their own domain. Is it necessary and how to ensure the quality of the experts?
3. In Appendix A, Limitations, the authors discussed that LFME cannot be applied to single-domain tasks. How does the number of domains affect the proposed approach? Will the algorithm benefit from having more domains with less data or more data in fewer domains?
limitations: limitations: The limitations has been discussed in Appendix A.
paper_summary: paper_summary: This paper addresses the problem of domain generalization from the perspective of  ''learning multiple experts''. In particular, they propose to train multiple experts specialized in different domains, whose output probabilities provide professional guidance by simply regularizing the logit of the target model. The proposed logit regularization provides effects of enabling the target model to harness more information, and mining hard samples from the experts during training. Experiments on standard DG benchmarks demonstrate the effectiveness of the proposed framework.
strength_weakness: strength_weakness: While adopting the idea of knowledge distillation (KD) and the concept of domain experts for improving DG is not new in the literature, it is appreciated that this method can be implemented in such simple form and be explained with extensive insights. Nevertheless, there are still a certain amount of claims in the paper that requires proper explanations or more justification:

- Performance reported in the original Meta-DMoE paper is higher than that in Table 1. For example, Meta-DMoE reports an average of 86.9 in PACS, surpassing the best result listed in the table. The authors should provide an explanation for this discrepancy. Moreover, I noticed that the training time comparisons between these two methods are provided, it would also be beneficial to highlight the test and training resource differences between these two methods in the related work section.

- The authors mention that their method introduces only one hyper-parameter, which is within a certain range. It would be beneficial to include an ablation study to explain why this specific range was chosen.

- It would be beneficial to evaluate this method with more real-world DG problem, such as using dataset from wilds benchmark.

- The method [1] with similar settings should be compared in Table 1. Additionally, some methods, such as MIRO, report improved performance when combined with SWAD. Including these results in Table 1 would provide a more comprehensive comparison.

- Minor issues. Formats of ”i.e.” in line 145, ”Tab.” in line 578, and ”Fig.” in line 581 are inconsistent with their usages in other places. The authors should standardize the format of similar abbreviations to maintain uniformity.

Reference: 

[1] Domain Generalization via Rationale Invariance, in ICCV 2023.
questions: questions: - This method needs to train additional expert per domain, what if there are large amount of domains for the training data, any potential solution for this occasion?

- Since multiple expert models are required when training the target model, how would this method be applied to the training process of large foundation models?

- The free lunch idea appears to be applicable beyond just the DG setting. How does it perform on ImageNet? Including such experiments could enhance the significance of this concept.
limitations: limitations: yes
paper_summary: paper_summary: This paper focuses on improving domain generalization by utilizing multiple experts. Particularly, a simple yet effective framework is proposed whereby a target (student) model is learned from multiple expert (teacher) models through logit regularization. After learning, the target model can grasp knowledge from multiple source domains and shows advantages in handling hard samples. The proposed approach demonstrates consistent improvements across different evaluation tasks, outperforming existing SOTAs.
strength_weakness: strength_weakness: 1: The contribution of the proposed approach is marginal. The performance of the proposed approach does not show much improvement compared to ERM or expert models (Table 5). 

2: The baseline models for comparison are not clearly introduced, which creates difficulties in understanding the Tables.

3: The comparison to the approaches that directly learn a foundation model using data from all source domains is not discussed. Hence, the benefits of the proposed approach, which have to use a set of expert models plus a target model, compared to the foundation model are not clear.
questions: questions: 1: What is the difference between the proposed approach and foundation models? Why not directly learn a model using all the source domains? How does the performance of the proposed approach compare to Segment Everything?

2: What’s the difference between the proposed approach and the baseline ERM? In lines 77-78, the authors mention ‘only one hyper-parameter’ without further illustration. The difference is clarified only until line 138. Statements near lines 77-78 need to be improved to avoid confusion. More importantly, it can be misleading to say “only one additional parameter upon ERM” since a target model is also involved.

3: How does the value q_*^E relate to the hardness of samples? What causes the sample to be ‘hard’? Low data frequence or image quality?  Without these clarified, the provided theoretical insights are not convincing.
limitations: limitations: No discussions on limitations and potential negative societal impact of the work are provided. 

One limitation is that the performance of the proposed approach can be upper bounded by the performance of the expert models. This limitation again raises the question of why not directly learn from all source domains?
paper_summary: paper_summary: This paper proposes a simple framework called LFME for learning from multiple experts in domain generalization. LFME introduces a logarithmic regularization term to enforce similarity between the target model and expert models, allowing the target model to acquire expertise from all source domains and perform well in any testing domain. Experimental results demonstrate that LFME outperforms existing techniques not only in classification and segmentation tasks but also reveals through in-depth analysis its ability to implicitly use more information for predictions and to discover difficult samples from experts, thereby enhancing generalization capability.
strength_weakness: strength_weakness: Regarding the proposed method, although large improvements over the baseline ERM are observed, their method involves higher training costs. I noticed the authors provide the training time comparisons with different methods in their supplementary material, which shows that their LFME requires 1.5 times more training time than ERM, and it is also one of the most time-consuming methods compared. While I agree with the authors that requiring more training time is inevitable for methods designed based on KD, the authors should at least list this as a major drawback in their limitation section.

As in recent times, Vision Transformers (ViTs) have demonstrated substantial improvements in the classification and semantic segmentation tasks. Although the authors conducted experiments using different ResNet models, they should also consider adopting a ViT backbone for their experiments. This could provide valuable insights into the performance of their method when applied to more advanced architectures, potentially highlighting further benefits or limitations. Including such experiments would enhance the comprehensiveness of the study and align it with current trends in the field.
questions: questions: In Section 4.2, the authors claim that their method improves DG by explicitly mining hard samples from the experts. However, Tables 3 and 6 show that specifically mining hard samples from the experts improves performance of ERM only in the TerraInc dataset, but not in the PACS dataset (with an average improvement of only 0.1 pp). Could the authors provide an explanation for this phenomenon?
limitations: limitations: Authors have adequately addressed the limitations.

Comments: title: Reply to your concerns
Reviewer Response
Response to further concerns
Reviewer Response
Thank you for the feedback
Reviewer Response
comment: We thank the reviewer for the prompt reply.

> Q: Is updating LFME mathematically equivalent to doing gradient descent using ERM followed by multiplying a negative value for gradient ascent? 

A: We want to clarify that the two gradient parts of $\mathcal{L}\_{all}$ w.r.t $z$ (e.g. $\partial \mathcal{L}\_{cla}$ and $\partial \mathcal{L}\_{guid}$) may better be interpreted as a whole in a updating step, as one updating step requires the combination of the two gradients for a joint classification task. 

According to your suggestion, if we specifically interpret the updating process by separately considering $\partial \mathcal{L}\_{cla}$ and $\partial \mathcal{L}\_{guid}$, we can see that in both terms, $z\_{\ast}$ correspond to largest labels. That is, $z\_{\ast}$ is encouraged to be close to $y\_{\ast}$ by minimizing $\mathcal{L}\_{cla}$ and close to $q\_{\ast}^E$ by minimizing $\mathcal{L}\_{guid}$. This means the updating process will maintain its discriminative ability, and the model will not forget the sample. The different gradient directions can be interpreted as $\mathcal{L}\_{cla}$ encourages $z\_{\ast}$ to become infinitely large, focusing on dominant features, while $\mathcal{L}\_{guid}$ optimizes $z\_{\ast}$ to the opposite direction to be close to $q\_{\ast}^E$ so that other information can also be involved. Therefore, $\partial \mathcal{L}\_{guid}$ should not be sees as tries to "unlearn" what has been learned by$\partial \mathcal{L}\_{cla}$. Rather, on the basis of not compromizing the target model's discriminative ability, they serve different purposes to encourage the model to focus on different types of features.

> Q: Is mining hard samples from experts counterintuitive?

A: The primary effect of knowledge distillation (KD)-based methods can be attributed to sample reweighting [b], where the weight of a training sample in the student model is related to the teacher's confidence. Although a different KD scheme is used, we apply the same rescaling analytical tool to demonstrate that one of the working mechanisms of the proposed KD form can still be seen as weighting different samples based on the teacher's confidence. Since different KD forms are utilized, the weighting strategies differ between the proposed LFME and the traditional KD approach where easier samples from teachers correspond to larger weights (comparisons between LFME and traditional KD ideas are presented in Sec. 6.3, where LFME performs better for the task of DG). Meanwhile, since hard samples from domain experts are more likely to be domain ambiguous data, enforcing the target model to learn these data more throughly can help it to explore more domain-invariant features, which are crucial for generalization [17]. In this regard, mining hard samples from domain experts is an intuitive motivation for improving DG.

> Q: Arguing from the the confidence perspective.

A: We thank the reviewer for the thoughtful suggestion. We want to clarify that the hard sample mining is aligned with the proposed confidence reduction perspective: in the given example, the target model will reduce confidence of $x_2$ more than that of $x_1$ even if they are with the same output logits, consistent with our explanation that hard samples from the experts will affect more on the target model's updating. Therefore, there may be no intrinsic difference between the hard sample mining and the suggested confidence reduction idea. Meanwhile, we provide empirical evidence in Sec. D.3 to show that the classification results of hard samples from corresponding experts are indeed improved in LFME, we thus argue the working mechanism from the hard sample mining perspective.

On the other hand, merely analyzing the confidence is similar to our initial analysis in Section 4.1, where the idea of smoothing the output probability of the target model is equivalent to making it less overconfident.

We sincerely thank you for your invaluable time and all thoughtful suggestions, which definetly help improve our paper. We wish these clarifications are helpful for easing your concerns.

[b] Born-Again Neural Networks, in ICML'18.
Thank you for the prompt clarifications. 

Unfortunately, I am still not convinced and couldn't agree with the counterintuitive rationale. 
> Specifically, we are not performing gradient descent using ERM followed by multiplying a negative value for gradient ascent.

I understand LFME is not doing this explicitly, but aren't they mathematically equivalent? If it doesn't align, then how does the analysis in Section 4.2 provide valid justifications? I think when we are trying to investigate the working mechanism of the proposed algorithm, the story needs to be coherent and not mislead the readers, especially when the analysis consists of a major section of the main paper. 

> Penalizing high-confidence outputs has been shown to be effective in improving a variety of learning tasks

As far as I understand, arguing from the confidence perspective might be more reasonable than hard sample mining.
We appreciate the reviewer's feedback and thoughtful response. We answer the additional questions below.

> Q: This is because originally we are doing gradient descent using ERM. When we multiply a negative value in front of it, it becomes gradient ascent instead.

A: We want to clarify that our only objetive is to minimize the loss $\mathcal{L}_{all}$ in Eq. (3) throughout the training process. It's important to emphasize that there is no gradient reversal operation in our framework; specifically, we are not performing gradient descent using ERM followed by multiplying a negative value for gradient ascent.

> Q: How is $\mathcal{F}$ calculated.

A: we want to clarify that we do not need to compute $\mathcal{F}$ during updating, $\mathcal{F}$ is used as an analytical tool to show how much the gradient magtitudes is rescaled w.r.t to the baseline model that without using the logit regularization term, by comparing ratio of gradients from the two models [64]. In our case, gradients of the two models (LFME and ERM) are $q_c - y_c + \alpha (z_c - q_c^E)$ (e.g. $\frac{\partial \mathcal{L}\_{all}}{\partial z_c}$) and $q_c - y_c$ (e.g. $\frac{\partial \mathcal{L}\_{cla}}{ \partial z_c}$), there gradient ratio is thus $\frac{\partial \mathcal{L}\_{all}}{\partial z_c} / \frac{\partial \mathcal{L}\_{cla}}{ \partial z_c}$, which leads to the computing of $\mathcal{F}$ and $\mathcal{F}'$ in Eq. (6) and (8).

> Q: In the given example, how the mining the "hard" sample from an expert achieved and its relation to the reduction of the logit value.

A: For the given example, where $z_1 = z_2$, it indicates that the model is with equal confidence w.r.t the two samples. Without experts involvements, these two samples will contribute equally to the target model's updating. However, when the expert is involved and $q_{\ast 1}^{E} > q_{\ast 2}^{E}$，Eq. (6) and (8) indicate $|\mathcal{F}_1| < |\mathcal{F}_2|$ and $|\mathcal{F}'_1| < |\mathcal{F}'_2|$: the gradient magtitude from $x_2$ is changing more significantly than that from $x_1$ both on the basis ERM, allowing $x_2$ to have a greater impact on the target model's update. The derivations imply that the expert can expose its hard samples to the target model, which is not applicable in ERM.   

From your point of view, the objetive in Eq. (3) implicitly encourages the target model to be less confident w.r.t $x_2$. We want to clarify that this confidence reduction does not conflict with mining $x_2$, because lower confidence results in a higher training loss, prompting the target model to explore the corresponding sample more thoroughly, where both the dominant features for classification and other information shared with other classes will be explored, with the dominant features being the primary focus (because both $\mathcal{L}\_{cla}$ and $\mathcal{L}\_{guid}$ in $\mathcal{L}\_{all}$ encourage the gt logit to be the largest, which also indicates that lower confidence in LFME does not equate to lower accuracy—a fact validated by our in-domain results). On the contrary, high confidence indicates that the model may overlook the sample after obtaining its dominant features, making it less important on the updating process. Moreover, penalizing high-confidence outputs has been shown to be effective in improving a variety of learning tasks [a], our objective can thus be viewed as specifically mining hard samples from the experts for penalization. From this perspective, in our case, penalizing confident can be integrated within our explanation of hard sample mining. 

We sincerely thank you for your invaluable time and the fruitful discussions. We are more than willing to provide any additional justifications or clarifications you may need.

[a] Regularizing Neural Networks by Penalizing Confident Output Distributions, in ICLR'17
Practically as the algorithm is working, I would like to believe that LFME is not forgetting all samples. However, the math and plot in Section 4.2 say otherwise. 

This is because originally we are doing gradient descent using ERM. When we multiply a negative value in front of it, it becomes gradient ascent instead. According to the plot, it's happening across most of the iteration steps, which is quite puzzling. I am not sure how is $\mathcal{F}$ calculated practically, but it might be helpful to clarify it. 

Moreover, if we just look at Eq. (4), if $q_{*1}^{E} > q_{*2}^{E}$, the direct interpretation is to reduce the logit value of $x_2$ more than $x_1$, so that the model should not be overconfident about it. I honestly couldn't see how this is mining the "hard" sample from an "unconfident" expert.
The author's rebuttal allayed some of my concerns, and I chose to raise my rating, considering that the paper still had some sparkle to it.
Thank you for the detailed responses. My previous concerns have been well addressed. After carefully reviewing the other reviewers' comments and the authors' replies, I believe the paper has no significant flaws, and therefore, I choose to maintain my score.
We sincerely thank the reviewer for the prompt reply!

We would like to clalrify that if two networks have different gradient directions, it does not mean one network is "forgetting" the samples compared to the other. The negative values of $\mathcal{F}$ and $\mathcal{F}'$ apply for all training samples, but this does not imply that LFME is forgetting all samples in contrast to ERM. Since hard samples from experts correspond to larger gradient magtitude for the target model, they will have more significant impacts on the target model's updating, which can shift the decision boundaries not aligned with easy samples that with small gradient magtitude, potentially causing these easy samples to be overlooked by the target instead. We thus say the target model can mine hard samples from the experts.

The argument in Sec. 4.2 can be better interpreted from the following example: assuming two samples $x_1$ and $x_2$ from the same class and domain, with the target model producing identical output logits for both: $z_1 = z_2$. If the corresponding expert has different levels of confidence regarding these samples, such that $q_{\ast 1}^{E} > q_{\ast 2}^{E}$, the analysis in Sec. 4.2 suggests that $x_2$ will have larger impacts on the updating of the target model than $x_1$, and the target model will focus more on $x_2$.

We thank again for the reviewer's time, and wish our clarification is helpful.
Thank you for the detailed responses and comprehensive experiments, which have addressed most of my concerns. 

However, I am still not convinced by the arguments on mining the hard samples and the analysis of why the proposed approach works. In Sec. 4.2, the rescaling factor $\mathcal{F}, \mathcal{F}’$ are shown to be negative in Figure 2(e). While it is true that the magnitude is larger for harder examples, if the gradient direction is reversed, then the model is no longer learning the sample as it's going the opposite of the original ERM gradient. I am not sure if we can still interpret this as "mining" the hard samples because they are somewhat being "forgotten". Please clarify.