Improving Domain Generalization with Domain Relations
Authors: Huaxiu Yao
Xinyu Yang
Xinyi Pan
Shengchao Liu
Pang Wei Koh
Chelsea Finn
Venue: ICLR 2024 spotlight (2024)
Keywords: Domain Generalization; Domain Relations; Distribution Shift
Primary area: general machine learning (i.e., none of the above)
TL;DR: enhancing domain generalization through meta-data-driven domain relations

Abstract: Distribution shift presents a significant challenge in machine learning, where models often underperform during the test stage when faced with a different distribution than the one they were trained on. In this paper, we focus on domain shifts, which occur when the model is applied to new domains that are different from the ones it was trained on, and propose a new approach called DG. Unlike previous approaches that aim to learn a single model that is domain invariant, DG leverages domain similarities based on domain metadata to learn domain-specific models. Concretely, DG learns a set of training-domain-specific functions during the training stage and reweights them based on domain relations during the test stage. These domain relations can be directly obtained and learned from domain metadata. Under mild assumptions, we theoretically prove that using domain relations to reweight training-domain-specific functions achieves stronger out-of-domain generalization compared to the conventional averaging approach. Empirically, we evaluate the effectiveness of DG using both toy and real-world datasets for tasks such as temperature regression, land use classification, and molecule-protein binding affinity prediction. Our results show that DG consistently outperforms state-of-the-art methods.

Published as a conference paper at ICLR 2024
IMPROVING DOMAIN GENERALIZATION WITH DOMAIN RELATIONS
"ang"*,Xiny" Xinyi Pan", Shengehao Liu Pang Wei Koh², Chelsea Finn' Stanford University, PUNC-Chapel Hill, SCMU, 'UCLA, CCalteh, University of Washington huaxiu@cs. unc. edu, xinyuya2@andrew cmu, edu, cbéinnêcs, stanford.
Huaxiu Yao Xinyu Yang",
ABSTRACT
Distribution shift presents a significant challenge in machine learning, where mod- els often underperform during the test stage when faced with different distribution than the one they were trained on. This paper focuses on domain shifts, which occur when the model is applied new domains that are different from the ones it was trained on, and propose a new approach called D³G. Unlike previous methods that aim to learn a single model that is domain invariant, D'G leverages domain similarities based on domain metadata to learn comainn-specitn models. Concretely, D'G learns set of training-domain- -oomain-spccffic functions during the training stage and reweights them based on domain relations during the test stage. These do- main relations can be directly obtained and learned from domain metadata. Under mild assumptions, training-donne dee--domiin--ee -omman--ppccfiic functions achieves stronger out-of-domain generalization compared to the conventional averaging approach. Empirically, effectiveness of D'G using real-world datasets for tasks such as temperature regres- sion, land use classification, and molecule-protein binding affinity prediction. Our results show that DOG consistently outperforms state-of-the-ar methods.
theoretically prove that using domain relations to reweight
evaluate the
INTRODUCTION
Distribution shift is a common problem in real-world applications (Gulrajani and Lopez-Pazi 2021 Koh et al. 2021a). When the test distribution differs from the training distribution, machine learning models often experience significant decline in performance. In this paper, specifically focus on addressing domain shifts, which arise when applying a trained model to new domains that differ from its training domains. An example of this is predicting how well a drug will bind to a specific target protein. In drug discovery, each protein is specific domain (Ji et al. 2022). and the binding task on each domain is expensive due the cost of lab experiments. Thus. an open challenge is to train robust model that can be generalized to novel protein domains, which is essential when searching for potential drug candidates that can bind to proteins associated with newly discovered diseases.
To address domain shifts, prior domain generalization approaches mainly learn a single model that is domain invariant (Arjovsky et al., 2019 Krueger al.) 2021a Li et al.) [21185] Sun and Saenko 2016) Yao et al. 2022b), and differ in techniques they use to encourage invariance. These methods have shown promise, but there remains significant room for improvement under real-world domain shifts such those in the WILDS benchmark (Koh et al.) 202 Unlike learning single domain invariantt model, we posit that models may perform better if they were specialized to a given domain. The advantage of learning multiple domain-specific models is that traditional domain generalization methods assume predictions are based solely on "causal" or general features. However, in practical scenarios, different domains can exhibit strong correlations with non-general features, and domain- specific models can exploit these features to make more accurate predictions. While there are clearly some possible benefits learning domain-specifie models, it remains unclear how construct domain-speciff model for a new domain seen at test time, without any training data for that domain.
To resolve this challenge, we propose a novel approach called D'G learn set of diverse, training domain-speciin functions during the training stage, where each function corresponds to a single
"qqual contribution. Work was done during Xinyu Yang's remote internship Stanford.

---

Published as a conference paper at ICLR 2024
domain. For each test domain, D'G leverages the domain relations weight these training domain- specific functions and perform inference. Our approach is based on two main hypotheses: firstly, similar domains exhibit similar predictive functions, and secondly, the test domain shares sufficient similarities with some of the training domains. By capitalizing on these hypotheses, we can develop robust model for each test domain that incorporates information about its relation with the training domains. These domain relations are derived from domain meta-data, such as protein-proteio interactions or geographical proximity, in various applications. Additionally, D*G incorporates consistency regularizer that utilizes training domain relations to enhance the training of domain- specific predictors, especially for insuffiiintt domains.
Through our theoretical amalysis under mild assumptions, we demonstrate that D'G achieves superior out-of-domain generalization by leveraging domain relations to reweight training domain-specific functions, surpassing the performance of traditional averaging methods. To further validate our findings, we conduct comprehensive empirical evaluations of DCG on diverse datasets encompassing both synthetic and real-world scenarios with natural domain shifts. The results unequivocally establish the superiority of D'G over best prior method, exhibiting an average improvement 10.6%.
2 PRELIMINARIES
Diitr--Diitriittion Generalization. In this paper, we consider the problem of predicting the label y E y based on the input feature x E X. Given training data distributed according p we train a model parameterized by 0 E θ using a loss function f. Traditional empirical risk minimization (ERM) optimizes the following objective:
arg min Par (e(fo(z). v)]- BEB
(I)
The trained model is evaluated on a test set from a test distribution pts. When distribution shift occurs, the training and test distributions are different, i.e., per + pts.
Concretely, following Koh et al. (2)21b), we consider a setting in which the overall data distribution is drawn from set of domains D {1, ........ D), where each domain d D is associated with a domain- specific data distribution Pa over set (X, d) = {(3r4, Vr. d) d The training distribution and test distribution are both considered to be mixture distributions of the domains, i... per EdED r'd Pa ELED r Pa. respectively, where r and r denote the mixture probabilities in the training set and test set, respectively. We also define the training domains and test domains as Der = {d E Dir > 0} and Dd {d E Dlr > 0}. respectively. In this paper, we consider domain shifts, where the test domains are disjoint from the training domains, i.e., Der n Dts 0. In addition, the domain of training and test datapoints are available.
and pts
Domain Relations and Domain Meta-Data. In this study, our objective is to address domain shift by harnessing the power of domain relations, which encapsulate the similarity or relatedness between different domains. To illustrate this concept, focus on the protein-ligand binding affinity prediction task, where each protein is treated as an individual domain. Domains are considered related if they exhibit similar protein sequences or belong to the same protein family. To formalize these domain relations, we introduce an undirected domain similarity matrix denoted as A = {a:z } where each element dij quantifies the strength of the relationship between domains and j. In this paper, derive the domain relations by leveraging domain meta-data M (m,) which depict the distinctive properties of each domain.
3 LEVERAGING DOMAIN RELATIONS FOR OUT-OF-DOMAIN GENERALIZATION
We now describe the proposed method - DåG (leveraging domain distances for out-of-domain generalization). The goal of D is to improve out-of-domain generalization by constructing domain- specific models. During the training phase, we employ a multi-heade network architecture where each head corresponds to a specific training domain (Figure (a)). This allows us learn domain- specific functions and capture the nuances of each domain. To address the challenge of limited training data in certain domains, introduce consistency loss that aids in training (Figure T()).
2

---

Published as a conference paper at ICLR 2024
(sl: Model Architacture
(c): Training Stage Teaining
[d]: Tast Stage
Triining domein
Training domain
Shared feature cxtraztor
Shared Festure extractor
Head (())
Head p(nl))
(b): Relation Extraction
f
comaii
(alllg)
Expr Rajf (x)())
9e Resf
(x))
Rgd
(xg)
Learmed mildion Ry
Supervised loss
Constatoncy loes
Relation-sware weighted inferencee
Metwcata
(00,)))))) < 00.1)
An illustration of D¹G. (a) The multi-headed architecture of D³G, where each training domain is associated with a single head for prediction. (b) The relation extraction module, where fixed relations are extracted from domain meta-data and refined through learning from the same meta-data. (c) The training stage of D¹G, where x represents a single example from domain d. and the loss is composed of both a supervised loss and a consistency loss. (d) The test stage, where the weighting of all training domain-specift functions is used to perform inference for each test example.
Figure 1:
During the testing phase, we construct test domain-specific models for inference by reweighing the training domain-specific models. This reweighting process takes into account the similarity between the training and test domains, allowing to adapt the model the specific characteristics of the test domain (Figure (d). To establish domain relations, we extract information directly from domain meta-data and refine the relationships through meta-data-driven learning (Figure In the following sections, we delve into the details of the training and inference processes, elucidating how we construct domain-specific models and obtain domain relations.
3.1 BUILDING DOMAIN-SPECIFIC MODELS
In this section, we present the details of D'G for learning a collection of domain-speciff functions during the training stage and leveraging these functions for relational inference during the test stage.
Training Stage. During the training phase, our approach utilizes multi-headed neural network architecture comprising N heads, where N denotes the number of training domains. Given an input datapoint (x, y) from domain d, we denote the prediction made by the d-th head as f(()()) = (d)) (e(x)), where (-) represents the domain-specific head for domain d, and e(-) represents the feature extractor. Our objective is minimize the predictive risk for each datapoint when using the corresponding head, ensuring accurate predictions within each domain. This is
achieved by minimizing the following loss function:
Cyredt EdeDer E P, (() (x).
(2)
In certain scenarios, some training domains may contain limited data compared the overall training set, posing difficulties in training domain-specifie predictors. To address this challenge, we leverage the assumption that similar domains tend to have similar predictive functions. Building upon this assumption, introduce a relation- consistency regularizer. For each example (x, y) within training domain d, the regularizer incorporates domain relations to weigh the predictions generated by all training predictors, except the corresponding predictor f((), The formulation of the relation-aware consistency loss is as follows:
Σ
Nrr I-lad Odj (x)
Cret = E B
B(
(3)
Ner Ek adk
where a,j is defined as the strength of the relation between domain d and j. This loss encourages the groundtruth to be consistent with the weighted average prediction obtained from all other training predictors, using the domain relations to weight their contributions. By doing so, the regularizer encourages the model to: (1) rely more predictions made by similar domains, and less predictions made by dissimilar domains; (2) strengthen the relations between predictors and help training predictors for domains with insufficient data.
3

---

Published as a conference paper at ICLR 2024
To incorporate the consistency loss into our training process, add it the predictive loss in equa- tion and obtain the final loss as L = Lpred + Ahrel, where À balances these two terms.
Test Stage. During the testing phase, D*G constructs test domain-specific models based on the same assumption that similar domains have similar predictive functions. Concretely, weight all training domain-specific functions and perform inference for each test domain by weighting the predictions from the corresponding prediction heads. Specifically, for each test datapoint drawn from the test distribution Pt. D'G makes a prediction as follows:
N d= (x)
(4)
Dk Akt
where adt represents the strength of the relation between the test domain t and the training domain d. According to equation for each test domain, training domains with stronger relations play more important role in prediction. This allows D³G to provide more accurate predictions by leveraging the knowledge from related domains.
3.2 EXTRACTING AND REFINING DOMAIN RELATIONS
We then discuss how to obtain the pairwise similarity matrix A {a:)): (iji between different domains. Here, domain relations are derived from domain meta-data. For example, in drug-target binding affinity prediction task, where each protein is treated as domain, we interaction network to model the relations between different proteins. Another scenario is, if we aim predict the land use category using satelite images (Koh et 2021b), and each country is treated as domain, we can use geographical proximity to model the relations among countries. The relation between domains and that is directly collected from domain meta-data defined as a Thus, the relation between domains i and is either relation graph in domain meta-data, or the pairwise similarity calculated from each domain's 's meta-data. define it as ofz-
a protein-proteie
One potential issue with directly collecting domain relations from fixed domain meta-data is that these fixed relations may not fully reflect accurate application-specife domain relations. For example, geographical proximity can be used in any applications with spatial domain shifts, but it is hard to pre-define how strongly two nearby regions are related for different applications. To address this issue and refine the fixed relations, we propose learning the domain relations from domain meta-data using a similarity metric function. Specifically, given domain meta-data m, and m, of domains i and j. we use two layer neural network g learn the corresponding domain representations g(m:) Chen and (m,)). Following Chen et al. (2020), we compute the similarity between domains and j with multi-headed similarity layer, which is formulated as follows:
dy =
Σ cws O g(m,), w. O g(m,)),
(5)
where @ denotes the Hadamard product and R is the number of heads. The collection of learnable weight vectors [w, JR has the same dimension the domain representation g(m;) and is used to highlight different dimensions of the vectors.
We weighted sum combine fixed and learned relations. Specifically, define the relation between domains and j is defined as follows:
asy = Baßy + (1 101-99 9)oo,
(6)
where 0 < 8 S 1 is hyperparameter that controls the importance of both kinds of relations. By tuning B, we can balance the contribution of the fixed and learned relations to the relation between domains. The final domain relations are used in the consistency regularization and testing stage. To summary, the pscudocodes of training and testing stages of D'G is detailed in Alg.
4 THEORETICAL ANALYSIS
In this section, we theoretically explore the underlying reasons why utilizing domain relations that are derived from domain meta-data can enhance the generalization capability to new domains. In our theoretical analysis, for an input datapoint (x, y) from domain d, we rearrange the predictive function (d (e(x)) 6, where (.)) and e(-) represent
presented in equation 2 as y f(d)(z) +
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Training and Test Procedure of D*G
Require: Training and test data, relation combining coefficient 3, loss balanced coefficient X, meta-data {ma)a= of all domains, learning rate y
1: Training stage
2: Initialize all Icarnable parameters
3: Extract fixed relations {als J=.==1
4: while not converge do
5: Compute learned relations {aly A
(igh) and obtain the final domain relations by equation 6
6: for each example (x. y. d) do
7: Calculated supervised loss Lurea by equation
8: Computed consistency loss Leet by equation using domain relations.
9: Update Icarnablc parameters with learning rate Y.
10: 1 Test stage
11: for each test domain do
12: Obtain the relations between the test domain and training domains {aae I N
13: for each example (I. t) do
14: Computed the prediction by equation 4
the head of domain and feature extractor, respectively. é is noise term which is assumed to be Sab-Gaussian with mean of 0 and variance of o².
During the testing process, we adopt the assumption that the outcome prediction function (()(z) for the test domain can be estimated using the following equation:
(z) (000 (e(x)), where; h EET (t
(7)
ΣM
Nrr
Oke
Here, h0 represents the learned head for the training domain é. In the case where the denominator in equation is equal to 0, we define 00) 0.
To facilitate our theoretical analysis, we make the following assumptions: (1) For each domain d, the
domain representation (dd) derived from the domain meta-data m. (i.e., Z(d) = g g(m)) is assumed to be uniformly distributed on [0, 1l". Furthermore, it is assumed that the domain relations accurately capture the similarity between domains. Specifically, there exists a universal constant G such that for all j.j E D, we have [[/(()) - 11 S G- G 1 (2) The relation az between domains i and is determined by the distance between their respective domain representations and bandwidth B, defined as ag ll | B); (3) For each training domain d, (d) is well-learned such that [E( (e(x)) (0 (e(x))) = ( na the function class H. Based on these assumptions, we then have the following theorem.
) ). where C(H) is the Rademacher complexity of
Theorem 4.1. Suppose we have the number of examples na Z n for all training domains d E D'r, where n is defined as the smallest number of examples across all domains. the loss function &. Lipschitz with respect to the first argument, then for the test domain t, the excess risk satisfies
C(H)/n B
B P,(())) (x), v) - Ei (e(_))) v))
(8)
The theorem above implies that by considering domain relations bridge the gap between training and test domains, the more training tasks we have, the smaller the excess risk will be. The detailed proofs are Appendix(A.1
we now present proposition that highlights the importance of obtaining a good relation matrix in enhancing out-of-domain generalization. Specifically, we compare our method with the traditional approach where all training domains are treated equally, and the similarity matrix is defined as À = {ā., ).= with each ã, = 1. We compare the well-defined similarity matrix and ill-defined Á the following proposition:
Building upon the results derived in Theorem 4.
Proposition 4.2. Under the same conditions as Theorem 4. suppose all āij = 1 and consider the function class H. E (h: (] - 11 & G (| Z for j.j E D). Define the excess risk with similarity matrix A by R A) = F(e. (x). v: A)] R(aw) (x), or A)), we have
inf sup (FO .A < inf sup Ra (a()))),
(9)
f00 AEA
(e) MCR
5

---

Published as a conference paper at ICLR 2024
The proposition presented above indicates that by leveraging accurate domain relations, achieve superior generalization performance compared to the approach of treating all training domains equally. The detailed proof of this proposition can be found in Appendix A.2 A.2
EXPERIMENTS
In this section, we conduct a series of experiments to evaluate the effectiveness of D³G. Here, we compare D'G with different learning strategies and categories including (i) ERM (Vapnik)
1999), (ii) distributionally robust optimization: GroupDRO (Sagawa et al., (020), (iii) invariant
learning: IRM (Arjovsky et al. 2019). IB-IRM (Ahuja et al., 202 IB-ERM (Ahuja et al., 2021b)
V-REx Krueger et al.
2021a). DANN (Ganin et al. (206bb) CORAL (Sun and Saenko, (0166).
MMD ct 201 RSC (Huang et al. 2020p. CAD (Ruan et al. 2022). SelfReg (Kim et al.
2021 Mixup (Xu et al. 2020 LISA (ao et al. 2022b), MAT (Wang et al. 2022b), (iv) domain-
specific learning: AdaGraph (Mancini et al. 2019). RaMoE (Bui et al. 2021) ) mDSDI (Bui et al.
2021) AFFAR (Qin al. (022), GRDA (Xu et al. 2022). DRM (Zhang al. 2022b). LLE
et al. )023), DDN (Zhang et al. 2023). TRO (Qiao and Peng 2023 Here, methods in categories
(i), (1), (ii) learn universal model for all domains. Additionally, for fair comparison, we
incorporate domain meta-data as features for all baselines during the training and test stages. All
hyperparameters are selected via cross-validation. Detailed setups and baseline descriptions are provided in Appendix D
5.1 ILLUSTRATIVE TOY TASK
Dataset Descriptions. Following Xu al. (0022), use the DG-15 dataset, synthetic binary clas- sification dataset with 15 domains. In each domain d, a two-dimensional key point zg = (Id,1 2d,2) is randomly selected in the two-dimensional space, and the domain meta-data is represented by the angle of the point (i.e., arctan (242)) )). 50 positive and 50 negative datapoints are generated from two Gaussian distributions, N (a I) and and (N(xx,, I) respectively. In DG-15, we construct the fixed relations between domain i and j as the angle difference between key points 2; and Ty, i.e., (5.2 The number of training, validation, and test domains are all set PP,
a arctan (سي2) arctan
as 5. We visualize the training and test data in Figure 2a and 2b
Results and Analysis. The performance of DCG on the DG-15 dataset is in Figure 2 The results highlight several important findings. Firstly, it is observed that learning a single model fails to adequately capture the domain-specifer information, resulting in suboptimal performance. To gain further insights into the performance improvements, Figures and 2d depict the predictions from the strongest single model learning method (GroupDRO) and D respectively. GroupDRO lears a nearly linear decision boundary that overfits the training domains and fails to generalize on the shifted test domains. In contrast, DCG effectively leverages domain meta-data, resulting in robust generalization to most test domains, with the exception of those without nearby training domains. Secondly, when compared domain-specific learning approaches such as LLE, DDN, and TRO. D'G demonstrates superior performance. This improvement can be attributed to D'G's enhanced ability to capture and utilize domain relations, enabling it to achieve stronger generalization capabilities.
5.2 REAL WORLD DOMAIN SHIFTS
Datasets Descriptions. In this subsection, we briefly describe three datasets with natural distribution shifts and Appendix E provides additional details.
TPT-4. TPT-48 is a weather prediction dataset, aiming to forecast the next 6 months' temperature based on the previous months' temperature. Each state is treated as a domain, and the domain meta-data is defined as the geographical location. Following Xu et al. (2022).
consider two dataset splits: I. N (24) + S (24): generalizing from the 24 states in the north to the 24 states in the south; II. E (24) > W (24): generalizing from the 24 states in the east to the 24 states in the west. FMoW. The FMoW task
predict the building or land use category based on satellite images. For each region, we use geographical information as domain meta-data. We first evaluate DºG on spatial domain shifts by proposing a subset of FMoW called F-ia.. including 18 countries from Asia. Then, we study the problem on the full FMoW dataset from the WILDS benchmark (Koh) et al. 202Tb) (FMoW- W- M-Mo--WWDDS. WILDS), taking into account shift over time and regions.

---

Published as a conference paper at ICLR 2024
Tustire
Tuilie Sarzi
Parkis Negaha
Nqaate
Nagaltx
(a) Training Distribution
(b) Test Distribution
(c) GroupDRO
(d) D²G
Model
ERM
GroupDRO
IRM
IB-IRM
IB-ERM
V-REx
DANN
CORAL
Accuracy
44.0%
47.7%
43.9%
45.4%
43.1%
44.0%
43.1%
43.5%
Model
MMD
RSC
CAD
SelfReg
Mixup
LISA
MAT
RaMoE
Accuracy
41.3%
58.2%
43.3%
40.7%
41.3%
47.4%
39.6%
53.7%
Model
mDSDI
AFFAR
GRDDA
DRM
L.EE
DDN
TRO
D (ours)
Accuracy
45.2%
58.2%
59.5%
61.4%
58.8%
66.2%
56.3%
77.5 %
Figure 2: Results domain shifts on toy task (DG-15). Figures (a) and (b) illustrate the training and test distributions, where datapoints in cireles with the same color originate from the same domain. Figures (c) and (d) show the predicted distribution of the strongest single model method (GroupDRO) and D*G. Bottom Table reports averaged accuracy
all test domains (see full table with standard deviation in Appendix F. We bold the best results and underline the second best results.
In drug discovery, we focus on molecule-protes prootenn binding affinity prediction. The ChEMBL-STRING (Liu al. 2022) dataset provides both the binding affinity score and the corresponding domain relation. Follow Liu et al. (2022) and treat proteins and pairwise relations nodes and edges in the relation graph, respectively. The protein-prote
CHEMBL-STRING:
relations and protein structures are treated as domain meta-data. Follow Liu et al. (2022), we evaluate our method using subsets named PPI and 101
Results. We present the results of DCG and other methods in Table The evaluation metrics utilized in this study were selected based on the original papers that introduced these datasets (see results with more metrics in Appendix G). The findings indicate that most invariant learning approaches (e.g., IRM, CORAL) demonstrate inconsistent performance in comparison to the standard ERM. While these methods perform well on certain datasets, they underperform on others. Furthermore, even when employing domain-specific learning approaches such LLE, DDN, and TRO, these methods exhibit inferior performance compared to learning universal model. These outcomes suggest that these methods struggle to effectively learn accurate domain relations, even when provided with domain meta-data (more analysis in Appendix G.3). In contrast, D¹G, which constructs domain-specine models, achieves the best performance by accurately capturing domain relations.
5.3 ABLATION STUDY OF D²G
In this section, provide ablation studies datasets with natural domain shifts to understand where the performance gains of D*G come from.
Diee Auseer
D'u
33%
Does consistency regularization improve per- formance? We analyze the impact of domain- aware consistency regularization. In Figure we present the results on FMoW and ChEMBL- STRING of introducing consistency regulariza- tion the setting where only fixed relations are used. According the results, observe better performance when introducing consistency reg- ularization, indicating its elfectiveness in learning domain-specific models by strengthening the correlations between the domain-specife functions.
19%
Telo
151.50
Detail
(a) FMoW
(b) STIMIG
Figure 3: Performance comparison w.r.t. consis- tency regularization. Only fixed relations are used.
How do domain relations benefit performance? Our theoretical analysis shows that utilizing appropriate domain relations enhance performance compared simply averaging predictions from all domainn-spccif functions. To test this, we conducted analysis in FMoW and ChEMBL- STRING, comparing the following variants of relations: (1) no relations used; (2) fixed relations only;

---

Published as a conference paper at ICLR 2024
Table 1: Performance comparison between D'G and other baselines. Here, all baselines use domain meta-data features. The discrepancy in performance between results and those reported on the leaderboard for FMoW-WILDS because we incorporate domain meta-date as features for all baselines. We bold the best results and underline the second best results.
TPT-48 (MSE N (24) - S (24) (24) W (24)
FMoW (Worst Acc. FMoW-WILDS
CHEMBL-STRING (ROC-AUC 1)
FMoW-Asin
100
Region Shift
Region Shift
Region Shift
Region-Time Shift
Protein Shift
Protein Shift
ERM
0.445 = 0.029
0.328 + 0.033
26.05 +: 3.84%
34.87 = 0.41%
74.11 0.35%
71.91 + 0.24%
GroupDRO
0.413 0.045
0.434 = 0.082
26.24 + 188%%
31.16 + 2.12%
73.98 0.25%
71.55 + 0.59%
IRM
0.429 = 0.043
0.262 + 0.034
25.02 + 2.38%
32.54 + 1.92%
52.71 + 0.50%
51.73 + 1.54%
IB-RRM
0.416 = 0.009
0,272 = 0,026
26.30 +: 1.51%
34.94 + 1.38%
52.12 + 0.91%
52.33 + 1.06%
IB-ERM
0.458- = 0.032
0.273 = 0.030
26.78 + 1.34%
35.52 = 0.79%
74.69 + 0.14%
73.32 + 0.21%
V-REx
0.412 = 0.042
0,343 = 0.021
26.63 + 0.93%
37.64 + 0.92%
71.46 1.47%
69.37 + 0.85%
DANN
0.394 = 0.019
0.515 + 0.156
25.62 +: 1.59%
33.78 - 1.55%
73.49 0.45%
72.22 : 0.10%
CORAL
0,401 0.022
0.283 0.048
25.87 1.97%
36.53 0.15%
75.42 0.15%
73.10 0.14%
MMD
0.409 = 0.067
0.279 = 0.026
25.06 + 2.19%
35.48 = 1.81%
75.11 = 0.27%
73.30 + 0.50%
RSC
0.421 = 0.040 n/a
0,330 + 0.068
25.73 1 0.70%
34.59 + 0.42%
74,83 + 0.68%
72.47 1 0.38%
CAD
n/a
26.13 + 1.82%
35.17 = 1.73%
75.17 0.64%
72.92 + 0.39%
SelfReg
n/a
n/a
24.81 1 1.77%
37.33 1 0.87%
75.42 1 0.42%
72.63 1 0.71%
Mixup
0.574 + 0.030
0.357 1 0.011
26.99 1: 1.27%
35.67 1 0.53%
74,40 1: 0.54%
71.31 1: 1.06%
LISA
0.467 0.032
0.345 + 0.014
26.05 + 2.09%
34.59 1.28%
74.30 0.59%
71,45 + 0,44%
MAT
0.423 1 0.027
0.291 1 0.024
25.92 1: 2.83%
35.07 1 0.84%
74.73 11 0.30%
72.07 1: 0.81%
AdaGraph
n/a
n/a 0.311 = 0.060
25.91 + 0.59%
35.42 + 0.55%
74.02 0.42%
72.10 0.06%
RaMoE
0.372 = 0.035
26.65 + 0.46%
36.51 + 0.71%
74.99 0999:00 0.22%
71,48 + 0.49%
mDSDI
0.445 = 0.027
0,315 = 0.089
25.54 + 0.46%
36.35 = 0.45%
13.09:0.11
71.23 + 0.69%
ADDAR
0.403 = 0.061
0.287 + 0.040
25.87 + 1.01%
35.77 + 0.70%
74.55 + 0.54%
71.93 + 0.33%
GRDA
0.373 = 0.040
0.355 + 0.068
26.57 + 0.70%
34.41 + 0.42%
75.01 + 0.68%
73.57 + 0.38%
DRM
0.571 = 0.038
0.557 = 0.027
25.22 + 2.33%
36.39 = 0.76%
74.34 + 0.48%
72.41 + 0.76%
LLE
0.603 0.041
0.467 = 0.047
26.37 + 1.19%
35.83 1.00%
74.01 0.63%
71.68 0.61%
DDN
0.537 0.024
0.601 = 0.038
26.77 + 1.72%
35.13 0.62%
75.17 0.61%
72.71 + 0.59%
TRO
0.371 = 0.054
0.281 = 0.066
26.87 + 1.26%
37.48 - 0.55%
74.85 = 0.27%
72.49 + 0.36%
D²G (ours)
0.342 0.019
0,236 + 0.063
28.12 + 0.28%
39.47 + 0.57%
78.67 0.16%
77.24 + 0.30%
(3) learned relations only; and (4) both fixed and learned relations. Our results, presented in Table 3 (Full results: Appendix HH.).. first indicate that using fixed relations outperforms averaging predictions, which confirms our theoretical findings and highlights the importance of using appropriate relations. However, only using learned relations resulted a performance that is worse than using no relations at all, indicating that it is challenging to learn relations without any informative signals (e.g.. fixed relations). Finally, combining learned and fixed relations results in the best performance, highlighting the importance of using learned relations to find more accurate relations for each problem.
5.4 COMPARISON OF DOG WITH DOMAIN-SPECIFIC FINE-TUNING
As stated in the introduction, a simple way to create domain-specific model is by fine-tuning a generic model trained by empirical risk minimization (ERM) on reweighted training data using domain relations. In this section, we compare our proposed model D'G with this approach (referred to as RW-FT) and present the results in Table We also include the performance of the strongest baseline (CORAL) for comparison. The results show that RW-FT outperforms ERM and CORAL, further confirming the effectiveness of using domain distances to improve -iitr--iistiittto generalization. Additionally, D'G performs better than RW-FT. This may be due to the fact that using separate models for each training domain allows for more effective capture of domain-spectie information.
5.5 ANALYSIS OF RELATION REFINEMENT
In this section, we conduct a qualitative analysis to determine if the relations learned can reflect application-specific information and improve the fixed relations extracted from domain meta-dar -ata. Specifically, we select three countries Turrey, Syria, and Saudi Arabia from FMoW-Asia and visualize the fixed relations and learned relations among them in Figure Additionally, we visualize one multi-unit residential area from each of the three countries. We observe that although Turkey is geographically close to other two countries in the Middle East (as shown by the fixed relations), its architecture style is influenced by Europe. Therefore, the learned relations refine the fixed relations and weaken the distances between Turkey and Saudi Arabia and Syria.
8

---

Published as a conference paper at ICLR 2024
Table 2: Comparison between D'G with domain-
Turkey
Syria
Saudi Arabia
specific fine-tume aheettnnng. Full results: Appendix H.2
Model
FMoW (Worst Acc. 1)
ChEMBL (AUC f)
Asia
WILDS
50
PPI 100
ERM
26.05%
34.87%
74.11%
71.91%
CORAL
25.87%
36.53%
75.42%
70%
RW-FT
27.03%
36.39%
76.31%
74.30%
Saudi Arabia
Saudi Arabia
DºG
28.12%
39.47%
78.67%
77.24%
0.20
0.25
0.07
0.20
Table 3: Comparison using different relations. The results on FMoW and ChEMBL-STRING are reported. When no relations are used, we take the average of predictions across all domains.
Turkey
0.20
Syria
Turkey
0.04
Syria
Fixed Relations
Learned Relations
Fixed
Learndd
FMoW (Worst t>
ChEMBL (AUC.1)
relations relations
Asia
WILDS
PPl
PP1
Figure 4: Analysis relation learning. Top
26.93%
35.32%
76.17%
73.38
figures show the multi-unit residential areas of
27.43%
39.37%
77.66%
76.59%
Turkey, Syria, and Saudi Arabia. Bottom fig- ures illustrate both fixed relations and learned relations.
21.18%
36.41%
77.09%
75.57%
28.12%
39.47%
78.67%
77.24%
RELATED WORK
In this section, discuss the related work from the following two categories: -uttof-diittrbbttion generalization, ensemble learning, and test-time adaptation (Appendix B).
Out-of-distribution Generalization. To improve outto--distribuution generalization, the first line of works aligns representations across domains to learn invariant representations by (1) minimizing the
divergence of feature distributions (Long et al. 2015 Tzeng et al. 2014 Ganin et al. 2016a [i et al.
2018b; (2) generating more domains and enhancing the consistency among representations (Shu
et al. 2021; Wang et al. [0200] Xu et al. 2020; Yan et al. 2020 Yue et al. 2019 Zhou et al. (022). Another line of works aims to find predictor that is invariant across domains by imposing
explicit regularizer (Arjovsky et al. 2019] [ a et al. 2021a; Guo et al. 2021 Khezeli et al. 2021
Koyama and Yamaguchi| 2020 Krueger et al. 202 Koyama and Yamaguchi 2020) selectively
augmenting more examples (Yao et al. 2022b a Gao et al.). Recent studies explored the concept of learning domain-specific models (Li et al., 2022] Pagliardini al.) 2022 Zhang et al., 2023 2022b). However, in comparison to these approaches, D'G stands out by leveraging domain meta- data, incorporating consistency regularization and domain relation refinement techniques, more effectively capture domain relations. Notably, even when compared to these prior domain-specinn learning approaches that incorporate meta-data, D'G consistently achieves superior performance, underscoring its effectiveness in domain relation modeling.
Ensemble methods. Our approach is closely related to ensemble methods, such as those that aggregate the predictions of multiple learners (Hansen and Salamon 1990, Dietterich] Laksh! minarayanan et al., 2017) ) or selectively combine the prediction from multiple experts (Jordan and Jacobs 1994 Bigen et al. 2013 Shazeer et al. 2017: Dauphin et al. 2017). When distribution shift occurs, prior works have attempted to solve the underspecification problem by learning a diverse set of functions with the help of unlabeled data (Teney et al. 2021 Paghardini et al.) 2022 Lee al. 2022). These methods aim to resolve the underspecification problem in the training data and disambi tisambiguatt guate the model, thereby improving distribuiitiiitionn robustness. Unlike prior works that rely on ensemble models to address the underspecification problem and improve out-of- of-distriiuutionn robustness, our proposed D'G takes a conceptually different approach by constructing Nomain-speciin models.
Laksh
7 CONCLUSION
In summary, the paper presents novel method called D'G for tackling the issue of domain shifts in real-world machine learning scenarios. The approach leverages the connections between different weighting
domains to enhance the model's robustness and employs domain-relationship system for each test domain. We evaluate the effectiveness of D'G on various datasets and observe that it consistently surpasses current methods, resulting in substantial performance enhancements.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To make sure our work can be easily reproduced, we've outlined the training and test steps in Algorithm | Our theory Section 4 supported by proofs the Appendix [ We've also provided more information about our experiments and settings in Appendix D along with description of the dataset Appendix Code reproduce our results will be made publicly available.
ACKNOWLEDGEMENT
We thank Linjun Zhang, Hao Wang, and members of the IRIS lab for the many insightful discussions and helpful feedback. This research was supported by Apple and Juniper Networks. CF is CIFAR fellow.
REFERENCES
Kartik Abuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio, loannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. In NeurlPS, 2021a.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio. Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for outto--disti generalization. 2021b.
David Swaree-Meliss and Nicolo Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 0111428-21433) 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz, Invariant risk minimization. arXiv preprint arXiv:1907.0288) 2019.
Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. M33: Meta test- time training for sulf-supe test-time adaption. In International Conference on Artificial Intelligence and Statisticcs, pages 3080-3090. PMLR, 2022.
Manh-Ha Bui, Toan Tran, A. Tran, and D.Q. Phung. Exploiting domain-specific features to enhance domain generalization. In Neural Information Processing Systems, 2021. URL https://api
semanticscholse org/ wwt.:oogg//ooooeeeeoe http:/cooos111::01199399
Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. ArXiv, abs/2006. 13009, 2020.
Yongxing Dai, Xiaotong Li, Jun Liu, Zekun Tong, and Ling yu Duan. Generalizable person re- identification with relevance-awan mixture of experts. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16149, 2021. URL https::/ /tt/://pp appi
semant semanticscholar htt.p:a...0.ooooooo011:::22221::
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pages 933-941, PMLR, 2017.
Thomas Dietterich. Ensemble methods in machine learning. International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint arXiv: 13/2.4314, 2013.
Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain--dversaa training of neural networks. J. Mach. Learn. Res., 2016a.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-advee adversarial training of neural networks. The journal of machine learning research,
2b1
10

---

Published as a conference paper at ICLR 2024
Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsumori Hashimoto, and Percy Liang. Out-of-distributien robustness via targeted augmentations. In NeurlPS 2022 Workshop on Distribution Shifts: Con- necting Methods and Applications.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-773, 2012. URL
http:// jm orgg wwwgggaapaeo papers vv.3/v//vvv/ htt////33//////ee//./// gretton 2a. html
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. ArXiv, abs/2007.01434,
Ruocheng Guo, Pengchuan Zhang, Hao Liu, and Emre Kıcıman. Out-of-distribution prediction with invariant risk minimization: The limitation and effective fix. ArXiv, abs/2101.07732. 2021.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10)993--001, 1990.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 12265,, 2019.
Strategies for pre- gre-training graph neural networks. arXiv preprint
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
Zeyi Huang. Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross-vommi generalization. ArXiv, has/2007.024455 2020. URL http:///cca.sssmmticccccccoooll
org/Corpus TD: (D: 220363892
Yuanfeng Ji, Lu Zhang, Jiaxiang Wu, Bing Wu, Long-Kai Huang, Tingyang Xu, Yu Rong. Lanqing Li. Jie Ren, Ding Xue, Houtim Lai, Shaoyong Xu, Jing Feng, Wei Liu, Ping Luo, Shuigeng Zhou, Junzhou Huang, Peilin Zhao, and Yatao Bian. Drugood: Out-of-distributiin (ood) dataset curator and benchmark for ai-aided drug discovery focus on affinity prediction problems with noise annotations. ArXiv, 0tt/2221..96677 2022.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures experts and the em algorithm. Neural computation, 6(2): 1994.
Kia Khezeli, Aro Blaas, Frank Soboczenski, Nicholas K. K. Chia, and John Kalantari. On invariance penalties for risk minimization. ArXiv, abs/2106.09777, 2021.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Selfreg: sule-vupe contrastive regularization for domain generalization. Proceedings of the IEEE/CVF International 2021.
Conference Computer Vision, pages
Pang Wei Koh. Shiori Sagawa, Henrik Marklund. Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard L. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in-the-wild distribution shifts. ICML, 2021a.
Pang Wei Koh, Shiori Sagawa, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, et al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021b.
Masanori Koyama and Shoichiro Yamaguchi. invariant predictor. ArXiv, abs/2008.0 018880 2020.
generalization with maximal
David Krueger, Ethan Caballero, foern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. -ittt--distii generalization via risk extrap- olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021a.
11

---

Published as a conference paper at ICLR 2024
David Krueger, Ethan Caballero, förn-Henrik Jacobsen, Amy Zhang. Jonathan Binas, Rémi Le Priol, and Aaron C. Courville. 2021b.
generalization via risk extrapolation (rex). In ICML,
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Conference on Neural Information Processing Systems, 2017.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspeci- fied data. arXiv preprint arXiv:2202.03411 2022.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 5400-5409, 2018a.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Chichung Kot. Domain generalization with adversarial feature learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 5400-5409, 2018b.
Zhiheng Li, Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. June 2023. URL htt.t::/t/ssattteee /ttaa/av.orgg/aabbba. www.oorg//aaaaaae.oeo.e.eeaaeeee 2212. 04825
Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng Zhang, and Dongsheng Li. Domain generalization using pretrained models without fine-tuning, arXiv preprint WxXiv:203.044600 2022.
Hyesu Lim, Byeonggeun Kim, Jaegul Choo, and Sungha Choi. Tin: A domain-shift aware batch normalization in test-time adaptation. arXiv preprint arXiv:2302.05755 2023.
Shengchao Liu, Meng Qu, Zuobai Zhang, Huiyu Cai, and Jian Tang. Structured multi-task learning for molecular property prediction. In International Conference on Artificial Intelligence and Statistics, pages 8906-8920. PMLR, 2022.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit++ When does self-supervised test-time training fail thrive? Advances in Neural Information Processing Systems, 14:21808 21820, 2021.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning transferable features with deep adaptation networks. ArXiv, abs/1502.02291, 2015.
Massimiliano Mancini, Samuel Rota Bulò, Barbara Caputo, and Elisa Ricci. Adagraph: Unifying predictive and continuous domain adaptation through graphs. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6561-6570, 2019. URL https
//api. sww.nt icscholar wttgCorpuue ID: 8 1982581
David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij. Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michal Nowotka, María Gordillo- Marañón, Fiona Hunter, Laura Junco, Grace Mugumbate, Milagros Rodriguez-Lopez, Francis Atkinson, Nicolas Bosc, Chris J Radoux, Aldo Segura-Cabrera, Anne Hersey, and Andrew R Leach. ChEMBL: towards direct deposition of bioassay data. Nucleic Acids Research, 47(DI): D930-D940, 2018. ISSN 0305-1048. doi: 10. 10.1093/nr//gy 1075. URL /ettps://doi.oor 10.1099/nar/ http0/9/////aa//y1ygg//a/ gky1075
Matteo Pagliardini, Martin Jaggi. François Fleuret, and Sai Praneeth Karimireddy. Agree to disagree: Diversity through disagreement for better transfcrabilityy arXiv preprint arXiw:2202.044414 2022.
Fengchun Qiao and Xi Peng. Topology- aware robust optimization for dut-off-iistiibttion gener- alization. ArXiv, abs/2307. 13943, 2023. URL
CoxpusID:25929822.
Xin Qin, Jindong Wang. Yiqiang Chen, Wang Lu, and Xinlong Jiang. Domain generalization for activity recognition via adaptive feature fusion. ACM Transactions Intelligent Systems and Technology, 14:1- 21, 2022. URL https::/// :tttps://apl.. http://app...aaaiiieeeeee sewanticschenteo org/ org/CorpusID. 2.51 01 8.355
12

---

Published as a conference paper at ICLR 2024
Yangjun Ruan, Yann Dubois, and Chris J Maddison. Optimal representations for covariate shift. 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. International Conference Learning Representations, 2020.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33: 13:111539-155111 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 1tt1::1701.66533. 2017.
Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain -agnenee meta-learning. 2021 IEEE/CVF Conference on Computer 2021.
Vision and Pattern Recognition (CVPR), pages
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, pages 443 440-450. Springer, 2016.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9229-9248 PMLR, 2020.
Damian Szklarczyk, Annika L Gable, David Lyon, Alexander Junge, Stefan Wyder, Jaime Huerta- Cepas, Milan Simonovic, Nadezhda T Doncheva, John H Morris, Peer Bork, et al. String vl1: protein-protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets. Nucleic acids research, 47(DI)::607-DDG 2019.
Damien Teney, Ehsan Abbasnejad, Simon Lucey, and Anton van den Hengel. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization. arXiv preprint arXiv:2105.05612 2021.
Eric Tzeng, Judy Hoffman, N. Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. ArXiv, abs/1412.3474, 2014.
Vladimir Naumovich Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 10 5:988-99, 1999.
Russell S Vosc, Scott Applequist, Mike Squires, Imke Durre, Matthew J Menne, Claude N. Jr. Williams, Chris Fenimore, Karin Gleason, and Derek Amdd. Gridded 5km ghen-daily temperature and precipitation dataset (nclimgrid) version 1. Maximum Temperature, Minimum Temperature, Average Temperature, and Precipitation, 2014.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022a.
Qixun Wang, Yifei Wang, Hong Zhu, and Yisen Wang. Improving out-of-distrihution generalization by adversarial training with structured priors. ArXiv, abs/221100652 2022b.
Yufei Wang, Haoliang Li, and Alex Chichung Kot. Heterogeneous domain generalization via domain mixup. ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3622-3626. 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019. URL https //openreviee ew www.:neettruteemm htt////ormmmdrrredeeeee
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In AAAI, 2020.
13

---

Published as a conference paper at ICLR 2024
Zihao Xu, Hao He, Guang- He Lee, Yuyang Wang, and Hao Wang. Graph-relational domain adaptation. arXiv preprint:2202.036225 2022.
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. ArXiv, abs/2001.00677, 2020.
Huaxiu Yao, Yiping Wang. Linjun Zhang, James Zou, and Chelsea Finn. C-mixup: Improving generalization in regression. arXiv preprint arXiv:2210.05775, 2022a.
Huaxiu Yao, Yu Wang. Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. arXiv preprint arXiv:2201.00290 2022b.
Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto L. Sangiovanni- Vincentelli, Kurt Keutzer, and Boqing Gong. Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2100-2110, 2019.
Daoan Zhang, Mingkai Chen, Chenming Li, Lingyun Huang, and Jianguo Zhang. Aggregation of dis- entanglement: Reconsidering domain variations in domain generalization. ArXiv, abs/2302.02350, 2023.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-38642, 2022a.
Yi-Fan Zhang, Han Zhang, Jindong Wang, Zhang Zhang, B. Yu, Liangdao Wang, Dacheng Tao, and Xingxu Xie. Domain-specific risk minimization. ArXiv, abs/2208.08661 2022b.
Bowen Zhao, Chen Chen, and Shu-Tao Xia. Delta: degradation-free fully test-time adaptation. arXiv preprint arXiv:2301.13012 2023.
Kaiyang Zhou, Yongxin Yang, Timothy M. Hospedales, and Tao Xiang. Deep domain- advessaral image generation for domain generalisation. ArXiv, abs/2003.06054, 2020.
14

---

Published as a conference paper at ICLR 2024
A DETAILED PROOFS
A.1 PROOF OF THEOREM 4.1
To prove theorem 4.1, let us first define an intermediate function
En Ogc(00
1
(1)
(10)
im 1k=1 Oke
We then define the event En {
N'r i=l au > 0}. Based on our assumption EE(h
16( (e(x)))] = = ( ((H) and nd 2 n for each domain d. On the event En. we have that
E E (e(o)) 110 (e(x)))
Ei(h 02 (c(x)) (e(x)))))
(E aRa)²
max; E (e(r)) N(
(11)
Σk=1 age
C(H)
=0
N'r k= akt
Moreover, since (1/(0) - 1100 10000 G. 1200 - 26011 that on the scenario En.
S B when 1 - -20)] S B, we have
Nrr Σ 1 aie (h(0) - 10((h(0)h0)) 1())
Σ 1[1 |0 B)(()) (1/0001009) -1,00
n1(33
SG. B.
Σ Ner k== ake
EMT 1( | < B}
(12)
On the other hand, on the complement event E, as the denominator equals to by our definition, we have h (10 (e(a:) = 0 and therefore
h (n (((x))
(13)
Consequently, have
(e(=)))) (e((()))) +G² +(())((()))((( 1
(14)
Therefore,
E (ht)) - -h()))]]]]]] < E
C(H)
+ B² 32+BB (((())))(((()) 1
1E,
N'r
k= agt
To bound the first term, we let E 1 1(1 distributed on [0. 1]", we have that S ~ Binomial(N .g) with q = P[] - -0 < B). Using the property of binomial distribution, we have
Z < B}. Since ( are uniformly
1 0}
1
1
E
z N
(15)
S
Nog
Therefore the first term
C(H)
C(H)
E
1
S
(16)
72 Ek=r aket
N. B
15

---

Published as a conference paper at ICLR 2024
The third term can be bounded as E (n()))(e(()) 1es] I (up((("))((())))((((
- 1) supph(())((( N"g N B"
(17)
Combining all the pieces, we get
< B² C(H)/n N
E [ (h() -(00)
(18)
Therefore, when / is Lipshictz with respect to the first argument, we have that (((()))) (x).
(e) B(
S [ n00
C(H)/n S v B(((()) 1100)99 B+ V
NW B"
(19)
A.2 PROOF OF PROPOSITION 4.2
If we treat all training domains are equally important, i.e., for all and t, we have that
NN-r ag
N
1000
M
(20)
Σk=1 Okt
Ntr
As it is simply the average of predictive values, we denote it by ((vvg)
In order to show that such an estimator performs worse than i in the minimax sense, all we need is to find an h E H. so that R2 (avg)) (f(z)) (f(()) = 21(1) even for N, n + cc.
We simply let U(0, 1], e(x) E (N 1) and (04) (e(x)) = d e(x). Under such setting, we have that
(avg) = 2 e(xx,
and therefore
(ars) (e())) E (d , ( (x)))
= = -(1).
1
(21)
12
We complete the proof.
B ADDITIONAL RELATED WORK: DISCUSSION WITH TEST-TIME ADAPTATION
Traditional test-time adaptation approaches for domain generalization typically involve utilizing unlabeled target data to adapt the model. These approaches can be categorized into two main groups: (1) jointly optimizing the model with supervised self-supervised loss during test time (Bartler et al.)
2022 Liu et al.) 202T; Sun et al.) 20200; (2) adapting the model solely during the test phase, which
includes techniques such as adapting batch normalization (Lim al. 2023 Schneider al. 2020
Zhao et al., )023), maximizing prediction consistency (Wang et al. 2022a Zhang et al.] 2022a) etc.
In contrast these test-time adaptation methods, D'G takes a different approach by not relying on unlabeled data from the test domain during inference. Instead, D3G leverages domain meta-data, which is distinct from unlabeled data. This fundamental distinction makes D3G more suitable for domain generalization, as it allows the model to generalize to unseen domains without accessing their data during training.
C DETAILED DESCRIPTION OF BASELINES
In this work, compare DOG large number of algorithms that span different learning strategies. We group them according to their categories, and provide detailed descriptions for each algorithm below.
16

---

Published as a conference paper at ICLR 2024
vanilla: ERM (Vapnik 1999) minimizes the average empirical loss across all training data.
distributionally robust optimization: GroupDRO (Sagawa et al. 2020) optimizes the worst-domain
loss.
2019) learns invariant predictors that perform well across (Ahuja et al. (021bb) performs IRM and ERM respectively with the information bottleneck constraint. V-REx (Krueger al.) 2021a) proposes penalty on the 2016b) employs an adversarial network to match 2016) matches the mean and covariance of feature distributions. MMD (Li et al. 2018a) matches the maximum mean discrepancy (Gretton et al.) 2012 of feature distributions. RSC (Huang al. 2020) discards the representations associated with the higher gradients in each epoch, foreing the model to predict with the remaining information. CAD (Raan al. 2022] introduces a contrastive adversarial domain bottleneck enforce support match using KL divergence. SelfReg (Kim et al.) (02T) utilizes the self-supervised contrastive losses to learn innaiinin representation. Mixup (Xu al. 2020) performs ERM on linear interpolations of examples from random pairs of domains and their labels. LISA (Yao et al.) 2022b) builds upon Mixup but interpolates samples with the same label but different domains, MAT (Wang et al. 2022b) conducts adversarial training with combinations of domain-wise multiple perturbations.
invariant learning: IRM (Arjovsky et al. different domains. IB-IRM and IB-ERM
variance of training risks. DANN (Ganin et al.
feature distributions. CORAL (Sun and Saenko
domain-speccfic learning: AdaGraph (Mancini et al. 2019) introduces GraphBN to provide each domain with its own BN statistics. RaMoE (Dai al. al.] 202T] use an effective voting-base based mix- ture mechanism to dynamically leverage edomain specific characteristics. mDSDI (Bui et al. 202I) introduce meta-optimization training framework boost domain-specific learning. AF- FAR (Qin et al. 2022 learns domain-specife, and domain-invarian representations, and fuse them dynamically. GRDA Xu al., (2022) generalize the adversarial learning framework with graph discriminator encoding domain adjacencys.. DRM (Zhang et al.) (002bb) utilizes test-time model selection and adaption to ensemble domain-specific classifiers. LLE (Li et al., 2023) aggregates the predictions from the ensemble of the last layers based the distributional shift classifier.
DDN (Zhang et al. 2023) performs domain-basse constrastive learning to decouple both domain invariant and task-specific domain variations. TRO (Qiao and Peng 2023) integrates distributional topology in principled optimization framework.
D DETAILED EXPERIMENTAL SETUPS AND HYPERPARAMETIRS
In this section, we detail our model selection for all datasets, where we use the same model architec- tures and use the same input (x. d) for all approaches for fair comparison, where domain meta-data used as features. Following (Xu al. 2022). we adopt two layer MLP network, and use no data augmentation for DG-15 and TPT-48. For the FMoW dataset, we fix the network architecture as the pretrained DenseNet- 121 model (Huang et al. 2017) and use the same data augmentation protocol as (Koh et al.) 2b2 random crоp and resize to 224 x 224 pixels, and normalization using the ImageNet channel statistics. For the ChEMBL-STRINE
dataset, we use graph isomorphism network (GIN) (Xu al. 2019) as the backbone network for all algorithms, and use data augmentation. We use an additional two layer MLP network to incorperate the domain meta-data as features for all approaches.
We list the hyperparameters in Table 4 for all datasets.
Table 4: Hyperparameters for D*G on all datasets.
Hyperparameters
DG-15
TPT-48
FMoW
CLEMBL-STRING
Learning Rate
lc-5
2c-3
lc-4
Ic-4
Weight Decay
Se-4
5e-4
0
0
Batch Size
10
64
32
30
Epochs
30
40
60
100
Loss Balanced Coefficient
0.5
0.5
0.5
0.5
Relation Combining Coefficient 3
0.8
0.5
0.8
0.5
17

---

Published as a conference paper at ICLR 2024
E ADDITIONAL DESCRIPTION OF DATASETS
E.
TPT-48
TPT-48 is real-world weather prediction dataset from the nClimDiv and nClimGrid (Vose et al. 2014) databases, which contains the monthly average temperature for the 48 contiguous states in the US from 2008 to 2019. The task is to forecast the next 6 months' temperature based the previous 6 months' temperature. Each state is treated as domain, and the domain meta-data is defined as the geographical location of each state. Following Xu et al.) (2022), we consider two dataset splits: N. (24) - S (24): generalizing from the 24 states in the north to the 24 states in the south; II. E (24) + W (24): generalizing from the 24 states in the east to the 24 states in the west. 0/1 adjacency matrix is used as the fixed domain similarity matrix, where a 1 represents that states and j are geographically connected. We show the fixed relations for N (24) - S (24) and E (24) - (24) in TPT-48. For the 24 test domains in these two tasks, we further random split them into 12 validation domains and 12 test domains.
VA
VAA
(a) N (24) - S (24)
(b) E (24) - (24)
Figure 5: Fixed relation graphs for the two tasks TPT-48, with red nodes indicating training domains and blue nodes indicating test domains. Left: Generalization from the 24 states in the north the 24 states in the south. Right: Generalization from the 24 states in the east to the 24 states in the west.
E.2 FMOW
Similar to TPT-48, we use 0/1 adjacency matrix to formulate the fixed domain relations in FMoW. where ag = represents that regions i and are geographically connected. For the construction of FMoW-Asia, we choose 18 countries or special administrative regions in Asia with most satellite images, and regions belong to the same sub-continent (Eastern Asia, Western Asia, Central Asia, Southern Asia and South-eastern Asia) are connected. The number of training, validation and test domains are 8. 5, respectively.
E.3 CHEMBL-STRING
We follow the dataset proposed in SGNN-EBM (Liu et al. 2022), CHEMBL-STRING, It multi- domain dataset with explicit domain relation. The three key steps are as follows:
Filtering molecules. Among 456,331 molecules in the LSC dataset, 969 are filtered out following the pipeline in (Hu et al.) (019).
Querying the PPI scores. Then we obtain the PPI scores by quering the ChEMBL (Mendez ct al. 2018) and STRING (Szklarczyk et al. 2019) databases.
Finally, calculate the edge weights Wiy. Lee, domain relation score, for domain i and in the domain relation graph to be Maxx(PPI(m,, m,) : m E M,, m; E My where M; denotes the protein set of domain The resulting domain relation graph has 1,310 nodes and 9,172 edges with non-zero weights.
The statistics of the resulting ChEMBL-STRING dataset with two thresholds can be found at Table
18

---

Published as a conference paper at ICLR 2024
Table Statistics about 50 and 100> subsets in ChEMBL-STRING, where use proteins
as domains. Sparsity here is defined as the ratio of zero values in the domain relation graph.
Threshold
# Samples
# Proteins
Sparsity
# Train Proteins
Valid Proteins
# Test Proteins
50
87908
140 121
0,914
19
101
58823
0.911
73
24
24
F ADDITIONAL RESULTS OF ILLUSTRATIVE Toy TASK
The full results on DG-15 are reported in T
Table 6: Full Results of domain shifts on DG-15. The standard deviation is computed across three seeds.
Model
ERM
GroupDRO
IRM
IB-IRM
IB-ERM
V-REx
Accuracy
44.0 4.6%
47.7 + 9.0%
43.9 5.1%
45.4 5.7%
43.1
1.9%
44.0 + 4.1%
Model
DANN
CORAL
MMD
RSC
CAD
SelfReg
Accuracy
43.1 + 4.5%
43.5 + 1.5%
41.3 + 1.0%
58.2 + 4.3%
43.3 + 2.8%
40.7 + 0.7%
Model
Mixup
LISA
MAT
RaMoE
mDSDI
AFFAR
Accuracy
41.3 3.9%
47.4 + 0.9%
39.6 + 1.4%
53.7 1.9%
45.2 1.0%
58.2 + 4.3%
Model
GRDA
DRM
LLE
DDN
TRO
D²G (ours)
Accuracy
59.5 + 4.1%
61.4 + 2.3%
58.8 + 3.1%
66.2 2.0%
56.3 + 2.7%
77.5 = 2.5%
G ADDITIONAL RESULTS OF REAL WORLD DOMAIN SHIFTS
G.1 FULL RESULTS FMOW
The full results FMoW reported in Table 7
G.2 FULL RESULTS ON CHEMBL-STRING
The full results on ChEMBL-STRING are reported in Table g
G.3 INCORPORATING DOMAIN RELATIONS INTO THE DOMAIN- DOMAIN-SPECIHIC LEARNING
To better emphasize the effectiveness of our domain relation refinement and consistency regularization, present additional evidence in Table 9 by incorporating existing domain relations into previous domain-specific learning methods. is important to note that we also include domain meta-data in these models. As shown in Table the introduction of existing domain relations does improve the performance of these Nomain--peciffi learning approaches. However, even with the incorporation of these relations, DCG outperforms them significantly. This outcome is not surprising, considering that refining domain relations with consistency regularization enables us to better capture domain similarity, leading superior performance.
H ADDITIONAL RESULTS OF ANALYSIS
H.1 FULL RESULTS OF THE ANALYSIS DIFFERENT RELATIONS
In Table TO report the full results of the analysis of different types of relations.
19

---

Published as a conference paper at ICLR 2024
Table 7: Full results of domain shifts on FMoW. The standard deviation is computed across three seeds.
FMoW-Aiss:
FMoW-WILDS
Worst Ace.
Average Ace.
Worst Acc.
Average Acc.
ERM
26.05 3.84%
35.50 + 0.20%
34.87 = 0.41%
52.64 + 0.30%
GroupDRO
26.24 1.85%
34.14 0.15%
31.16 2.12%
46.79 + 0.08%
IRM
25.02 2.38%
33.28 0.68%
32.54 + 1.92%
50.39 = 0.66%
IB-IRM
26,30 = 1.51%
35,43 + 0.37%
34.94 + 1.38%
51.90 + 0.27%
IB-ERM
26.78 + 1.34%
35.65 + 0.62%
35.52 = 0.79%
52.36 + 0.31 %
V-REx
26.63 0.93%
35.71 0.21%
37.64 + 0.92%
52.89 + 0.10%
DANN
25.62 1.59%
34.53 0.76%
33.78 + 1.55%
50.50 + 0.25%
CORAL
25.87 1.97%
35.43 = 0.12%
36.53 + 0.15%
51.89 = 0.35%
MMD
25.06 = 2.19%
33.77 + 0.57%
35.48 + 1.81% %
50.17 + 0.17%
RSC
25.73 0.70%
34.27 0.38%
34.59 = 0.42%
51.32 + 0.77%
CAD
26.13 1.82%
34.97 0.41%
35.17 1.73%
50.92 + 0.35%
SelfReg
24.81 1.77%
36.27 0.50%
37.33 0.87%
51.71 + 0.28%
Mixup
26.99 = 1.27%
36.41 + 0.31%
35.67 + 0.53%
53.50 + 0.11%
LISA
26.05 2.09%
35.17 1 0.69%
34.59 1 1.28%
50.95 1 0.13%
MAT
25.92 2.83%
34.68 0.23%
35.07 11 0.84%
52.11 + 0.25%
AdaGraph
25.91 0.59%
34.18 0.17%
35.42 0.55%
50.74 1 0.62%
RaMoE
26.65 0.46%
34.37 = 0.36%
36.51 = 0.71%
52.29 = 0,30%
mDSDI
25.54 = 0.46%
34.63 + 0.35%
36.35 + 0.45%
52.14 0,49%
ADDAR
25.87 1.01%
34.04 = 0.83%
35.77 = 0.70%
52.23 = 0.50%
GRDA
26.57 0.70%
34.47 + 0.62%
34.41 0.42%
53.03 + 0.97%
DRM
26.37 1.19%
35.22 0.47%
35.83 + 1.00%
52.66 + 0.41%
LLE
25.22 2.33%
35.67 + 0.19%
36.39 + 0.76%
51.80 + 0.19%
DDN
26.77 = 1.72%
35.30 + 0.31%
35.13 + 0.62%
52.36 + 0.34%
TRO
26.87 1.26%
35.03 0.42%
37.48 + 0.55%
52.66 + 0.28%
DOG (ours)
28.12 0.28%
37.71 + 0.48% %
39.47 0.57%
54.36 0.12%
H.2
FULL RESULTS OF COMPARISON WITH DOMAIN-SPECIEE FINE-TUNING
In Table we report the full results of comparison with domain -specffi fine-tuning.
H.3 FULL RESULTS OF COMPARISON WITH USING SEPARATE FEATURE EXTRACTORS FOR EACH TRAINING DOMAIN.
We conduct an additional comparison between D&G and one variant, which employs separate feature extractors for each training domain. The results are reported in Table According to the results, D*G shows superior performance compared to using separate feature extractors for each training domain. This underscores the importance of employing shared feature extractor to learn universal representation, while domain- specific heads can further identify domain-speciffe features. Additionally, our D model demonstrates superior efficiency compared to this variant since it utilizes a shared feature extractors while the variant needs N feature extractors.
H.4 FULL RESULTS OF COMPARISON WITH VARIANT THAT USE LIMITED DOMAIN INFORMATION
We further compare D7G with two addition variants, in case where we only have limited domain information. The two variants with the corresponding analysis described as:
D w/o domain split information: In situations where domain split information is unavailable, employ clustering approach to group all training data into several clusters, with the number of clusters matching the actual number of domains. Each cluster is treated as a separate domain, and domain relations established based the distances between the clustering centers.
20

---

Published as a conference paper at ICLR 2024
Table 8: Full results domain shifts on ChEMBL-STRING.
50
PPI
ROC-AUC
Accuracy
ROC-AUC
Accuracy
ERM
74.11 + 0.35%
71.15 + 0.43%
71.91 + 0.24%
70.39 + 0.36%
GroupDRO
73.98 0.25%
69.59 0.56%
71.55 + 0.59%
67.00 + 0.85%
IRM
52.71 0.50%
64.30 0.02%
51.73 + 1.54%
63.16 + 1.82%
IB-IRM
52.12 + 0.91%
63.57 + 0.21%
52.33 + 1.06%
63.39 + 1.35%
IB-ERM
74.69 = 0.14%
71.47 + 0.43%
73.32 + 0.21%
71.18 + 0.27%
V-REx
71.46 1.47%
71.33 0.90%
69.37 = 0.85%
70.93 = 1.06%
DANN
73.49 0.45%
70.74 0.38%
72.22 + 0.10%
70.41 = 0.25%
CORAL
75.42 0.15%
71.71 + 0.34%
73.10 + 0.14%
70.88 + 0.10%
MMD
75.11 0.27%
71.57 + 0.40%
73.30 + 0.50%
71.15 + 0.63%
RSC
74.83 0.68%
71.20 + 0.34%
72.47 = 0.38%
70.77 = 0.50%
CAD
75.17 0.64%
71.97 0.83%
72.92 1 0.39%
71.34 0.46%
SelfReg
75.42 0.42%%
70.34 0.57%
72.63 1 0.71%
69.14 1 0.90%
Mixup
74.40 0.54%
71.39 1 0.29%
71.31 + 1.06%
70.29 1: 0.15%
LISA
74.30 = 0.59%
71.72 0.66%
71,45 + 0.44%
70.37 + 0.29%
MAT
74.73 0.30%
72.03 0.51%
72.07 = 0.81%
71.09 0.44%
AdaGraph
74.02 -0.42%
71.09 1 0.16%
72.10 0.06%
70.75 = 0.56%
RaMoE
74.99 0.222%
71.23 0.41%
71.48 0.49%
70.60 + 0.71%
mDSDI
75.09 0.47%
71.50 0.40%
+ 71.23 0.69%
70.26 + 0.92%
ADDAR
74.55 0.54%
71.16 0.79%
71.93 + 0.33%
70.95 + 0.89%
GRDA
75.01 0.68%
0.91% 71.47
73.57 + 0.38%
70.44 0.29%
DRM
74.01 0.63%
71.68 0.29%
71.68 1 0.61%
70.55 + 0.82%
LLE
74.34 0.48%
71.26 1 0.17%
72.41 + 0.76%
70.61 + 0.59%
DDN
75.17 0.61%
71.75 + 0.52%
72.71 + 0.59%
71.02 + 0.27%
TRO
74.85 0.27%
71.53 + 0.16%
72.49 - 0.36%
70.88 + 0.51%
D`G (ours)
78.67 0.16%
74.25 1 0.34%
77.24 0.30%
73.50 1 0.24%
Table 9: Full results of incorporating domain relations into the domain-specific learning.
TPT448 (MSH L)
HMoW (Worst Acc. t)
(ROCU-
Model
N (24) + (24)
(24) - W (24)
FMoW-Asia
FMoW-WILDS
PPL
PPI
0.603 0.041
0.467 0.047
26.37 1.19%
35.83
1.00%
74.01 + 0.63%
71.68 + 0.61%
DRM
Fixed
27.62 0.42%
37.58
76.35
0.77%
74.32 +:
0.571 0.038
0.557
0.027
25.22 2.33%
36.39
0.76%
74.34 0.48%
72.41 + 0.76%
LLE
Fixed
0.519 0.072
0.417 0.049
26.91 1.05%
37.11 0.21%
76.27 0.69%
74.81 + 0.36%
0.537 0.024
0.601
27.89 0.95%
35.13 0.62%
75.17 0.61%
+ 0.59%
DDN
Fixed
0.529 : 0.017
0.052 0.433
26.77 1.72%
37.25 + 0.44%
76.91 = 0.50%
75.57 + 0.68%
D'G
0.342
0.236
28.12 0.28%
39.47 0.57%
0.16%
77.24 0.30%
The results, as presented in Table 13 demonstrate that our method's variant outperforms ERM in out of settings, although still lags behind D&G. This suggests that the model architecture and inference algorithm can be adapted to learn domains and domain relations simultaneously in an end-to-end manner. Moreover, the appropriate utilization domain meta-data and domain relations significantly improves performance.
D w/o domain meta-data: In situations where domain meta-data unavailable, we learn domain relations directly from the data. To achieve this, compute the fixed relation using optimal transport Alvarez-Melis and Fusi (2020). For the learnable part, we utilize the average of features from each domain as input to the two-layer neural network g.
According to Table 13 the findings indicate that this variant exhibits inferior performance compared to the with domain meta-data. However, still outperforms the best baselines most datasets. These outcomes suggest that the model design D inherently enhances generalization capabilities, which are further augmented by the use of domain meta-data.
21

---

Published as a conference paper at ICLR 2024
Table 10: Comparison of using different relations. The results on FMoW and CHEMBL-STRING are reported. In this case, when no relations are used, we take the average of predictions across all domains.
Fixed
Learned relations
FMoW (Worst Acc. 1) FMoW-WIIDDS
CHEMBI-STRING (ROC-AUC
relations
FMoW-Asia
PPT
26.93 0.47%
35.32 + 0.66%
76.17 + 0.21%
73.38 + 0.13%
27.43 0.41%
39.37 + 0.34%
77.66 + 0.32%
76.59 + 0.40%
21.18 2.30%
36.41 1.09%
77.09 + 0.94%
75.57 + 1.20%
28.12 = 0.28%
39.47 + 0.57%
78.67 = 0.16%
77.24 + 0.30%
Table 11: Full results of comparison between D³G with domain-specific fine-tuning.
Model
FMoW (Worst Acc. t)
ChEMBL (ROC-AUC f)
FMoW-Asia
FMoW-WILDS
PP1
ERM
26.05 + 3.84%
34.87 + 0,41%
74.11 + 0.35%
71.91 + 0.24%
CORAL
25.87 + 1.97%
36.53 = 0.15%
75.42 = 0.15%
73.10 + 0.14%
RW-FT
27.03 1.03%
36.39 1.28%
76.31 + 0.35%
74.30 + 0.40%
D²G
28.12
0.28%
39.47
0.57%
78.67
0.16%
77.24
0.30%
I
LIMITATIONS
While work provides theoretical results, it is important to acknowledge the limitations of our approach. We rely on certain assumptions, and therefore, D'G does not guarantce a proven enhance- ment in out-of-domain generalization in all scenarios. However, our theoretical results in future research. Furthermore, it is worth noting that D'G extracts domain relations from domain meta-data. However, the performance can be affected in cases where the domain meta-data is of inferior quality. In our future work, we will address this issue by providing detailed insights obtaining high-quality domain meta-data.
committed expanding
22

---

Published as a conference paper at ICLR 2024
Table 12: Full results of comparison between D*G with using separate feature extractors for each training domain.
Model
FMoW (Worst Acc. FMoW-WILDS
ChEMBL (ROC-AUC 1)
FMoW-Asia
50
PPI
Using separate feature extractors
23.62 + 0.47%
35.44 + 0.58%
76.11 + 0.31%
74.19 + 0.74%
D
28.12 0.28% %
39.47 + 0.57%
78.67 + 0.16%
77.24 +: 0.30%
Table 13: Full results of comparison between D with its variant that limited domain information.
Model
FMoW (Worst Acc. 1) FMoW-Asia
ChEMBL (ROC-AUC 1)
FMoW-WIIDDS
PPI
PPI
D'G w/o domain information
25.46 1.35%
36.06 + 0.39%
74.94 1 0.79%
73.38 0.91%
D'G w/o domain meta-data
27.17 + 1.09%
37.82 + 0.31 %
76.65 + 0.22%
75.01 + 0.36%
D'G
28.12 0.28%
39.47 + 0.57 %
78.67 + 0.16 %
77.24 = 0.30%
23


Reviews:
[{"review_id": "SO7ufX8SDm", "reviewer": "Reviewer_nUD5", "summary": "This paper attempts to solve the multi-source multi-target domain generalization problem using domain relations. The authors claim that existing single domain-invariant or multiple domain-specific models that leverage equal weights on all domains fail to capture appropriate domain-specific correlations. To tackle this problem, the authors extract domain relations from domain meta-data and design a relation-aware consistency regularizer to weight the training domain-specific functions. The weighted functions on source domains are then transferred to predictions on target domains. A theoretical analysis is provided to prove that the proposed approach can perform better than the models using equal domain weights.", "questions": "Please see the weakness points.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is clearly written and organized. The underlying idea is straightforward and easy to follow.\n\n2. The authors provide a theoretical analysis that verifies that the proposed relation-aware consistency loss can achieve superior generalization performance compared to the approach of treating all training domains\nequally. \n\n3. The experimental evaluation is comprehensive and thorough. An illustrative toy task is designed to examine the proposed methods and many tables and figures are presented to analyze the results.", "weaknesses": "1. The novelty of the proposed method may be somewhat limited as it bears similarities to ensemble methods, which also involve assigning varying weights to different domains. The essential distinctions between the proposed methods and ensemble methods should be clearly elucidated, and it is important to consider baseline models that incorporate ensemble methods for comparison.\n\n2. The evaluation metrics used in this paper are different from previous works conducted on the TPT-48, FMoW, and ChEMBL-STRING datasets. It is unclear whether the metrics used, such as MSE, are suitable and fair to be used to evaluate domain generalization models.\n\n3. As stated by the authors, the theoretical analysis of this paper relies on certain assumptions. It is questioning whether these assumptions reflect the real-world datasets. For example, the assumptions that the domain relations accurately capture the similarity between domains, and that they are determined solely by the distance between domain representations, may not always hold true in real-world datasets. In the proofs of the theorems, the authors only prove that the proposed estimator outperforms the equally weighted estimator in the minimax sense, but not a theoretically global superiority. Moreover, the theorems only reveal that uneven weights can be better than equal weights, which does not entirely support the key ideas of acquiring appropriate weights for different domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7rjNHAyVhY", "reviewer": "Reviewer_2jGa", "summary": "This paper considers the problem of out-of-domain generalization of discriminative models. The presented approach leverages domain relations and domain-specific meta-data to adapt to new domains at test time. Domain similarities are determined from a combination of a user-defined function and a learned function of the domain meta-data. The domain similarities are used to weight the different neural network heads in a mixture-of-experts fashion. At test time, the domain similarities are used to weight the output of all of the training set domain-specific heads to produce a prediction. Theoretical results provide justification for the proposed approach. Empirical results are shown on open source toy and real-world datasets, including several useful ablations.", "questions": "* Is it possible for the proposed method to utilize negative relationships between domains?\n* How will the proposed method behave in a situation where there are no related domains according to the meta-data? Will the method perform as well as a domain-invariant approach? Is this what the consistency loss is for?\n* If domain information was unavailable, could a domain discovery method (e.g. clustering) be used to learn domains and domain relations from unlabeled data? In the limit, could the model architecture and inference algorithm be adapted to learn domains and domain relations end-to-end?\n* Could this method be used in data impoverished applications to improve performance? \n\nMinor typos:\n* Equations are referenced with \"Eqn. equation #\", maybe just \"(#)\" or \"Equation (#)\"?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation for the paper is strong, in a relevant area.\n* The paper is well-written.\n* The empirical results are very strong and well executed, including a large number of baselines. The ablation studies are beneficial, answering fundamental questions about the method.", "weaknesses": "* The method requires a user-defined domain similarity function based on meta-data. This might not always be available. Additionally, for specific applications, a user might not be able to credibly define such a function.\n* The theoretical work makes (understandably) quite a few strong assumptions that are unlikely to be true in practice. These results could be improved to show how performance is affected by, for example, noisy domain relations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O0oANCPFmC", "reviewer": "Reviewer_bK1N", "summary": "The paper addresses the domain shift problem and proposes a new approach called D3G to this problem. Instead of learning a single model from multiple source domains, the proposed method learns domain-specific models and infer a test domain specific model by exploiting the domain relations. The test domain model is a weighted combination of multiple source domain models whilst the weights are learned from the domain metadata. Theoretic and empirical analyses have been made and experimental results demonstrate the superiority of the proposed approach to many state-of-the-art counterparts.", "questions": "1. Does it work when there is no domain metadata available? Can such domain relations be learned from the data themselves?\n2. How the domain metadata are used in the comparative methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "-- The approach distincts from most existing ones in that it learns domain-specific models instead of a unified model. \n\n-- The method is clearly presented and theoretic analysis has been made to clarify why it should work for domain generalization.\n\n-- The proposed approach is applicable to many real-world applications as demonstrated in the experiments. The superior performance makes a significant difference to practical problems of this type.\n\n-- Ablation studies have been conducted to demonstrate the effectiveness of each component of the proposed approach.", "weaknesses": "-- The comparison of the proposed method with ensemble models of existing approaches should have been given.\n\n-- There exist some typos/language issues, e.g., \"Eqn. equation x...\"; \"We using weighted...\";", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfK9S6DJPS", "reviewer": "Reviewer_79WW", "summary": "This paper presents a novel method called D3G for tackling the issue of domain shifts in real-world machine learning scenarios. The approach leverages the connections between different domains to enhance the model’s robustness and employs a domain-relationship aware weighting system for each test domain.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The authors provide a theoretical proof that using domain relations to reweight a specific function of the training domain can obtain stronger extraterritorial generalization.", "weaknesses": "The novelty is limited, perhaps the authors have not sufficiently explained the differences between their work and existing methods.  \n\nThe authors state that \" Unlike prior works that rely on ensemble models to address the underspecification problem and improve out-of-distribution robustness, our proposed D3G takes a conceptually different approach by constructing domain-specific models.\" However, the  MoE[1] constructed domain-specific models and they adopted ensemble models to improve out-of-distribution robustness.\n\nThe meta-data is key of domain-relation in this work. However, there is no clear definition of meta-data, how to obtain meta-data, and why the meta-data work is lack of detailed elaboration and analysis.\n\nHow the consistency loss address the challenge of limited training data in certain domains?\n\nThe organizational of the method section needs to be adjusted. The description and calculation of the domain relationship \"a\" in advance make the method part more clear.\n\nIt is recommended to add the venue and year of the comparison method to the table in the experiment section.\n\n\n[1]Dai, Yongxing, et al. \"Generalizable person re-identification with relevance-aware mixture of experts.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3l6APVdfp", "reviewer": "Reviewer_Htjr", "summary": "This paper proposed a new approach for out-of-distribution generalization named as D$^3$G. It leverages domain relations estimated from domain metadata to learn training domain specific models and the ensemble of training-domain-specific functions. Under some assumptions, for example the domain relations can accurately capture the model similarity between domains, the authors proved that the proposed method can generalize better to out-of-domain samples compared to the traditional averaging approach. Empirical results on both synthetic and real-world datasets show that D$^3$G surpassing the performance of traditional averaging methods and some other baseline methods.", "questions": "Please refer to the concerns in the \"weaknesses\" part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Originality: good. Though learning domain specific classification head has been proposed in exiting works, it is novel to ensemble domain specific classifiers by weights learned from domain meta-data.\n- Significance: the proposed method is simple but shown improved performance on several tasks.\n- The paper is overall well written and easy to follow.\n- Experiment setups are introduced in detail.", "weaknesses": "- The robustness and generality of the proposed method is unclear. Practically, it is unclear how to design a good similarity definition for metadata of different tasks. The current way of constructing the fixed relations requires specific and expertise knowledge on each task. Nevertheless a learning approach is proposed, the ablation study results in Table 10 show that the learned relations can be less helpful on some tasks.\n- The limitations of this work are not fully discussed. For example, the generality of the assumptions in the theoretical analysis part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NWahxoHWQj", "reviewer": "Reviewer_u1oN", "summary": "This paper proposed D3G, which in contrast to previous approaches, utilizes domain metadata to create domain-specific models. During training, it learns domain-specific functions, and during testing, it reweights them based on domain relations, enhancing model adaptability using directly acquired domain metadata.", "questions": "Please refer to the Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to follow. The authors use appropriate notation and equations where necessary.\n- This paper utilizes domain meta-data to extract relations between domains.\n- The domain relations are learned rather than fixed.", "weaknesses": "(1) In section 3.2, it is mentioned that both $m_i$ and $a_{ij}^{g}$ are derived from domain metadata, but the specific relationship or distinction between them is not clearly explained. The experimental results suggest that $a_{ij}^{g}$ is extracted from $m_i$, yet this section lacks explicit clarification.\n\n(2) It is unclear whether the two layer neural network $g$ should be trained or directly be used from other previous works. It is mentioned that weight vectors $w_r$ are learnable vectors. Are these weights and the neural network trained by the consistency loss?\n\n(3) The consistency loss implies that each head is trained on all the domains with different weights. This is similar to the case where each head is trained on all the domains with a weighted combination of the losses of all the domains. For the predictor $f^{(d)}$, the loss is in this form: $l(\\frac{\\sum_j^{N^{tr}} a_{dj} f^{(d)}(x^j)}{\\sum_j^{N_{tr}} a_{dj}})$. Would this loss also help the learning of domain-specific models?\n\n(4) Suppose that each head is trained on all the domains by the loss function (2). In this case, regardless of the values of $a_{ij}$, if both the supervised loss and consistency loss reach their minimum, then the learning of $a_{ij}$ would be useless. Does the proposed method face this issue?\n\n(5) In the experiment section, one baseline of the proposed method would be training each head on all the training domains by supervised loss and averaging the predictions of these heads as the test prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)