Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization
Authors: Jivat Neet Kaur
Emre Kiciman
Amit Sharma
Venue: ICLR 2023 notable top 25% (2023)

Abstract: Recent empirical studies on domain generalization (DG) have shown that DG algorithms that perform well on some distribution shifts fail on others, and no state-of-the-art DG algorithm performs consistently well on all shifts. Moreover, real-world data often has multiple distribution shifts over different attributes; hence we introduce multi-attribute distribution shift datasets and find that the accuracy of existing DG algorithms falls even further. To explain these results, we provide a formal characterization of generalization under multi-attribute shifts using a canonical causal graph. Based on the relationship between spurious attributes and the classification label, we obtain realizations of the canonical causal graph that characterize common distribution shifts and show that each shift entails different independence constraints over observed variables. As a result, we prove that any algorithm based on a single, fixed constraint cannot work well across all shifts, providing theoretical evidence for mixed empirical results on DG algorithms. Based on this insight, we develop Causally Adaptive Constraint Minimization (CACM), an algorithm that uses knowledge about the data-generating process to adaptively identify and apply the correct independence constraints for regularization. Results on fully synthetic, MNIST, small NORB, and Waterbirds datasets, covering binary and multi-valued attributes and labels, show that adaptive dataset-dependent constraints lead to the highest accuracy on unseen domains whereas incorrect constraints fail to do so. Our results demonstrate the importance of modeling the causal relationships inherent in the data-generating process.

Published as a conference paper at ICLR 2023
MODELING THE DATA- PROCESS IS NECESSARY FOR OUT-OF-DISTRIBUTION GENERALIZATION
Jivat Neet Kaur, Emre Kıemaan Amit Sharma Microsoft Research {t-kaur jivat, emrek, awwhaz
ABSTRACT
Recent empirical studies domain generalization (DG) have shown that DG algorithms that perform well on some distribution shifts fail on others, and no state-of-ther tff-the-art art DG algorithm performs consistently well all shifts. Moreover, real-world data often has multiple distribution shifts over different attributes; hence introduce multi-ate mulsi-attribute distribution shift datasets and find that the accuracy of existing DG algorithms falls even further. To explain these results, we provide a for- mal characterization of generalization under multi-attribute shifts using canonical causal graph. Based on the relationship between spurious attributes and the classifi- cation label, we obtain realizations of the canonical causal graph that characterize common distribution shifts and show that each shift entails different independence constraints over observed variables. As a result, we prove that any algorithm based on single, fixed constraint cannot work well across all shifts, providing theoretical evidence for mixed empirical results on DG algorithms. Based on this insight, we develop Causally Adaptive Construint Minimization (CACM), an algorithm that uses knowledge about the data tata-gennrating process to adaptively identify and apply the correct independence constraints for regularization. Results fully synthetic, MNIST, small NORB, and Waterbirds datasets, covering binary and multi-valued attributes and labels, show that adaptive dataset-dependent constraints lead to the highest accuracy on unseen domains whereas incorrect constraints fail to do so. Our results demonstrate the importance of modeling the causal relationships inherent the data-generating process.
INTRODUCTION
To perform reliably in real world settings, machine learning models must be robust to distribution shifts - where the training distribution differs from the test distribution. Given data from multiple domains that share common optimal predictor, the domain generalization (DG) task (Wang et al.) Zhou
Zhou et al.) 2021) encapsulates this challenge by evaluating accuracy on an unseen domain. Recent empirical studies of DG algorithms (Wiles et al. 2022 Ye et al. 2022) have characterized different kinds of distribution shifts across domains. Using MNIST as an example, a diversity shift when domains are created either by adding new values of spurious attribute like rotation (c.g.. Rotated-MNIST dataset (Ghifary et al. 20155; Piratla et al. 2020 whereas a correlation shift is when domains exhibit different values correlation between the class label and spurious attribute (Arjovsky et al. 2019)). Partly because advances in representation learning for DG (Ahuja et al. 2021 Krueger et al. 2222 Mahajan al., 202T Arjovsky et al.) 2019) either one of the shifts, these studies find that performance of Mate-offthe-aa DG algorithms are not consistent across different shifts: algorithms performing well on datasets with one kind of shift fail on datasets with another kind of shift.
like color (e.g., Colored-MNN
Li et al. 2018a| [Sun & Saenko, (016) have focused
In this paper, we pose a harder, more realistic question: What if dataset exhibits two or more kinds of shifts simultaneously? Such shifts over multiple attributes (where an attribute refers to spurious high-level variable like rotation) are often observed real data. For example, satellite imagery data demonstrates distribution shifts over time well the region captured (Koh et al.) 2021). To study this question, we introduce multi attribute distribution shift datasets. For instance, in our Col+ dataset (see Figure T), both the color and rotation angle of digits can shift

---

Published as a conference paper at ICLR 2023
Train
Test
N
0.9
0.8
0.1
Y=0
5
2
E
Color
Y=1
3
60'
(b)
Y=0
6
Rot
Y=1
Algo.
Color
Rotation
Col+Rot
Multi-atiin aatt-buttbbbbte shift
ERM
30.9 + 1.6 61.9 + 0.5 25.2 + 1.3
IRM
50.
61.2
0.3
39.6 6.7
Col+Rot
(0.9,15")
(0.8,60")
(0.1,90")
MMD
29.7 + 1.8
62.2
24.1 0.6
Y=0
C-MMD
+9.4
62.3
0.4
32.2
Y=1
CACM
70.4 + 0.5
62.4
+ 0.4
54.1 1.3
(a)
Figure 1: (a) Our multi-attribute distribution shift dataset Col+Rot- We combine Colored MNIST (Arjovsky et al. 2019) and Rotated MNIST (Ghifary al. 2013) introduce distinct shifts over Color and Rotation attributes.
(b) The causal graph representing the data generating process for Col+Rot-MNISS Color has a correlation with Y which changes across environments while Rotation varies independently. (c) Comparison with DG algorithms optimizing for different constraints shows the superiority of Causally Adaptive Constraint Minimization (CACM) (full table in Section (5)
across data distributions. We find that existing DG algorithms that are often targeted for a specific shift fail to generalize such settings: best accuracy falls from 50-62% for individual shift MNIST datasets to <50% (lower than random guess) for the multi-attribute shift dataset.
To explain such failures, we propose a causal framework for generalization under multi-attribute distribution shifts. We use canonical causal graph to model commonly observed distribution shifts. Under this graph, we characterize a distribution shift by the type of relationship between spurious attributes and the classification label. leading to different realized causal DAGs. Using d-separation on the realized DAGs, we show that each shift entails distinct constraints over observed variables and prove that conditional independence constraint is valid across all shifts. As a special case of multi-attribute, when datasets exhibit a single-attribute shift across domains, this result provides an explanation for the inconsistent performance of DG algorithms reported by |Wiles al. (2022); Ye itt al. (202)). It implies that any algorithm based on a single, fixed independence constraint cannot work well across all shifts: there will be dataset on which will fail (Section 3.3)
We go on to ask if we can develop an algorithm that generalizes to different kinds of individual shifts shifts. For the common shifts modeled by the canonical graph.
as well as simultancous we show that identification of the correct regularization constraints requires knowing only the type of relationship between attributes and the label, not the full graph. As we discuss in Section 3.T] the type of shift for attribute is often available or can be inferred for real-world datasets. Based on this, we propose Causally Adaptive Constraint Minimization (CACM), an algorithm that leverages knowledge about the data-genedd generating process (DGP) to identify and apply the correct independence constraints for regularization. Given a dataset with auxiliary attributes and their relationship with the target label, CACM constrains the model's 's representation to obey the conditional independence constraints satisfied by causal features of the label, generalizing past work on causality- regularization (Mahajan al. 202T Veitch al. 2021 Makar et al. 2022) to multi-attribute shifts.
We evaluate CACM on novel multi-attribute shift datasets based on MNIST, small NORB, and Waterbirds images. Across all datasets, applying the incorrect constraint, often through an existing DG algorithm, leads to significantly lower accuracy than the correct constraint. Further, CACM achieves substantially better accuracy than existing algorithms on datasets with aulti-attribute shifts as well as individual shifts. Our contributions include:
Theoretical result that an algorithm using a fixed independence constraint cannot yield an optimal classifier on all datasets.
An algorithm, Causally Adaptive Constraint Minimization (CACM), to adaptively derive the correct regularization constraint(s) based on the causal graph that outperforms existing DG algorithms. Multi-attribute shifts- based benchmarks for domain generalization where existing algorithms fail.
2

---

Published as a conference paper at ICLR 2023
2 GENERALIZATION UNDER MULTI-ATTRIBUTE SHIFTS
We consider the supervised learning setup from (Wiles et al. 2022) where each row of train data (ar,, a4 V:) contains input features x;; (e.g., X-ray pixels), a set of nuisance spurious attributes a, (e.g., vertical shift, hospital) and class label y, (e.g., disease diagnosis). The attributes represent variables that are often recorded or implicit in data collection procedures. Some attributes represent property of the input (e.g., vertical shift) while others represent the domain from which the input was collected (e.g., hospital). The attributes affect the observed input features I; but do cause the target label, and hence are spurious attributes. The final classifier g(x) is expected to use only the input features. However, as new values of attributes are introduced or as the correlation of attributes with the label changes, we obtain different conditional distributions XX) Given set of data distributions P. we assume that the train data is sampled from distributions, Petr = {PEI, PEZ, .. P while the test data is assumed to be sampled from single unseen distribution, Pête = {PEEL) C P. Attributes and class labels are assumed to be discrete.
2.1 RISK INVARIANT PREDICTOR FOR GENERALIZATION UNDER SHIFTS
The goal is learn classifier g(x) using train domains such that it generalizes and achieves similar, small risk on test data from unseen P'tte as it achieves on the train data. Formally, given set of distributions P, we define a iisk-invariant predictor (Makar et 2022) as,
Definition 2.1. Optimal Risk Invariant Predictor for P (from (Makar al. 2022)) Define the risk of predictor 9 on distribution P E P as Rp(g) = p (g(x), (.) where È is cross-entropy or another classification loss. Then, the set of risk-invariant predictors obtain the same risk across all distributions P E P, and set of the optimal risk-invariant predictors is defined as the risk-invariant predictors that oblain minimum risk on all distributions.
grine arg min gEGrino
Rr(g) VP E P where Grine = : ()V P' EP) (1)
An intuitive way obtain risk-invariant predictor is to consider only the parts of the input features (X) that cause the label Y and ignore any variation due to the spurious attributes. Let such latent, unobserved causal features be X Due to independence and stability of causal mechanisms (Peters it al. (017). we can assume that P(Y|X, remains invariant across different distributions. Using the notion of risk invariance, we can now define the multi-attribun generalization problem as,
Definition 2.2. Generalization under Multi-attribute shifts. Given target label Y, input features X, attributes A, and latent causal features X consider set of distributions P such that P(Y|X.) remains invariant while P(A Y) changes across individual distributions. Using training dataset (æ;, ag 31 31))==1 sampled from a subset of distributions Petr C.P. P, the generalization goal is to learn an optimal risk- iik-ivvrriant predictor over P.
Special case of single-attribute shift. When |A| = 1, we obtain the single-attribute shift problem
that is widely studied (Wiles et al. 2022 Ye et al. 2022 Gulrajani & Lopez-Paz 202
2.2 GENERAL PRINCIPLE FOR NECESSARY CONDITIONAL INDEPENDENCE CONSTRAINTS
In practice, the causal features X. are unob- served and a key challenge is to learn X, using the observed (X, Y, A). We focus on represen- tation learning-bast algorithms, typically characterized by a regu- larization constraint that is added to standard ERM loss such as cross-entropy. Table shows three independence constraints that form the ba- sis of many popular DG algorithms, assuming environment/domiin E as the attribute and & as the main classifier loss. We now provide a gen- eral principle for deciding which constraints to choose for learning a risk invariaat predictor for dataset.
Table 1: Statistic optimized by DG algorithms.
match matches the statistic across E. h is domain classifier (loss fa) using shared representation
(Wang et al., 202T) DG
Constraint
Stutistic
Algo.
TT E
match Blo(x) A E max ElC.(h(o(=)), E)]
MMD
DANN
match Cov Cov(o(=)] V
CORAL
Elø
match [[Y(@(x), E] V match Var(f(g(=)).))) V
IRM
VRES
I. B|Y
match B[o(x) vl E maxg Bl(x(h(o(=)). E v)]
C-MMD
CDANN
3

---

Published as a conference paper at ICLR 2023
We utilize strategy from past work (Mahajan al. 2021: Veitch et al. 2021) to graph structure of the underlying data-ge gentrraing process (DGP). We assume that the predictor can be represented as g(x) 9i (o(x)) where ø is the representation. To learn fisk-invariant o, we identify the conditional independence constraints satisfied by causal features X, in the causal graph and enforce that learnt representation Φ should follow the same constraints. If Φ satísfies the constraints, then any function 91 (Ø) will also satisfy them. Below we show that the constraints are necessary under simple assumptions on the causal DAG representing the DGP for dataset. All proofs are in Suppl. B
Theorem 2.1. Consider causal DAG G over (X.; A.A) and corresponding generated dataset (I,,, at B= where X is unobserved. Assume that graph G has the following property: Xe is defined as the set of all parents of Y (X. - -YY;; Y); and X A together cause X (X - X, and A + X). The graph may have any other edges (see, e.g., DAG in Figure] (b)). Let Pg be the set of distributions consistent with graph G, obtained by changing P(A but not P(Y) Then the conditional independence constraints satisfied by X. are necessary for (cross-entropy) risk- invariant predictor over Pg. That is, if a predictor for Y does not satisfy any of these constraints, then there exists a data distribution P E Pg such that predictor's 's risk will be higher than its risk in other distributions.
Thus, given a causal DAG, using d-separation on X. and observed variables, we can derive the correct regularization constraints to be applied on ø. This yields general principle to learn risk-invariant predictor. We use it to theoretically explain the inconsistent results of existing DG algorithms (Sec. (.3) and propose
generalization algorithm CACM (Sec. 4 Note that constraints from CACM are necessary but not sufficient as X, is not identifiable.
STUDYING DISTRIBUTION SHIFTS THROUGH A CANONICAL CAUSAL GRAPH
3.1 CANONICAL CAUSAL GRAPH FOR COMMON DISTRIBUTION SHIFTS
We consider a canonical causal graph (Figure 2) specifyy the common data-generating processes that can lead to a multi-attribute shift dataset. Shaded nodes represent observed variables X, Y; and the sets of attributes And Aind, and E such that And U Aind U {E} = A. Aid represents the a special
attributes correlated with label, Aind the attributes that are independent of label, while attribute for the domain/environment from which data point was collected. Not all attributes need to be observed. For example, in some cases, only E and a subset of A ind Aind may be observed. In other cases, only A and and A and may be observed while E is not available. Regardless, that all attributes, along with the causal features X., determine the observed features X. And the features X are the only features that cause Y. In the simplest case, we assume no label shift across environments i.e. marginal distribution of Y is constant across train domains and test, Petr (t) Pete(y (y) (see Figure 2a). More generally, different domains may have different distribution of causal features (in the X-ray example, more women visit one hospital) as shown by E <-> X (Figure 2b).
assume
Under the canonical graph, we characterize different kinds of shifts based on the relationship between spurious attributes A and the classification label Y. Specifically. Aind is independent of the class label and a change in P(A ) leads to an Independent distribution shift. For Ann there are
X
X
Y
Asai
A ind
Aont
B
B
x
Azu
Ainot
(a)
(b)
(c)
Figure 2: (a) Canonical causal graph for specifying multi-attribute distribution shifts; (b) canonical graph with B-X, correlation. Anti-causal graph shown in Suppl. Shaded nodes denote observed variables; since not attributes may be observed, we use dotted boundary. Dashed lines denote correlation, between X. and E, and Y and A And E-X, correlation can be due to confounding, selection, or causal relationship; all our results hold for any of these relationships (see Suppl. F). (c) Different mechanisms for Y-Ama relationship that lead to Causal. Confounded and Selected shifts.
4

---

Published as a conference paper at ICLR 2023
three mechanisms which can introduce the dashed-ling correlation between And and Y (Figure 2ch direct-causal (Y causing And ), confounding between Y and Ann due to a common cause, or selection during the generraing process. Overall, we define four kinds of shifts based on the causal graph: Independent, Causal, Confounded, and Selected While the canonical graph in Figure 2a is general, resolving each dashed edge into specific type of shift (causal mechanism) leads to a realized causal DAG for particular dataset. As we shall see, knowledge of these shift types is sufficient to determine the correct independence constraints between observed variables.
Our canonical multi-attribute graph generalizes the DG graph from Mahajan et al. (202)) that considered an Independent domain/environment as the only attribute. Under the special case of a single attribute (|A) = 1), the canonical graph helps interpret the validity of popular DG methods for a dataset by considering the type of the attribute-label relationship in the data-generating process. For example, let us consider two common constraints in prior work on independence between é and
spurious attribute: unconditional (Φ(x)) IL. A) (Veitch et al., 2021 Albuquerque et al. 2020 Ganin
et al. 2016) or conditional on the label (δ(x) 11 ) (Ghilary et al. 2016; Hu et al. 2019 Li
et al. 2018c d). Under the canonical graph in Figure 2a the unconditional constraint is true when
A IL Y (A E Aind) ) but not always for And (true only under Confounded shift). If the relationship
is Causal or Selected, then the conditional constraint is correct. Critically, as Veitch et al. (2021)
show for lingle-attribute graph, the conditional constraint is not always better; it is an incorrect constraint (not satisfied by X.) under Confounded setting. Further, under the canonical graph [with E-X, edge] from Figure 2b] none of these constraints are valid due to a correlation path between X. and E. This shows the importance of considering the generating process for a dataset.
Inferring attributes-label relationship type. Whether an attribute belongs to Aind or Aind can be learned from data (since Aind IL Y). Under some special conditions with the graph in Figure 2a assuming all attributes are observed and all attributes Ainz are of the same type- -e can also identify the type of A ind shift: Y IL E|A implies Selected; if not, then X 1L Aind A And Y implies Causal, otherwise it is Confounded. In the general case of Figure 2b) however, it is not possible to differentiate between Acause Acomf and Aset using observed data and needs manual input. Fortunately, unlike the full causal graph, the type of relationship between label and an attribute is casier to obtain. For example, in text toxicity classification, toxicity labels are found to be spuriously correlated with certain demographics (Annd) ) (Dixon et al., 2018 Koh et al., 2021 Park et al. 2018); while in medical applications where data is collected from small number of hospitals, shifts arise due to different methods of slide staining and image acquisition (Aind) ) (Koh et al. [021 Komura & Ishikawa 2018 Tellez et al. 2019). Suppl. A contains additional real-world examples with attributes.
3.2 INDEPENDENCE CONSTRAINTS DEPEND ON ATTRIBCTE++LBEL RELATIONSUIIP
We list the independence constraints between (Xe, Y, under the canonical graphs from Figure 2 which can be used to derive the correct regularization constraints to be applied on & (Theorem 2.1
Proposition 3.1. Given causal DAG realized by specifying the target-attribute relationship in Figure 2a the correct constraint depends on the relationship of label Y with the attributes A. As shown, A can be split into A And' Aind and E, where Aind can be further split into subsets that have a causal (Acause), confounded ), selected (Asel) relationship with Y (And = A U Acon, U Agei). Then, the (conditional) independence constraints X should satisfy are,
1. Independent: X. TL Aindi X. TL E; Xx TL AmndlY: Xe 11 ALnn E; X 11 Aind Y, E
2. Causal: X 1L Acause IY; X. .II. E; X. ILI Acause| |Y, E.
3. Confounded: X .LL Aconf; X, .1L E; X, .1L. Acon
4. Selected: X, LL AselYY; X, .LL AYalY,
However, in the presence of a correlation between E and X. (Graph 2b), only the constraints conditioned on E hold true.
Corollary 3.1. All the above derived constraints are valid for Graph 2a
Corollary implies that if we are not sure about F-X. correlation, F-conditioned constraints should be used. By considering independence constraints over attributes that may represent any observed variable, our graph-based characterization unites the single-domain (group-wise) (Sagawal et al. 2020) and multi-domain generalization tasks. Whether attributes represent auxiliary attributes, group indicators, or data sources, Proposition 3.1 provides the correct regularization constraint.
Note that for Selected to satisfy the assumptions of Theorem 2.1 implies that X, is fully predictive of Y that the noise in Y-x; relationship is independent of the features driingg the selection process.
5

---

Published as a conference paper at ICLR 2023
3.3 A FIXED CONDITIONAL INDEPENDENCE CONSTRAINT CANNOT WORK FOR ALL SHIFTS
Combining with Theorem 2.1 Proposition 3.1 shows that the necessary constraints for a risk-invariant predictor's 8 representation p are different for different types of attributes. This leads to our key result: under multi-attribute shifts, a single (conditional) independence constraint cannot be valid for all kinds of shifts. Remarkably, this result is true even for single-attribur shifts: any algorithm with a fixed conditional independence constraint (c.g.. as listed in Table (rretton et al., 2012 Arjovsky et al.
2019 Li et al. 2018b; Sun & Saenko, 2 cannot work for all datasets.
Theorem 3.1. Under the canonical causal graph in Figure aa, there exists no (conditional) independence constraint over (Xe; A, Y that valid for all realized DAGs as the type of multi- attribute shifts vary. Hence, for any predictor algorithm for Y that uses single (conditional) independence constraint over its representation ((X), A and Y, there exists a realized DAG 9 and corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor for distributions in Pg. where Pg is the set of distributions obtained by changing PYA
Corollary 3.2. Even when |A| = 1, algorithm using single independence constraint over (o( (), A, Y cannot yield a risk-invuriant predictor for all kinds of single-attribute shift datasets.
Corollary adds theoretical evidence for past empirical demonstrations of inconsistent performance of DG algorithms (Wiles et al. 2022: Ye et al.] [022))To demonstrate its significance, we provide OoD generalization results on a simple "slab" setup (Shah et al. (020) with three datasets (Causal, Confounded, and Selected shifts) in Suppl. F.2 We evaluate two constraints motivated by DG literature Mahajan al. I222 unconditional X. T AE,, and conditional on label X. TT A E. As predicted by Corollary 3.2 neither constraint obtains best accuracy on all three datasets (Table 6).
4 CAUSALLY ADAPTIVE CONSTRAINT MINIMIZATION (CACM)
Motivated by Sec. 3] we present CACM, algorithm that adaptively chooses regularizing constraints for multi-attribute shift datasets (full algorithm for any general DAG in Suppl. C It has two phases.
Phase I. Derive correct independence constraints. If a dataset's DGP satisfies the canonical graph, CACM requires a user to specify the relationship type for each attribute and uses the constraints from Proposition 3.1 For other datasets, CACM requires a causal graph describing the dataset's DGP and uses the following steps to derive the independence constraints. Let V be the set of observed variables in the graph except Y, and be the list of constraints.
1. For each observed variable V E V, check whether (Xe, c, V) are d-separated. Add X, TL V to C. If not, check if (Xe. V) are d-separated conditioned on any subset of the remaining observed variables in 2 = (Y) UY (V). For each subset Z with d-separation, add X. JL Viz to C.
Phase II. Apply regularization penalty using derived constraints. In Phase II, CACM applies those E(gs (ø(#:)) v) RegPenally, where É is cross-entropy loss. The regularizer optimizes for valid constraints over all observed variables V E V. Below provide the regularizer term for datasets following the canonical graphs from Figure = A). We choose Maximum Mean Discrepancy (MMD) (Gretton et al. 2012) to apply our penalty (in principle, any metric for conditional independence would work).
constraints as a regularizer to the standard ERM loss, 91, Φ = arg ming
Since A includes multiple attributes, the regularizer penalty depends on the type of distribution shift for each attribute. For instance, for A E Aind (Independent), to enforce Φ(x) TT A, we aim to minimize the distributional discrepancy between P()(x)) A = (a;) and P()(z) A = az). ). for all j values of A. However, since the same constraint is applicable on F,, it is statistically efficient apply the constraint on E (if available) as there may be multiple closely related values of A in a domain (c.g.. slide stains collected from one hospital may be spread over similar colors, but not exactly the same). Hence, we apply the constraint on distributions P(ő(x) = -Eg) and P(o())E = E2) if E observed (and A may/may not be unobserved), otherwise we apply the constraint over A.
|Aind
Reg Reggllnlly Aind
Σ MMD(P(((2)) (as,in),, P(b(x)) (znnna))
(2)
For A E Agause (Causal), following Proposition 3.1
we consider distributions (( a..
v) and A aj. Y y). We additionally condition on E as there may be correlation

---

Published as a conference paper at ICLR 2023
between E and X (Figure 2b which renders other constraints incorrect (Corollary
3.1. We
similarly obtain regularization terms for Confounded and Selected (Suppl. C
RegPenaltyA Acauaa
MMD(P(6(()) a1, w), P(o(2))aa.cmeen aj.causes v))
WEY
The final RegPenalty is of penalties over all attributes, Reg RegPenalty E As A PenaltyA. where A are the hyperparameters. Unlike prior work (Makar et al. 2022 Veitch et al. 2021) we do
restrict ourselves to binary-valued attributes and classes.
CACM's relationship with existing DG algorithms. Table T shows common constraints used by popular DG algorithms. CACM's strength lies in adaptively selecting constraints based the causal relationships in the DGP. Thus, depending on the dataset, applying CACM for a single-attribute shift may involve applying the same constraint in MMD C-MMD algorithms. For example, in Rotated-MNIST dataset with E = Aind = rotation, the effective constraint for MMD, DANN, CORAL algorithms (6 TT E) is the same as CACM's s constraint Φ TT Aind for Independent shift.
EMPIRICAL EVALUATION
We perform experiments on MNIST, small NORB, and Waterbirds datasets demonstrate our main claims: existing DG algorithms perform worse multi-attribute shifts; CACM with the correct graph-based constraints significantly outperforms these algorithms; and incorrect constraints cannot match the above accuracy. While we provide constraints for all shifts in Proposition 3.1 our empirical experiments with datasets focus on commonly occurring Causal and Independent shifts. All experiments are performed in PyTorch 1.10 with NVIDIA Tesla P40 and P100 GPUs, and building on DomainBed (Gulrajani & Lopez-Par Paz, (02T) and DoD-Bench (Ye et al. 2022). Regularizing on model's logit scores provides better accuracy than d(x); hence we adopt for all our experiments.
5.1 DATASETS BASELINE DG ALGORITIMS
We introduce three new datasets for the multi-attribute shift problem. For all datasets, details of environments, architectures, visualizations, and setup generation are in Suppl. D.I
MNIST. Colored (Arjovsky et al. 2019) and Rotated MNIST (Ghifary et al. 2015) present Causal
(A (acuse color) and Independent (Annd = rolation) distribution shifts, respectively. We combine these to obtain aulti-attribute dataset with Acause and Aind (col + rot). For comparison, also evaluate on single-attribute Acause (Colored) and A (Rotated) MNIST datasets.
small NORB (LeCun al., 2004). This dataset was used by Wiles et al. (0022) to create challenging DG task with single-attribtte shifts, having multi-valued classes and attributes over realistic 3D objects. We create multi-attribus shift dataset (light+azi), consisting of causal connection, Avause =lighting, between lighting and object category Y: and Aind =azimauth that varies independently across domains. We also evaluate on single-attribute Acause (lightigg) and Aid (azimuthh datasets.
Waterbirds. We use the original dataset (Sagawa al. 2020 where bird type (water or land) (Y) is spuriously correlated with background (Acause) ). To create a multi-attribute setup, we add different weather effects (A,nd train and test data with probability p 0.5 and 1.0 respectively.
Baseline DG algorithms & implementation. We consider baseline algorithms optimizing for different constraints and statistics compare causal adaptive regularization: IRM (Arjovsky
2019). IB-ERM and IB-IRM (Abuja et al. 202T). VREx (Krueger et al.) (222T), MMD (Li et al.
et al.
2018b). CORAL (Sun Saenko (016), DANN (Greeton et al. 2012). Conditional-MMin (C-MMD) (Li et al.) (018b). Conditional-DANN (CDANN) (Li et al. 201 &d). GroupDRO (Sagawa
cct al.
2020), Mixup (Yan et al. 2020). MLDG eL al. (0188)) SagNet (Nam et al. (021), and
RSC (Huang et al. 2020). Following Bed (Gulrajani & Lopez-Paz 2021), a random search is performed 20 times over the hyperparameter distribution for 3 seeds. The best models obtained across the three seeds are used to compute the mean and standard error. We validation set that follows the test domain distribution consistent with previous work on these datasets (Arjovsky et al.
2019 Sagawa al. 2020 Wiles al. 2022
Ye et al. 2022). Further details are in Suppl.
7

---

Published as a conference paper at ICLR 2023
Table 2:
Colored Rotated MNIST: Accuracy on unseen domain for singe-attribute
(color, rotation) and multi-attribute (col + rot) distribution shifts; small NORB: Accuracy on unseen domain for single-attribute (lighting, azimuth) and autti-at (light + az1) distribution shifts. Waterbirds: Worst-group accuracy on unseen domain for single- and multi-attribute shifts.
Colored+RRtated MNIST
small NORB
Waterbirds
Algo
Accuracy
Accuracy
Worst-group accuracy
color
rotation
coltrot
lighting
azimath
light+azi
original
multi-attr
ERM
30.9 #1.6
61.9 #0.5
25.2 #1.3
65.5 =0.7
78.6 #0.7
64.0 #1.2
66.0 3.7
37.0 1.1
IB-ERM
27.8 =0.7
62.1 #0.8
41.2
66.0 #0.9
75.9 #1.2
61.2 #0.1
66.9 + 4.6
40.8 + 5.6
IRM
=0.1
61.2 #0.3
36.7
61.7
75.7 #0.4
61.7 #0.5
5.2
37.7 1 1.7
IB-IRM
40.1
61.4
49.3
60.7
77.6 20.3
61.2
7.7
46.9 6.5
VREx
31.3
62.1
20.4
61.7
77.6
68.8 62.5
38.1
MMD
29.7
0..5
24.1 #0.6
66.6
76.7
62.5
68.1
4.4
CORAL
20.8
62.5 #0.7
23.5
6..7
77.2
62.9 #0.3
4.86
5..1
DANN
20.8
#0.7
32.0
64.6
78.6
60.8 #0.7
5.6 1.8
C-MMD
20.2
#0.4
37.2
60.8
71.9
60.0
52.3 1.2
CDANN
30.8
60.2
32.2
64.9
77.3
60.8 =0,9
3.3
4..
DRO
30.9
60,9
25.3
60.5
77.1 #1.0
62.3
70.4
53.1 +.2
Mixup
21.1
20.6
21.1
66.2
80.4 #0.5
$1.
64.7 3.9
MLDG
30.0
60.6
20.7
60.7
77.9 20.5
60.6
70.8
1.7
SagNet
20.
60.7 20.7
23.7
61.9
76.1
62.2
69.1 1.0
40.6
RSC
29.1
62.3 #0.4
22.8 +0.3
62.4 #0.4
75.6 #0.6
61.8 #1.3
64.6 6.5
40.9
CACM
70.4
62.4 #0.4
54.1 #1.3
80.4
80.5
69.6 #1.6
84.5
70.5 1.1
Table 3: small NORB Causal shift. Comparing X, LL A Acause |Y, E with incorrect constraints.
Table 4: Comparing X. TT Acause Y. E and X .L Acause Y for Causal shift in MNIST and small NORB. The constraint implied by E-X, correlation (Fig. 2b Prop. 3.1 affects accuracy.
Constraint
Accuracy
X.
Acause
72.7 + 1.1
X.
_1
Acause
E
76.2 + 0.9
Constraint
MNIST
small NORB
X.
11
Acause
Y
79.7 + 0.9
X. 11 Acause Y
69.7 + 0.2
79.7 = 0,9 85.4 = 0.5
X.
11 Acause YY, F.
85,4 + 0.5
X. 11 Acause
Y,E
70.4 0.5
5.2 RESULTS
Correct constraint derived from the causal graph matters. Table 2 shows the accuracy on test domain for all datasets. Comparing the three prediction tasks for MNIST and small NORB, for all algorithms, accuracy on unseen test domain is highest under Aind shift and lowest under two- attribute shift (Aind U Acause ), reflecting the difficulty of multi-attribute distribution shift. On the two-attribute shift task in MNIST, all DG algorithms obtain less than 50% accuracy whereas CACM obtains 5% absolute improvement. Results on small NORB dataset similar: CACM obtains 69.6% accuracy on the two-attribute task while the nearest baseline is MLDG at 64.2%.
On both MNIST and small NORB, CACM also obtains highest accuracy on the Acouse task. On MNIST, even though IRM and VREx have been originally evaluated for the Color-only (Aoause task, under an extensive hyperparameter sweep recommended past work (Gulrajani & Lopez-Pat 2021 Krueger et al. 2021: Ye et al. 2022). we find that CACM achieves substantially higher accuracy (70%) than these methods, just units lower than the optimal 75%. While the Aind task relatively easier, algorithms optimizing for the correct constraint achieve highest accuracy. Note that MMD, CORAI,, DANN, and CACM are based on the same independence constraint (see Table []. As mentioned in Section 4 we use the the domain attribute E for CACM's regularization constraint for Ann task, for full comparability with other algorithms that also use E. The results indicate the importance of adaptive regularization for generalization.
Table 2 also shows the OoD accuracy of algorithms on the original Waterbirds dataset (Sagawa et al. 2020) and its multi-attribute shift variant. Here evaluate using worst-group accuracy consistent with past work (Sagawa et al., 2020: Yao et al. (202).. We observe that on single-attribus (Arause as well as multi-atrribute shift, CACM significantly outperforms baselines (~6% absolute improvement).
8

---

Published as a conference paper at ICLR 2023
Incorrect constraints hurt generalization. We now directly compare the effect of using correct versus incorrect (but commonly used) constraints for a dataset. To isolate the effect of a single constraint, we first consider the single-attribute shift on Acause and compare the application of different regularizer constraints. Proposition 3.1 provides the correct constraint for A X, LL Acause |Y, F. In addition, using d-separation following invalid constraints, X. TT Acause E. X. TT Acause. Without knowing that the DGP corresponds Causal shift, one may apply these constraints that do not condition on the class. Results on small NORB (Table show that using the incorrect constraint has an adverse effect: the correct constraint yields 85% accuracy while the best incorrect constraint achieves 79.7%. Moreover, unlike the correct constraint, application of the incorrect constraint is sensitive to the À (regularization weight) parameter: as À increases, accuracy drops to less than 40% (Suppl. E.3
Causal-realized DAG from Figure 3 we the
Figure 7
Comparing these constraints on small NORB and MNIST (Table reveals the importance of making the right structural assumptions. Typically, DG algorithms assume that distribution of causal features X. does not change across domains (as in the graph in Fig. 2a). Then, both X, TT Acause |Y, E and X. T1 Acause |Y should be correct constraints. However, conditioning on both Y and E provides 5% point gain over conditioning on Y in NORB while the accuracy is comparable for MNIST. Information about the data- generating process explains the result: Different domains in MNIST include samples from the same distribution whereas small NORB domains are sampled from a different set of toy objects, thus creating a correlation between X, and F, corresponding to the graph in Fig. 2b Without information on the correct DGP, such gains will be difficult.
Finally, we replicate the above experiment for the multi-attribute shift setting for small NORB. To construct an incorrect constraint, we interchange the variables before inputting to CACM algorithm (Aind gets used as Acause and vice-versa). Accuracy with interchanged variables (65.1 + 1.6) is lower than that of correct CACM (69.6 1.6). More ablations where baseline DG algorithms are provided CACM-like attributes as environment are in Suppl. E.4
6 RELATED WORK
Improving the robustness of models in the face of distribution shifts is a key challenge. Several works have attempted to tackle the domain generalization problem (Wang et al.) [021 Zhou et al.
2021) using different approaches- data augmentation (Cubuk et al. | 2020; He et al. 2016 Zhu
et al. 2017). and representation learning (Arjovsky et al. 2019 Deng et al. 2009 Higgins et al. 2017) being popular ones. Trying to gauge the progress made by these approaches, Gulrajani and Lopez-Paz (Gulrajani & Lopez-Paz 2021) find that existing
ofte-oof-theart DG algorithms do not improve over ERM. More recent work (Wiles et al. 2022 Ye et al. 2022) uses datasets with different single-attribur shifts and empirically shows that different algorithms perform well over different distribution shifts, but no single algorithm performs consistently across all. We provide (1) multi- attribute shift benchmark datasets; (2) a causal interpretation of different kinds of shifts; and (3) an adaptive algorithm to identify the correct regularizer. While we focus on images, OoD generalization on graph data is also challenged by multiple types of distribution shifts (Chen al. 2022).
Causally- mottivvted learning. There has been recent work focused on causal representation learn- ing (Arjovsky et al., 2019 Krueger et al.) 2021 Locatello al. 2020 Schölkopf et al. 2021 for OoD generalization. While these works attempt to learn the constraints for causal features from input features, we show that it is necessary model the data- gatt-generating process and have access to auxiliary attributes to obtain a risk risk-invariant predictor, especially in multi-attribute distribution shift setups. Recent research has shown how causal graphs can be used to characterize and analyze the different kinds of distribution shifts that occur in real-worke settings (Makar et al.) 2022 Veitch et al.) 202I). Our approach is similar in motivation but we extend from single-domain, single-attribus setups in past work to formally introduce multi-attribute distribution shifts in more complex and real-world settings. Additionally, we do not restrict ourselves to binary-valued classes and attributes.
7 DISCUSSION
We introduced CACM, an adaptive OoD generalization algorithm to characterize multi-atrribute shifts and apply the correct independence constraints. Through empirical experiments and theoretical analysis, we show the importance of modeling the causal relationships in the data- data-generann generating process. That said, our work has limitations: the constraints from CACM are necessary but not sufficient for a risk-invariant predictor (e.g., unable to remove influence of unobserved spurious attributes).

---

Published as a conference paper at ICLR 2023
8 ETHICS AND BROADER IMPACT STATEMENT
Our work on modeling the data-generating process for improved out-of-distribution generalization is an important advance in building robust predictors for practical settings. Such prediction algorithms, including methods building representation learning, are increasingly key element of decision- support and decision-making systems. We expect our approach to creating a robust predictor to be particularly valuable in real world setups where spurious attributes and real- world multi-attribute settings lead to biases in data. While not the focus of this paper, CACM may be applied to mitigate social biases (e.g., in language and vision datasets) whose structures can be approximated by the graphs in Figure [ Risks of using methods such as CACM, include excessive reliance or false sense of confidence. While methods such CACM ease the process of building robust models, there remain many ways that an application may still fail (e.g., incorrect structural assumptions). AI applications must still be designed appropriately with support of all stakeholders and potentially affected parties, tested in a variety of settings, etc.
REPRODUCIBILITY STATEMENT
We provide all required experimental details in Suppl. D including dataset details, training details and hyperparameter search sweeps. We additionally submit our code part of supplementary material which can be used to reproduce the experiments. We provide a demo notebook for prediction using CACM in the DoWhy library. We provide proofs for all our theoretical results in Suppl.
10 ACKNOWLEDGEMENTS
We thank Abhinav Kumar, Adith Swaminathan, Yiding Jiang, and Dhruv Agarwal for helpful feedback and comments on the draft. We would also like to thank the anonymous reviewers for their valuable feedback.
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, -hrnsChhiittpphe Gagnon- Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL htt.::ttp/////m..... /tctps::/<peerev/ lw. www.net/forie dd= d=c1sssFFLffF
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H. Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization, 2019. http::://arrivvor////aa//
02893
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi. Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, Quanzheng Li, Farhad Ghazvinian Zanjani, Sveta Zinger, Keisuke Fukuta, Daisuke Komura. Vlado Ovtcharov, Shenghua Cheng, Shaoqun Zeng, Jeppe Thagaard, and Geert Litjens. From detection of individual metastases to classification of lymph node status at the patient level: The camelyon17 challenge. IEEE Transactions Medical Imaging, PP:1-1, 08 2018. doi: 10. 1091TM99.2200 2018. 1.277305
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metries for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, WWW '19, pp. 491-500, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 478145066 http:/pppe///a.ooo//000101113/10 333777739
doi: 10.1 00.11453308360.33111
https::/gittubb.cmm
10

---

Published as a conference paper at ICLR 2023
Yongqiang Chen, Yonggang Zhang, Han Yang, Kaili Ma, Binghui Xie, Tongliang Liu, Bo Han, and James Cheng. Invariance principle meets out-of-distribution generalization on graphs. ArXiv, abs/2202.05441, 2022.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), ,, Advances in Neural Information Processing Systems, volume 33, pp- 18613-18624. Curran Associates, Inc., 2020. UtL www.::eiiiiiinnnno...o..e.....ee
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. http:://2222//eeeeeeeee200011111
2009. doi: 10. 109/C/PR.2009. 5206848.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference Al, Ethics, and Society, AIES 18, pp. 67-73, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450360128. doi: 10.1145/3278721132777777 URL https 3278729
7/doi.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200) 2016.
M. Ghifary, W. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2551 25599, Los Alamitos, CA, USA, dec 2015. IEEE Computer Society. doi: 10. 10.1100//CCV.200 2015. 293. htLhttts:///// doi wwt.:/eoooa..eeeeec.....ooor///. /tt.://.............////////////
Muhammad Ghifary, David Balduzzi, W Bastiaan Kleijn, and Mengjie Zhang. Seatter component analysis: A unified framework for domain adaptation and domain generalization. IEEE transactions on paltern analysis and machine intelligence, 39(7): (9(7))14144144) 1414-1430, 2016.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.
A kernel two-sample test, Journal of Machine Learning Research, 13(25):723-77, 2012. URL http:// r. .rg. wwwpr//pppppra.aeee /varss/vl333 httl////////een////e/
html
Ishaan Gulrajani and David Lapez-Paz. In search of lost domain generalization. ArXiv, abs/2007.01434 2021. 2021.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR.2066.
Irina Higgins, Loie Mathee, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multidomain discriminant analysis. In Uncertainty in artificial intelligence: proceedings of the... conference. Conference on Uncertainty in Artificial Intelligence, volume 35. NIH Public Access, 2019.
Zeyi Huang, Haohan Wang, Eric P. Xing, and Dong Huang. Self-challenging improves cross- cross-domain generalization. In ECCV, 2020.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard L. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: benchmark of in-the-wild distribution shifts. In ICML, 2021.
11

---

Published as a conference paper at ICLR 2023
Daisuke Komura and Shumpei Ishikawa. Machine learning methods for histopathological image analysis. Computational and Structural Biotechnology Joumnl, 16:34 42, 2018. ISSN 2001-0370. doi: http:://bbiior////..0006.....bbt m/ https//eeee/atticceeee///////0/0 7300867
URL htt.t:ttp/////mm wttp///w.icnceeeee.ttt...m
David Krueger, Ethan Caballero, Jeern- Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang. Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrap- olation (rex). In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 5815-5826. PMLR, 18-24 Jul 2021. rueger21a. html
URL https: ptttps://proceeclieee
mlr. www.prrsssssee 0ress/v139//
Y. LeCun, Fu Jie Huang, and L. Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Puttern Recognition, 2004. CVPR 2004., volume 2, pp. I1-104 Vol.2, 2004.
doi: 10.1 1L11099CCPR..204411111111
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 201 8a. httpt::ts:///33 aaai iaaaai.oo org/ ndex. Atmp//AATT.O http://AAA/aratccee/eeeee 99
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex Kot. Domain generalization with adversarial feature learning. In 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5400-5409, 2018b. doi: 10. 10.11099CPR..21..00055.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Secood AAAI Conference Artificial Intelligence, 2018c.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings the European Conference on Computer Vision (ECCV), pp. 624-639, 2018d.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. ICML'20. JMLR.org. 2020.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7313-7324. PMLR, 18- 24 Jul 2021. URL https::/ppocceeeaaassss
mlr prwsspesspppp v wttps///v19//aaaaaaaaaa
html
Maggie Makar, Ben Packer, Dan Moldovan, Davis Blalock, Yoni Halpern, and Alexander D' Amour. Causally motivated shortcut removal using auxiliary labels. In Gustau Camps- Valls, Francisco R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 739-766. vl 5mma://aa1mmmaaaa2aaaaaa
PMLR, 28-30 Mar 2022. URL https: httts:://proceeeee pww.//dooeeeeeeooooooooocoooo..e ngs. ml www.srressvvv.
html
Hyeonseob Nam, Hyunjae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8686-8695, 2021.
Ji Ho Park, Jamin Shin, and Pascale Fung. Reducing gender bias in abusive language detection. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2799-2888 Brussels, Belgium, Novemerr 2018. Association for Computational Linguistics. doi: 10.18653/w1/DI33
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithmss Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA. 2017. ISBN 978 978-------2200073331- URL houltttpss.. /ttmt/mttttsssseeeeeteteetssssst mit www.eduuuliiie bbokkskokkkeeeeeeee /
12

---

Published as a conference paper at ICLR 2023
Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. Proceedings of the International Conference of Machine Learning (ICML) 2020, 2020.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https
//openreview. / forumiid I
B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, Goyal, and Y. Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)612-634, 2021. doi: 10. ROC.2021.3058954 URL
peattps://ieeexpliee Uttttttss/sttttttsstsseeeeeeeeee
ieee org/ stamp/ /ttp//ttappppp../ jsp?ar
number=9363929
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Gang Hua and Hervé Jégou (eds.), Computer Vision ECCV 2016 Workshops, pp. 4434444444 450, Cham, 2016. Springer International Publishing. ISBN 978-3-319-49409-8.
David Tellez, Geert Litjens, Péter Bándi, Wouter Bulten, John-Mell Bokhorst, Francesco Ciompi, and Jeroen van der Laak. Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology. Medical Image Analysis, 58: 18440,4 2019. ISSN 1361-8415. doi: https:///oooorgg/0001166.....ee
http:::.mmmiia2221111101... 101544. URL https
www. www.sciiencdiic rect com httpa//cc.c...c......../c.ooooaa
Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations text classification. In A. Beygelzimer, Y. Dauphin, Liang, and
J. Wortman Vaughan (eds.), Advances in Neurul Information Processing Systems, 2021. URL
net forum? wiepummmid-Boooe dddddd-xxxpiiiB
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-200 1-001, California Institute of Technology, 2011.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. arXiv preprint arXiv:2103.0309) 2021.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. fine- fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022. URL httpt:ttp//cccccm httptpp:///opn.eiew...o. wwt.:/ewwwweeo.rr..eeeee ?id S.d-D14LetuLdyls
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. ArXiv,
2020.
Huaxiu Yao, Yu Wang. Sai Li, Linjun Zhang, Weixin Liang, James Zou, and Chelsea Finn. Improving out-of-distribution robustness via selective augmentation. arXiv preprint arXi::2201.0029) 2022.
Nanyang Ye. Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In CVPR, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 40(6)) 2018. doi: 10. 199////M1..011..723000.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. arXiv e-prints, pp. arXiv-2103, 2021.
Jun Yan Zhu, Tacsung Park, Phillip Isola. and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. pp. 2242-2251, 10 2017. doi: 10.1109//C 22017. 244.
13

---

Published as a conference paper at ICLR 2023
A PRESENCE OF AUXILIARY ATTRIBUTE INFORMATION IN DATASETS
Unlike the full causal graph, attribute values as well as the relationships between class labels and attributes is often known. CACM assumes access to attribute labels only during training time, which are collected as part of the data collection process (e.g., metadata with training data (Makar al. 2022)) ). We start by discussing the availability of attributes in WILDS (Koh et al. (022)) set of real-world datasets adapted for the domain generalization setting. Attribute labels available in the datasets include, the time (year) and region associated with satellite images in FMoW dataset (Christie) et al. (018)) for predicting land use category, hospital from where the tissue patch was collected for tumor detection in Camelyon17 dataset (Bandi et al.] (2018) and the demographic information for
CivilComments dataset (Borkan al. 2019). (Koh et al.) (021) create different domains in WILDS using this metadata, consistent with our definition of E E A as a special domain attribute.
In addition, CACM requires the type of relationship between label Y and attributes. This is often known, either based on how the dataset was collected or inferred based on domain knowledge or observation. While the distinction between Aind and A Annd can be established using a statistical test of independence on a given dataset, in general, the distinction between Acause Aget and Acon/ within And needs be provided by the user. As we show for the above datasets, the type of relationship can be inferred based on common knowledge or information on how the dataset was collected.
For FMoW dataset, time can be considered an Independent attribute (Aind) since it reflects the time at which images are captured which not correlated with Y: whereas region is a Confounded attribute since certain regions associated with certain Y labels are over-represented due to ease of data collection. Note that region cannot lead to Causal shift since the decision to take images in a region was not determined by the final label nor Selected for the same reason that the decision was not taken based on values of Y. Similarly, for the Camelyon17 dataset, it is known that differences in slide staining or image acquisition leads to variation in tissue slides across hospitals, thus implying that hospital is an Independent attribute (Aind) ) (Koh et al., 2021 Komura & Ishikawa 2018 Tellez et al. 2019); As another example from healthcare, study in MIT Technology Review" disiussseeessss biased data where a person's position (Acoon ) was spuriously correlated with disease prediction as patients lying down were more likely be ill. As another example, (Sagawa et al. 2020)
adapt MultiNLI dataset for OoD generalization due to the presence of spurious correlation between negation words (attribute) and the contradiction label between "premise" and "hypothesis" inputs. Here, negation words are result of the contradiction label (Causal shift), however this relationship between negation words and label may not always hold. Finally, for the CivilComments dataset, we expect the demographic features to be Confounded attributes as there could be biases which result in spurious correlation between comment toxicity and demographic information.
To provide examples showing the availability of attributes and their type of relationship with the label, Table 5 lists some popular datasets used for DG and the associated auxiliary information present metadata. In addition to above discussed datasets, weinclude the popularly used Waterbirds dataset (Sagawa et al.] (0020) where the type of background (land/water) is assigned to bird images based on bird label; hence, being Causal attribute (results on Waterbirds dataset are in Table 2
Table 5: Commonly used DG datasets include auxiliary information.
Dataset
Attribute(s)
Y A relationship
FMoW-WILDS (Koh et al. 2021
time region hospital
A
Himelyon/7-WILDS Koh et al.
2021
Aind
Waterbirds (Sagawa et al. 2020
background (land/water)
A cause
MultiNLI
Sagawa et al.
2020)
negation word demographic
A cause
CivilComments WILD-IILDD Koh et al. 2021
Acon
httpts//www..eeeooooyerreerrr/or http:///aa2///2/////22////////// (330299macch ne- learning---... ng-a fai Ted- covi d-ioooooooospppaaaaiiiaaaa-aeee- pandemicncd c/
14

---

Published as a conference paper at ICLR 2023
B PROOFS
B.1 PROOF OF THEOREM 2.1
Theorem 2.1. Consider causal DAG 9 over (Xe, X, A.Y) and corresponding generated dataset ( ais Yi Bi)i== where X. is unobserved. Assume that graph G has the following property: X is defined as the set of all parents Y (Xe-) Y); and Xx, together cause (Xe - X, and A + X). The graph may have any other edges (see, e.g., DAG in Figure (bj). Let Pg be the set of distributions consistent with graph G, obtained by changing P(A(Y) but not P(Y Then the conditional independence constraints satisfied by X. are necessary for (cross-entropy) risk- invariant predictor over Pg. That is, if a predictor for Y does not satisfy any of these constraints, then there exists data distribution P' E Pg such that predictor's 's risk will be higher than its risk in other distributions.
Proof. We consider X, x.. A as random variables that are generated according the data- generating process corresponding to causal graph G. We assume that X represents all the parents of Y. X also causes the observed features X but X may be additionally affected by the attributes A. Let ú g(x) be candidate predictor. Then g (X) represents a random vector based on deterministic function of X.
Suppose there is independence constraint & that is satisfied by X. but not g Below we show that such a predictor g is not risk-invariant: there exist two data distributions with different P(A(Y) such that the risk of g is different for them.
Without loss of generality, we can write g(x) as,
g(x) (g(x=)/(<x.) h(xx)) g((x, (x).x))((.)))) V'
where is an arbitrary, non-zero, deterministic function of the random variable X.. Since X. satisfies the (conditional) independence constraint e and h is deterministic function. h(X,) also satisfies v. Also since the predictor g(X) does not satisfy the constraint 1. it implies that the random vector g'(X: X.) cannot satisfy the constraint th. Thus, g' 3X(X, X.) cannot be a function of X, only; it needs to depend on X too. Since X has two parents in the causal graph, X, and A, this implies that 8((X, (x) must depend on A too, and hence g' (X, X.) and A are not independent.
Now, let construct two data distributions P1 and P2 with the same marginal distributions of P(Y), P(A) and P( P(X.), such that P(A changes across them. Note that | stays invariant because of the independent and stable causal mechanism property, i.e., Pi Xx.) X..)) For these two data distributions, change in P(A(Y) implics a change in P( A), AA.iie. i... AA( # P2 [A), since P(Y A) P(A P(Y)) Also, since g' (X, X.) and A are not independent,
(x)))) will change, i.e., PL g (X, (X,X,) X.)) + (g (X, X.)).
The risk over any distribution P can be written as (using the cross-entropy loss),
Rp(g) = EEl o (x. X (xx)))
y logg (X, (xX)( = -p Σ
(4)
y
= EEp] >
y ylogg (X,X.) - Epl > y log hX)))
The risk difference is,
Rp, (g) - Rp, (g)
= =EAl Σ y log o xxxx)) - Ep, Σ y log (X,X.)) + E..
Σ h (x.) - Epp Σ y log hi ((Xx))
E,, Σ y log g' (X, X.) - Ey,, Σ y log g (X, Xx)
'In practice. the constraint may be evaluated on an intermediate representation of such that g can be written x(X) 91 (o(X) where & denotes the representation function. However, for simplicity, we assume
it is applied on x(X).
15

---

Published as a conference paper at ICLR 2023
where the third and fourth terms cancel out because Pi Y) P2(Xx,)) Y) and thus the risk of (())) Xx) is the same across P1 and P2. However, the risk for (X.X.)) is not the same since Pi (x.))) Pa X, X))). Thus the absolute risk difference is non-zero,
(o) Rr, (g) > 0
(5)
and g is risk-invariant predictor. Hence, satisfying conditional independencies that X. satisfies is necessary for a iivarinvar predictor.
Remark. In the above Theorem, we considered the case where P(A(Y) changes across distri- butions. In the case where A and Y are independent, P(AoY) P(A) and thus P(A) would change across distributions while A)Y = P(Y) remained constant. Since g' g'(X, Xx) depends on A and X., obtain (Y (X, x))) g' X,)). However, the risk difference can still be non-zero since (A) F (2) and the risk expectation Ep Ep(Ey log g (X, Xx)] is over P(Y, Xe, A).
B.2 PROOF OF PROPOSITION 3.1
Proposition 3.1. Given a causal DAG realized by specifying the target-attributes relationship in Figure Za the correct constraint depends the relationship of label Y with the attributes A. As shown, can be split into Ann Aind and E, where And can be further split into subsets that have causal (Acause), confounded (Aconf). selected (Asel) relationship with Y (Aind Acause U Aconf U Agel). Then, the (conditional) independence constraints X should satisfy are,
1. Independent: X IL Aindi X TL E; Xx TT Amnal: Xe 11 AmadEE E; X 11 Aind YY
Causal: X 1L. AcauselY; X 1L. E; X. 11 Acause (Y, E
3. Confounded: X 1L Aconfi X. 11 E; X 11 Acon
4. Selected: Xo TT A Xo 1L Asel IY, YY
Proof. The proof follows from d-separation (Pearl, 2009) on the causal DAGs realized from Figure 2a For each condition, Independent, Causal, Confounded and Selected, we provide the realized causal graphs below and derive the constraints.
X
Aind
XG
Acuse
x
E
x
E
(a) Independent shift
(b) Causal shift
C
s=1
X
Y
Aum!
X
A sel
x
(c) Confounded shift
(d) Selected shift
Figure 3: Causal graphs for distinct distribution shifts based on A relationship.
Independent: As we in Figure 3a have collider X on the path from X, Aind and X, to E. Since there a single path here, we obtain the independence constraints X, _L A,nd and X. 1.I. F. Additionally, that conditioning Y or E would not block the path from
16

---

Published as conference paper at ICLR 2023
X. to Annd which results in the remaining constraints: X. TT Aind (Y; X TT Ain |E and
Xe 1T Aind E. Hence, we obtain,
X TT Aind X TT E; X. TT And Xe TT Aind E Xe TT Aind Y
From Figure 3b we see that while the path X + X + Acause from X to Acause contains collider X, X Acause due the presence of node Y as a chain. By the d-separation criteria, X, and pause are conditionally independent given Y = X. LI Acause, Additionally, conditioning on E. is valid since E does not appear as a collider on any paths between X. and Acouse X. TT Acause E. We get the constraint X. TT E' since all paths connecting
Causal:
X. to contain collider (collider X in X. 7 X 7 Acause + E, collider Acause in X Y Acause + E). Hence, obtain,
X T1 Arause YY X. TT LL X. 11 Avause |Y,
Confounded: From Figure 3c see that all paths connecting X, and Aconf contain a collider (collider X in X + X + Aconf collider Y in X - C + Aconf). Hence, X TT Aconf Additionally, conditioning on valid since does not appear as collider on any paths between X. and Aronf = X _L Acon JE. We get the constraint X. 1L E since all paths connecting X. and also contain a collider (collider in X. + X + Acos 7 E, collider Y in X. +Y +-++0 +0 Acons E). Hence, obtain,
X. Aconf X TT X TT Aconf |
Selected:
For the observed data, the selection variable is always conditioned on, with S indicating inclusion of sample in data. The selection variable S is collider in Figure 3d and we condition on Y breaks the edge X, + Y. and hence all paths between X and Aset now contain collider (collider X in X + X + Asel = X TT. Aset IY. Additionally, conditioning on valid since does not appear
it. Hence, 4L Asel Conditioning
collider on any paths between
X. and Asel
X. Aget |Y, E. Hence, we obtain,
X TT Aget \Y; TT Agel (Y, E
B.2.1 PROOF COROLLARY 3.0.1
Corollary 3.1. All the above derived constraints are valid for Graph 2a However, the presence of correlation between E and X. (Graph 2b), only the constraints conditioned on E hold true.
there correlation between X. and E, X. E. We can see from Figure 3 that in the presence of X. E correlation, XL Aind X, K4 Aind (Y (Ba), X. Arause. [Y [bb.. X, K A 3cb and X. J Asel Y (d)). Hence, conditioning on environment is required for the valid independence constraints.
B.3 PROOF OF THEOREM 3.1
Theorem 3.1. Under the canonical causal graph in Figure aa,,), there exists (conditional) independence constraint over (Xe, A,Y) that valid for all realized DAGs as the type of multi- attribute shifts vary. Hence, for any predictor algorithm for Y that independence constraint over its representation 6(X), A and Y, there exists a realized DAG & and a corresponding training dataset such that the learned predictor cannot be a risk-invariant predictor for distributions in Pg. where Pg is the set of distributions obtained by changing P(AYY..
single (conditional)
Proof. The proof follows from an application Proposition 3.1 and Theorem 2.1
First claim. Under the canonical graph from Figure 2 b), the four types of attribute shifts possible are Independent, Causal, Confounded and Selected. From the constraints provided for these four types of attribute shifts in Proposition 3.I it is easy to observe that there is single constraint that is satisfied across all four shifts. Thus, given data distribution (and hence, dataset) with specific types of multi-atie multi-attribute shifts such that X, satisfies (conditional) independence constraint w.r.t.
17

---

Published as a conference paper at ICLR 2023
subset of attributes A, S A, it is always possible to change the type of least one of the those attributes' shifts to create new data distribution (dataset) where the same constraint will not hold.
Second Claim. To prove the second claim, suppose that there exists a predictor for Y based single conditional independence constraint over its representation, ())((),, A,, Y) where A, S A. Since the same constraint is not valid across all attribute shifts, can always construct realized graph g (and a corresponding data distribution) by changing the type of at least one attribute shift A As, such that X. would not satisfy the same constraint 6 6()). Further, under this G, X, would satisfy a different constraint on the same attributes. From Theorem [2.] all conditional independence constraints satisfied by X under 9 are necessary be satisfied for risk-invariant predictor. Hence, for the class of distributions Pg. a single constraint-basee predictor cannot be risk-invariant predictor.
Corollary 3.2. Even when | =1 an algorithm using a single independence constraint over (x)X), ), Y') cannot yield risk-invariant predictor for all kinds of single-altribute shift datasets.
Proof. Given a fixed (conditional) independence constraint over predictor's representation, (x)(), A., Y), the proof of Theorem 3.1 relied on changing the target relationship type (and hence distribution shift type) for the attributes involved in the constraint. When |A] 1, the constraint is single attribute A, (X)) A, Y) and the same proof logic follows. From Propo- sition 3.T given a fixed constraint, we can always choose single-attribute shift type (and realized DAG such that the constraint is not valid for X Moreover, under 9, X. would satisfy different conditional independence constraint wrt the same attribute. From Theorem [.1] since the predictor does not satisfy conditional independence constraint satisfied by X., it cannot be risk-invariant predictor for datasets sampled from Pg.
C CACM ALGORITHM
We provide the CACM algorithm for a general graph & below (Algorithm
Algorithm 1 CACM
Input: Dataset (æ;; a4, y1 Y= causal DAG G
Output: Function g(x) gi (+()))) : X +
A < set of observed variables in G except Y, E' (special domain attribute)
C - O
>> mapping of A to A,
Phase I: Derive correct independence constraints
for A E A do
if (X,; A) are d-separated then
X. .IL. A is valid independence constraint
else if (Xe: A) are d-separated conditioned any subset A, of the remaining observed
variables in A \. {A}} U {Y} then
X. 1TL A Al is valid independence constraint
C[A] A. end if end for
Phase II: Apply regularization penalty using constraints derived for A E do X. TT A then
PregPeaatyA=3 = E
Σ |Al
E f P(1)(P)
else if A is in O then
A, C[A]
RegPenaltya = E|E EnEr, E!!I Ejsi MMDDPP(tt)))A,,0))11(((.))
end if end for
Reg = EAEA da ReggPenallyy
g1.0 ó = arg mi ((g(((())) v) + Reg RegPenalty
18

---

Published as a conference paper at ICLR 2023
Remark. If E is observed, we always condition on E because of Corollary 3.1
For the special case of Figure 2 CACM uses the following regularization penalty (RegPenalty) for Independent, Causal, Confounded and Selected shifts,
Reg Penaltyy.....
MMD(P((()))o..ndd) P(o()))
jzi
RegPenalty Acause
MMD(P(d(2) as causee v), P(ó(x)) (fj(aa)aj..cuuss jausse v))
NEY
j>i
[Acons
RegPenalty Acons
MMD P(d())))
jzi
Auel
RegPenalty A.e.
MMD(P(o(z))a...) (i,,st v). P(o()))aa) 8))
EE BEY
j>b
D EXPERIMENTAL DETAILS
D.1 DATASETS
MNIST. Rotated (Ghifary et al. 2015) and Colored MNIST (Arjovsky al. 2019) present distinct
distribution shifts. While Rotated MNIST only has Aind wrt. rotation attribute (R), Colored MNIST only has Acouse dataset with A Acuuse = {C} and Aind [R]. Each domain E, has a specific rotation angle r; and a specific correlation corr; between color C and label Y. Our setup consists of 3 domains:
color attribute (C). We combine these datasets obtain multi-attribute
E,, E2 E,, (training), Eg E &te (test). We define corr; P(Y 1|C = 1) P(Y 0|C 0)
E. our setup, ri = 15", 72 = 60", r3 = 90 and corri 0.9, corrz 0.8, corr3 = 0.1. All environments have 25% label noise, as in (Arjovsky et al. 2019). For all experiments
MNIST,
we use a two-layer perceptron consistent with previous works (Arjovsky et al. 2019 Krueger et al. 2021)
5
2
8
O
4
4
S
3
9
6
7
A
=
Y
Y=0
==1
Y=0
Y=1
(a)
(b)
(c)
Figure 4: (a), (b) Train and (c) Test domains for MNIST.
small NORB. Moving beyond simple binary classification, we use small NORB [LeCun et al. 2004), object recognition dataset, to create challenging setup with multi-valued classes and attributes over realistic 3D objects. It consists of images of toys of five categories with varying lighting, clevation, and azimuths. The objective is to classify unseen samples of the five categories. (Wiles et al., 2022) introduced single-attribute shifts for this dataset. We combine the Causal shift, causs lighting wherein there is correlation between lighting condition lighting; and toy
19

---

Published as a conference paper at ICLR 2023
category y4 and Independent shift, Ann azimuth that varies independently across domains, to generate our multi-attribute dataset light + azú. Training domains have 0.9 and 0.95 spurious correlation with lighting whereas there is no correlation in test domain. We add 5% label noise in all environments. We use ResNet-18 (pre-trained on ImageNet) for all settings and fine tune for our task.
Y=1
Y+2
Y
Y
Y=0
Y-1
Y=2
Y
Y=4
Y=0
Y+1
Y=2
Y
(a)
(b)
(c)
Figure 5: (a), (b) Train and (c) Test domains for small NORB.
Waterbirds. We use the Waterbirds dataset from Sagawa al. 2020). This dataset classifies birds as "waterbird" or "landbird", where bird type (Y) spuriously correlated with background (bgd) "waterbird" images are spuriously correlated with "water" backgrounds (ocean, natural lake) and "landbird" images with "land" backgrounds (bamboo forest, broadleaf forest). Since background selected based Y. Acause background. The dataset is created by pasting bird images from CUB dataset (Wah et al. (2111) onto backgrounds from the Places dataset (Zhou et al. [2018] 8). There 0.95 correlation between
bird type and background during training i.e., 95% of all waterbirds are placed against a water background, while 95% of all landbirds are placed against land background. We create training domains based on background ([Eerl = |Acause = 2) as Yao al. (2022). We evaluate using vorst-group error consistent with past work, where group is defined as (background, y). We generate the dataset using the official code from Sagawa et (2020) and
the same train- validation-test splits.
To create the multi-attribute shift variant of Waterbirds, we add weather effects (Aind ) using the Automold library We add darkness effect (darkness coefficient 0.7) during training with 0.5 probability and rain effect (rain type = 'drizzle', slant = 20) with 1.0 probability during test. Hence, Aind 33 (nno effect, darkness, rain)). Weather effect is applied independent of class label Y. Our training domains are based on background and we perform worst-group evaluation, same as the setup described above. Examples from train and test domains for multi-attribute shift dataset are provided in Figure 6
We use ResNet-50 (pre-trained on ImageNet) for settings consistent with past work (Sagawa et al. 2020: Yao et al., 2022). All models were evaluated at the best early stopping epoch (as measured by validation set), again consistent with Sagawa et al. (2020).
=
Y ==1
Y=0
Y=1
0
==1
(a)
Figure 6: (a), (b) Train and (c) Test domains for Waterbirds.
https://ggtthbb.com http::/www..oooooae/eeeeeeeeeeee -irp-:--oooaoooaaaaaaiiiooiiiii-
20

---

Published as a conference paper at ICLR 2023
D.2 IMPLEMENTATION DETAILS
All methods are trained using Adam optimizer. MNIST dataset is trained for 5000 steps (default in DomainBed (Gulrajani & Lopez-Paz 202T)) while Waterbirds and small NORB are trained for 2000 steps. Consistent with the default value in DomainBed, we use a batch size of 64 per domain for MNIST. For small NORB, we use batch size of 128 per domain and batch size of 16 per domain for Waterbirds.
Model Selection. We create 90% and 10% splits from each domain be used for training/evaluaatn and model selection (as needed) respectively. We use a validation set that follows the test domain dis- tribution consistent with previous work on these datasets (Arjovsky et al. 2019; Ye et al. 2022 Wiles ct al. 2022; Yao et al., 2022). Specifically, we adopt the test-domain validation from DomainBed for Synthetic, MNIST, and small NORB datasets where early stopping is not allowed and all models are trained for the same fixed number of steps to limit test domain access. For Waterbirds, we perform
early stopping using the validation set consistent with past work (Sagawa et al. 2020 Yao et al.
2022).
MMD implementation details. We use the radial basis function (RBF) kernel to compute the MMD penalty. Our implementation is adopted from DomainBed (Gulrajani Lopez-Paz, [202T). The kernel bandwidth is hyperparameter and we perform a sweep over the hyperparamaeter search space to select the best RBF kernel bandwidth. The search space for hyperparameter sweeps is provided in where 7 corresponds to l/bandwidth.
Table 9
CACM and baselines implementation details. We provide the regularization constraints for different shifts used by CACM in Section C For statistical efficiency, hyperparameter for MNIST and small NORB datasets. The search space for hyperparameters is given in Table
single À value as
In MNIST and NORB, we input images and domains (E = Aind to all baseline methods; CACM receives additional input Acause Hence, in the Independent single-attribute shift, CACM and all baselines have access to exactly the same information. In Waterbirds, since E is not defined in the original dataset, we follow the setup from Yao al. (2022) to create domains based on backgrounds. Here, we provide images and domains (E = back-rooud)) as input to all baselines except GroupDRO; to ensure fair comparison with GroupDRO, we follow [Sagawa et al. (2020) and provide groups as input based on (background, y), along with images. For CACM, we do not use background domains but provide the attribute Acause = background for the single-attribute dataset, and both Acouse background and Aind weather for the aulti-attribute shift dataset. Hence, for the single-shift Waterbirds dataset, all baselines receive the same information as CACM.
D.3 HYPERPARAMETER SEARCH
Following Bed Guirajani & Lopez-Paz 202T.
perform random search 20 times over the hyperparameter distribution and this process is repeated for total 3 seeds. The best models are obtained across the three seeds over which we compute the mean and standard error. The hyperparameter search space for all datasets and algorithms is given in Table
E RESULTS
E. SYNTHETIC DATASET
Our synthetic dataset is constructed based on the data-generating processes the slab dataset (Maha- jan et al.] |021] Shah et al.) 2020). The original slab dataset was introduced by (Shah et al.) (020) to demonstrate the simiplicity bias in neural networks as they learn the linear feature which is casier to learn in comparison to the slab feature. Our extended slab dataset, adds to the setting from (Mahajan) et al. 2021) by using non-binary attributes and class labels to create a more challenging task and allows us study DG algorithms in the presence of linear spurious features.
Our dataset consists of 2-dimensional input X consisting of features X, and A This is consistent with the graph in Figure where attributes and causal features together determine observed features
21

---

X, E 00225, 0.5) Published as a conference paper at ICLR 2023
X; concatenate X. and And generate X in synthetic setup. Causal feature Xe has non-linear "slab" relationship with while Ana has a linear relationship with Y. We create three different datasets with Causal (E.II,, Confounded (.1.2) and Selected (E.13) A And Y relationship respectively.
Implementation details. In all setups, X. is single-dimensional variable and has uniform distribution Uniform O0, across all environments. We use the default 3-layer MLP architecture from DomainBed and use mean difference (L2) instead of MMD as the regularization penalty given the simplicity of the data, We use a batch size of 128 for all datasets.
E.1.1 Causal SHIFT
We have three environments, E1, E2 & (training) and Es Ele (test). X has uniform distribution Uniform(0. [0, 1] across all environments.
0
if X 0.2)
if X. E 0.2,, 0.4)
2
if X. 004,
3
if if, 0.6, 0.8
4
Xf, 00.8, 1.0
with prob. abssy 1) with prob. p
A Acouse
Hence, we have a five-way classification setup (Y) 5) with multi-valued attributes. Following (Ma hajan al. 202T), the two training domains have p 0.9 and 1.0, and the test domain has p 0.0. We add 10% noise to Y in all environments.
E.1.2
Confounded SHIFT
We have three environments, E1, E2 E &ir (training) and E3 E &te (test). X. has uniform distri- bution Uniform[0. 1] across all environments. Our confounding variable has different functional relationships with Y and Acons which vary across environments.
E3 = {
with prob. = 0.25
{
1
with prob. 0.75
with prob. 0.75
CE =
with prob. 0.25
The function for Y given by.
if X, E 00, 0.25)
ytrue
2 if X. 00.5, 0.75) X. E [0.75, 1.0
Observed and Acons are functions of confounding variable and their distribution changes across environments as described below:
Vtrue + with prob. 0.9
FE =
ytrue with prob. 0.1
VFa = Utrue
2 with prob. = p
A Acons
:EE = 1.0, L.. = 0.9. = 0.8
0 with proh. 1
22

---

if Xe 00.25, 0.5) Published as a conference paper at ICLR 2023
E.1.3 Selected SHIFT
Selected shifts arise due selection effect in the data generating process and induce an association between Y and Aeet A data point is included in the sample only if selection variable S = holds; function of and Agei The selection criterion may differ between domains (Veitch 2021).
We construct three environments, E E2 E & (training) and E3 E Ete (test). X. has a uniform distribution Uniform (0, 1] across all environments. Our selection variable S is function of Y and Aset We add 10% noise to in all environments.
X Uniform [0, 1];
Aset (1,2,3,4)
The true function for Y is given by,
X E [0, 0.25)
Utrue
Xx E [0.5, 0.75) if X [0.75, 1.0)
The function used to decide the selection variable S (and hence the selection shift) varies across environments through the parameter p.
S =1 if
Ase + 4 with prob. p Ase - yy= with prob. = p iPE, = 0.,, = 1.0,PE, = 0.0
E.2 A FIXED CONDITIONAL INDEPENDENCE CONSTRAINT CANNOT WORK ACROSS ALL SHIFTS
Here. we compare the performance of two popular independence constraints in the literature Mahajan cct al. 2002I: unconditional X. TT A E, and conditional on label X. TLL A|Y, E (we condition on E for fully generality) on Synthetic Causal, Confounded and Selected shift datasets (Table 6
We train a model using ERM (cross-entropy) where the representation is regularized using either of the constraints. As predicted by Theorem 3.1 neither constraint obtains best accuracy all three datasets. The conditional constraint is better on Acause and A sel datasets, whereas the unconditional constraint is better Aconf consistent with Proposition 3.1 Predictors with the correct constraint are also more risk-invariant, having lower gap between train and test accuracy.
Table 6: Comparison of constraints X. TL A and X. TT A in Causal, Confounded and Selected shifts. X. IL A|Y, E is correct constraint for Causal and Selected shift but invalid for Confounded shift; while X, .L A is correct for Confounded but invalid for Causal, Selected.
Accuracy
Constraint
Acouse
Aconf
Agel
train
test
train
test
train
test
X. 1. A
96.5 + 0.2
62.4 5.7
81.1 2.0
67.1 1.7
96.4 + 0.4
72.3 0.9
X.
AY, E
+ 3.8
89.3 2,3
78.4 + 2.6
60.3 - 1.2
91.1 1.7
88.7 + 0.9
E.3 EFFECT OF VARYING REGULARIZATION PENALTY COEFFICIENT
To understand how incorrect constraints affect model generalization capabilities, we study the Causal shift setup in small NORB. From Theorem 3.I we know the correct constraint for Acause the following invalid constraint, X, Il Acause E.
X. 1. Acause [Y, E. In addition, We compare the performance of these two conditional independence constraints while varying the regularization penalty coefficient (A) (Figure 7). We perform evaluation across three setups
23

---

Published as a conference paper at ICLR 2023
with different spurious correlation values of training environments and have consistent findings. We observe that application of the incorrect constraint sensitive to À (regularization weight) parameter: as À increases, accuracy drops to less than 40%. However, accuracy with the correct constraint stays invariant across different values A.
100
80
Accuracy
60
CACM
Incurrect (0.90, 0.95)
20
CACM
Incorrect (0.85, 0.90)
CACM
Incurrect (0.80, 0.85)
0
0
20
40
60
80
100
penalty coeff (A)
Figure 7: Accuracy of CACM (X, TT Acause |Y, E) and incorrect constraint (x TT Acause E))
on small NORB Causal shift with varying À {1, 10, 100} and spurious correlation in training environments (in parantheses in legend).
E.4 PROVIDING ATTRIBUTE INFORMATION TO DG ALGORITHMS FOR A FAIRER COMPARISON
CACM leverages attribute labels to apply the correct independence constraints derived from the causal graph. However, existing DG algorithms only use the input features X and the domain attribute. Here we provide this attribute information to existing DG algorithms to create more favorable setting for their application. We show that even in this setup, these algorithms are not able to close the performance gap with CACM, showing the importance of the causal information through graphs.
E.4.1 SYNTHETIC DATASET
We consider our Synthetic dataset with Causal distribution shift where our observed features X = (Xer Acause Note that by construction X, one of our input dimensions already consists of A Acause Hence, all baselines do receive information about Acase in addition to the domain attribute E.
However, to provide a fairer comparison with CACM, we now additionally explicitly make Acauee available to all DG algorithms for applying their respective constraints by creating domains based on Acause in our new setup. Using the same underlying data distribution, we group the data (i.e., create environments/domains)) based on Acause i.e, each environment has samples with same value of Acoues
In this setup (Table third column), we see IB-IRM, DANN, and Mixup show significant improve- ment in accuracy but the best performance is still 14% lower than CACM. We additionally observe baselines to show higher estimate variance in this setup. This reinforces our motivation to use the causal graph of the data-generating process derive the constraint, as the attribute values alone are not sufficient. We also see MMD, CORAL, GroupDRO, and MIDG perform much worse than earlier, highlighting the sensitivity of DG algorithms to domain definition. In contrast, CACM the causal graph to study the structural relationships and derive the regularization penalty, which remains the same in this new dataset too.
E.4.2 WATERBIRDS
We perform similar analysis the Waterbirds multi-attribute shift dataset. In order to provide the same information to other DG algorithms as CACM, we create domains based on Acause x Aind in this setup (Table 8). We observe mixed results while some algorithms show significant improvement (ERM, IRM, VREx, MMD, MLDG, RSC), there a performance drop for some others (IB-ERM, IB-IRM, CORAL, C-MMD, GroupDRO, Mixup). CACM uses the knowledge of the causal
24

---

Published as a conference paper at ICLR 2023
Table 7: Synthetic dataset. Accuracy on unseen domain for Causal distribution shift when Acause is provided in input (column 2) and when Acause is additionally used to create domains (column 3).
Accuracy
Agoo
Acause (input)
A oouse
(input+domains)
ERM
73.3 + 1.3
71.8 + 3.8
IB-ERM
69.3
68.2 4.9
IRM
68.4 2.9
64.1 + 0.8
B-IRM
6
73.6 - 0.4
VREx
77.4 1.2
75.0 1.6
MMD
72.3 4.3
68.8 4.1
CORAL
75.5 0.7
72.1 0.8
DANN
60.8
65.8
C-MMD
2.7
4.9
CDANN
71.1
6.84
GroupDRO
7.2
6..4
Mixup
1.8
61.5 10.7
MIDG
73.3 2.6
3.3
SagNet
72.5 + 2.3
71.6 + 2.8
RSC
70.9 3.4
71.5 1.7
CACM
89.3 2.3
relationships between attributes and the label and hence the evaluation remains the same. Hence, we empirically demonstrate the importance of using information of the causal graph in addition the attributes.
Table 8: Waterbirds. Accuracy on unseen domain for multi-attribute distribution shift when Acause used to create domains (column 2) and when A Acauee Asnd used to create domains (column 3).
Worst-group Accuracy
Algo.
Acause
Acause Aind
ERM
37.0 + 1.1
43.0 + 7,8
IB-ERM
40.8 + 5.6
34.4 + 1.0
IRM
+
42.2 + 2.5
IB-IRM
6.5
+
VREx
2.3
48.0
MMD
2.4
13.319
CORAL
3.01
47.5 + 2.8
DANN
4.6
6.5
C-MMD
1.9
45.9
CDANN
4.97
5
GroupDRO
53.1
3.1
Mixup
6..7
50.3 1 1.5
MLDG
3.7
43.6 1
SagNet
38.0 1 1.7
RSC
40.9 3.6
46.5 + 5.3
CACM
84.5 + 0.6
F --X X. RELATIONSHIP
The E-X, edge shown in Figure 2b represents correlation of E with Xe, which change environments. As we saw for the Y A edge, this correlation can be due to causal relationship (Figure 8a), confounding with common cause (Figure selection (Figure Sc); all our results
25

---

Published as a conference paper at ICLR 2023
(Proposition 3.1 Corollary 3.1 hold for any of these relationships. To why, note that there is collider introduced on X or in any of the above cases.
Ains
Amii
(a)
(b)
(c)
Figure 8: (a) Causal, (b) Confounded and (c) Selection mechanisms leading F-X, correlation.
G
ANTI-CAUSAL GRAPH
X
X
A
Aind
Aind
(a)
(b)
Figure 9: Corresponding anti-causal graphs for Figure 2 Note the graphs are identical Figure with the exception of the causal arrow pointing from Y X instead of from X - Y.
Figure shows causal graphs used for specifying multi-atrriuute distribution shifts in an anti-cause setting. These graphs are identical to Figure with the exception of change in direction of causal arrow from X.
Y X..
We derive the (conditional) independence constraints for the anti-causal DAG for Independent, Causal, Confounded and Selected shifts.
Proposition G.1. Given causal DAG realized from the canonical graph in Figure 9a the correct constraint depends on the relationship label Y with the nuisance attributes A. As shown, A can be split into Aind Aind and E, where Annd can be further split into subsets that have a causal (Acause). confounded (Aconf) 5 ), selected ( (Aset relationship with Y (A end = Acause U Aconn U Uset ). Then, the (conditional) independence constraints that X. should satisfy are,
1. Independent: X. TT Aind X IL E; X 1L Aind (Y; X. TT Aind E: X. TT Aind [Y
Causal: X. I1 Acouse (Y; X. ll E; X. IL Acause |Y,
3. Confounded: X. 1L Aconf YY; X, 11 E; X LL Aconf [Y,
Selected: X. TL Asel (Y; X. 1L Asel Y E
Proof. The proof follows from d-separation using the same logic earlier proof in Section B.2 We observe that for attributes A E A (Arause Aconf Aget ), is required to condition on Y obtain valid constraints Y node appears as a chain or fork in the causal graph but never collider due to the Y - X. causal arrow.
However, in the presence Corollary G.1. All the above derived constraints are valid for Graph 9a of a correlation between E and X. (Graph 9b), only the constraints conditioned on hold true.
26

---

Published as a conference paper at ICLR 2023
Table 9: Search space for random hyperparameter sweeps.
Condition
Sweeps
MLP
learning rate: [le-2, le-3, lc-4, lc-5] dropout:
ResNet
learning rate: [1c-2, 1e-3, Ic-4, le-5]
dropout: [0, 0.1, 0.5]
MNIST
weight decay: 0
generator weight decay: 0
not MNIST
weight decay: 10Uhiifom(--0--2
generator weight decay: 10Unifor((66,-2)
IRM
[0.01, 0.1, 1, 10, 100]
iterations annealing: [10, 100, 1000)
IB-ERM, IB-IRM
λι [0.01, 0.1, 10, 100]
iterations annealing,ni [10. 100, 1000]
AIRM: [0.01, 0.1, 1, 10, 100]
iterations annealingzRar" [10, 100, 1000]
VREx
A: [0.01, 0.1, 1, 10, 100]
iterations annealing: [10, 100, 1000)
MMD
A: [0.1, 1, 10, 100]
7: [0.01, 0.0001, 0.000001]
CORAL
[0.1, 1, 10, 100]
DANN, CDANN
generator learning rate: [1e-2, 1e-3, le-4, le-5]
discriminator learning rate: [le-2, le-3, le-4, le-5] discriminator weight decay: 10 Un
[0.1, 1, 10, 100]
discriminator steps: [1, 2, 4, 8) gradient penalty: [0.01, 0.1, 1, 10]
adam B1: [0, 0.5]
C-MMD
X: [0.1, 1, 10. 100]
7: [0.01, 0.0001, 0.000001]
GroupDRO
[0.001, 0.01, 0.1]
Mixup
[O.1, 1.0, 10.0]
MLDG
3: [0.1, 1.0, 10.0]
SagNet
adversary weight: [0.01, 0.1, 1.0, 10.0]
RSC
feature drop percentage: Uniform (0, 0.5)
batch drop percentage: Uniform(0, 0.5)
CACM
A: [0.1, 1, 10. 100]
[0.01. 0.0001, 0.000001]
27


Reviews:
[{"review_id": "puDFHrzQz2", "reviewer": "Reviewer_QsVS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows  that Modeling the Data-Generating Process is Necessary for Out-of-Distribution Generalization, otherwise any single, fixed constraint algorithms can fail under certain distribution shifts. The authors then propose Causally Adaptive Constraint Minimization (CACM) to adaptively identify and apply the correct independence constraints for regularization. The authors also provide extensive evidences to justify their claims both theoretically and empirically.", "summary_of_the_review": "I enjoyed reading the paper. The authors identify an important issue in OOD generalization, provide detailed causal analysis as well as a novel solution. ", "strength_and_weaknesses": "\n### Strength: \n\n- The authors provide a new explanation from the perspective of data-generating process for the unexpected performance of DG algorithms.\n- The causal modeling and analysis, along with the proposed solution are full of details, novel and interesting. \n- The authors provide extensive empirical evidence to justify their claims and the effectiveness of CACM.\n\n\n\n### Weaknesses \n\n- The causal analysis seems to neglect the relationship between $A$ and $X_c$, which are discussed in IB-IRM [Ahuja et al. 2021]. Besides, the authors seem to miss a line of important related works, i.e., invariant graph learning, where [Chen et al. 2022] show that multiple potential distribution shifts can co-occur together in graphs and each of them can have a different causal relationship with the labels. \n\n### References:\n\nAhuja, K., Caballero, E., Zhang, D., Gagnon-Audet, J. C., Bengio, Y., Mitliagkas, I., & Rish, I. (2021). Invariance principle meets information bottleneck for out-of-distribution generalization. Advances in Neural Information Processing Systems, 34, 3438-3450.\n\nYongqiang Chen et al., Invariance Principle Meets Out-of-Distribution Generalization on Graphs, ICML 2022: Workshop on Spurious Correlations, Invariance and Stability.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nOSrpXTCRyb", "reviewer": "Reviewer_C2TU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors address the problem of a multi-attribute shift in Domain Generalization. They extensively and theoretically characterize various realizations of canonical causal graph modeling distribution shifts. They theoretically demonstrate that every such shift would entail a different independence constraint which explains previous empirical evidence that different domain generalization approaches that are focused only on a single attribute shift, show inconsistent performance between different datasets. The authors develop an algorithm (CACM) to tackle this problem of a multi-attribute shift in the case when the attributes are observed (or partially observed). The authors introduce 3 novel multi-attribute datasets, on which they demonstrate that CACM outperforms previously proposed benchmarks.\n", "summary_of_the_review": "I think this is an important piece of work, which will have a direct contribution to the study of domain generalization in real-world scenarios, such as for example medical imaging domain. The paper is well-written, well-supported with theory and experiments, and contributes an important piece to the view of domain generalization in a practical setting.  For example, this paper is well aligned with the discussions on the understanding of causal modeling in medical imaging applications (Castro et al. 2020) and would facilitate further research into this direction in medical imaging with demonstrations on real-world medical datasets and benchmarks.\n\n", "strength_and_weaknesses": "Strengths:\n\nThis paper extends the view on the robustness toward distribution shift and brings the problem closer to the real-world scenario. The theoretical findings of this paper are not only important on their own but additionally demonstrate how the language of causal modeling could be used to correctly formulate the hypothesis about the independence constraints needed for method development. The authors provide a comprehensive discussion on how much the use of incorrect constraints would hurt generalization. The theoretical justifications in the paper are clear and convincing. \n\nWeaknesses. A few things were not entirely clear to me:\n\n- In the example of label shift due to more women coming to one of the hospitals: isn't it a case where the attribute \"gender\" is caudal to $X_c$? Isn't a case like this explicitly disallowed by the assumptions of the paper? Or is this case covered by a dotted arrow from E to $X_c$? From the description, it is not clear if the causation from the environment $E$ is also not allowed as causation from the attribute to the label? Also, in general, how restrictive is the assumption that attributes can't cause labels?\n- The authors discuss that not all attributes have to be observed and that conditioning on the environment can substitute conditioning on the attribute. However, it is not clear to me, how it will hurt the performance of the method because this is a weaker constraint. I agree with the reasoning of the authors about the fact that in practical scenarios attribute shifts are sometimes known. However, even in the examples they describe, these distribution shifts are quite often discovered postfactum. Methods like IRM are designed to tackle the situation when the attributes are not explicitly observed. They don’t explicitly use the information about the attribute but tackle a more complicated problem when we have information that attributes shifts happened, but we don’t know in which way exactly. Intuitively, I don’t find it surprising that on the benchmarks introduced by the author or in general on any dataset which has explicit information about the attribute, the method which is using attribute is superior to the method that doesn’t. Would CACM still outperform IRM if only the environment was observed, but the attributes were not? Because for the case of IRM, it is not clear to me which constraint is violated.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "d2z2O_tt8V", "reviewer": "Reviewer_kwon", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors describe regularization based strategies for learning feature transformations that result in risk-invariant classifiers. They give two canonical generative DAGs and study domain generalization under these DAGs. The authors prove that any feature transformation must satisfy all relevant (conditional) independence relationships between the transformed features and the attributes and environment variables if it is risk-invariant (necessary condition). In particular, more than one independence relationship is required, which explains the inconsistent results from other domain generalization works. Finally, the authors propose an algorithm for identifying the set of required constraints when given a DAG or the shift-type of each attribute. The claims of the authors are demonstrated empirically through experimentations on three data sets. \n", "summary_of_the_review": "A solid paper shedding light on domain generalization.\n", "strength_and_weaknesses": "This paper provides a framework for understanding domain generalization and when/why it sometimes fails. Further, the paper extends the typical domain generalization formulations to what the authors call multi-attribute shifts. Numerous baseline methods are used.\n\nThe authors mention the possibility that not all attributes are observed, yet give little information on the consequences of observing only some attributes. What are these consequences? What are the consequences or limitations of not observing E?\nOn a related note, the authors treat A a discrete. Does it have to be? Practically speaking, if A is discrete then it would seem that it must be of low cardinality as well (else one would need a lot of data). \n\nBy comparison, the NURD method appears to address unobserved attributes through the use of proxies (e.g. the border of an image) since it allows for high dimensional nuisance variables. \nFurther, the authors of NURD remark \"Counterfactual invariance promises that a representation will not vary with the nuisance but it does not produce optimal models in general because it rejects models that depend on functions of the nuisance\". They seem to argue that the uncorrelating property used in that work is superior as a result. \n\nI would really like to see the authors compare to NURD, both in experiment and in discussion.\nPuli, Aahlad Manas, et al. \"Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlations.\" International Conference on Learning Representations. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QvYSL72J_1", "reviewer": "Reviewer_LyJp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to unify generalization methods by accounting for potential data-generating processes. The paper is presented to unify potential choice of generalization methods under a canonical causal graph which can account for i) label indepedent attributes, ii) label dependent attributes and iii) environment variables all of which can result in changes across distributions. ", "summary_of_the_review": "Overall I believe this is a valuable contribution. I would want a clarification on the above questions for me to more strongly endorse the paper and I look forward to the rebuttal from the authors.", "strength_and_weaknesses": "\n-> Not clear if Figure 2(c) is indeed exhaustive.\n\n-> Second para on page 5 is insightful but the names are likely to create more confusion than they help. I urge authors to reconsider the naming.\n\n-> Authors don't talk about whether imposing the independence constraints is *sufficient*.\n\n-> Empirical evaluation: Ablation on results w/o regularization of the logits representation compared to benchmarks (E.3 is only evaluating for regularization penalty).\n\n-> What conditional indpendence testing is used in empirical evaluation?\n\n-> Can the authors specify how all hyperparameters are chosen? Is there a domain designated as a validation \"domain\"? There have been results showing inconclusivity of domain generalization purely due to hyperparameter tuning challenges.\n\n-> When E and X_c are correlated, do use E conditioned regularization essentially reduces to regularizing only within domain? In that case the parameters/representations being shared is the more crucial contributor to the performance. May be I am missing something and the authors can clarify intuition. \n\n-> Comment on generalizability of the canonical graph. It seems like the authors have come up with a canonical graph that encapsulates most of the methods and/or datasets. But it is not clear how exhaustive this graph is. \n\n-> In the algorithm description, I found that you assume the test for independences are perfect. Is that reasonable?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%