Target Conditioned Representation Independence (TCRI); from Domain-Invariant to Domain-General Representations
Authors: Olawale Elijah Salaudeen
Oluwasanmi O Koyejo
Venue: Submitted to ICLR 2023 (2023)
Keywords: Domain Generalization
Out-of-distribution Generalization
Transfer Learning
Distribution Shift
Covariate Shift

Abstract: We propose a Target Conditioned Representation Independence (TCRI) objective for domain generalization. TCRI addresses the limitations of existing domain generalization methods due to incomplete constraints. Specifically, TCRI implements regularizers motivated by conditional independence constraints that are sufficient to strictly learn complete sets of invariant mechanisms, which we show are necessary and sufficient for domain generalization. Empirically, we show that TCRI is effective on both synthetic and real-world data. TCRI is competitive with baselines in average accuracy while outperforming them in worst-domain accuracy, indicating desired cross-domain stability.

Under review as a conference paper at ICLR 2023
TARGET CONDITIONED REPRESENTATION INDEPEN- DENCE (TCRI); FROM DOMAIN-INVARIANT TO DOMAIN-GENERAL REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a Target Conditioned Representation Independence (TCRI) objective for domain generalization. TCRI addresses the limitations of existing domain gen- eralization methods due to incomplete constraints. Specifically, TCRI implements regularizers motivated by conditional independence constraints that are sufficient to strictly learn complete sets of invariant mechanisms, which show are necessary and sufficient for domain generalization. Empirically, we show that TCRI is effec- tive on both synthetic and real-world data. TCRI is competitive with baselines in average accuracy while outperforming them in worst-domain accuracy, indicating desired cross-domain stability.
INTRODUCTION
Machine learning algorithms are evaluated by their ability to generalize (generate reasonable pre- dictions for unseen examples). Often, learning frameworks are designed exploit some shared structure between training data and the expected data at deployment. A common assumption is that the training and testing examples are drawn independently and from the same distribution (iid). Given the iid assumption, Empirical Risk Minimization (ERM; Vapnik (1991)) and its variants give strong generalization guarantees and are effective in practice.
Nevertheless, many practical problems contain distribution shifts between train and test domains, and ERM can fail under this setting (Arjovsky et al., 2019). This failure mode has impactful real-world implications. For example, in safety-critical settings such as autonomous driving (Amodei et al. 2016, Filos et al. 2020). where lack robustness to distribution shift can lead to human casualties; or in ethical settings such as healthcare, where distribution shifts can lead to biases that adversely affect subgroups of the population (Singh et al. 2021). To address this limitation, many works have developed approaches for learning under distribution shift. Among the various strategies achieve domain generalization, Invariant Causal Predictions (ICP:Peters Peters et al. (2016)) has emerged as popular. ICPs assume that while some aspects of the data distributions may vary across domains, the causal structure (or duta-generating mechanisms) remains the same and try to learn those Jomain-general causal predictors.
Following ICP, Arjovsky et al. (0019) propose Invariant Risk Minimization (IRM) to identify invariant mechanisms by learning a representation of the observed features that yields a shared optimal linear predictor across domains. However, recent work (Rosenfeld et al., 2020), has shown that the IRM objective does not necessarily strictly identify the causal predictors, L.e., the representation learn may include noncausal features. Thus, we investigate the conditions necessary to learn the desired domain- general predictor and diagnose that the common domain-invariance Directed Acyclic Graph (DAG) constraint is insufficient to (i) strietly and (īi) wholly identify the set of causal mechanisms from observed domains. This insight motivates us to specify appropriate conditions to learn domain-geneet models which we propose to implement using regularizers.
Contributions. We show that neither a strict subset nor superset of existing invariant causal mechanisms is sufficient learn domain- general predictors. Unlike previous work, outline the constraints that identify the strict and complete set of causal mechanisms to achieve domain generality. We then propose regularizers implement these constraints and empirically show the efficacy of our

---

Under review as a conference paper at ICLR 2023
proposed algorithm compared to the otate-of-the-art synthetic and real-world data. To this end, observe that the conditional independence measures are effective for model selection outperforming standard validation approaches. While our contributions are focused on methodology. highlight existing gaps in standard evaluation using dovain-average metrics which we show can hide worst-case performance; arguably a more meaningful measure of domain generality.
results also
RELATED WORK
Domain adaptation and generalization have grown be large sub-fields in recent years. Thus, do not attempt an exhaustive review, and will only highlight a few papers most related to our work. To address covariate shift, Ben-David et al. (2009) gives bounds on target error based on the H- divergence between the source and target covariate distributions. which motivates domain alignment methods like the Domain Adversarial Neural Networks (Ganin et al. (2166). Others have followed up on this work with other notions of covariant distance for domain adaptation such as mean maximum discrepancy (MMD) (Long et al. 2016) and Wasserstein distance (Courty et al. 2017), etc. However, Kpotufe and Martiner (2018) show that these divergence metrics fail to capture many important properties of transferabilitty such as asymmetry and non-overlapping support. Zhao et al. (2019) show that even with distribution alignment of covariates, large distances between label distributions inhibit transfer; they propose a label conditional importance weighting method to address this limitation. Additionally, [Schrouff al. (2022) show that many real-world problems contain more complicated 'compound' shifts than covariate shifts. Furthermore, domain alignment methods are useful when one has unlabeled or partially labeled samples from the domain one would like adapt during training, however, the domain generalization problem setting may not include such information. The notion of invariant representations starts address the problem of domain generalization, the topic of this work, rather than domain adaptation.
Arjovsky et al. (2019) propose an objective to learn a representation of the observed features which, when conditioned on, yields distribution on targets that is -nvaainnnvvarian,, that is, conditionally independent of domain. They argue that satisfying this invariance gives a feature representation that only uses domain-general information. However, [Rosenfeld et al. (2020) shows that the IRM objective can fail to recover a predictor that does not use spurious correlations without observing a number of domains greater than the number of spurious features, which can inhibit generalization. Variants of this work Krueger ct al. (2021); Wang et al. (2022) address this problem by imposing higher order moment constraints to reduce the necessary number of observed domains. However, as we will show, invariance on the observed domain not sufficient for domain generalization.
Additionally, of the motivations for domain generalization is mitigating the worst domain performance. Guulaaaani and Lopez- Paz (2020) observe empirically that ERM is competitive and often best in worst domain accuracy across range of real-world datasets. [Rosenfld al. (2022) analyze the task of domain generalization as extrapolation via bounded affine transformations and find that ERM remains minimax optimal in the linear regime. However, the extent of the worst-domain shift is often unknown in practice and may not be captured by bounded affine transformations Shen et al. (2021)
In contrast, our work allows for arbitrary distribution shifts, provided that invariant mechanisms remain unchanged. In addition, we show that our proposed method gives predictor that recovers domain-general general mechanisms and is free of spurious correlations without necessitating examples
all domain- (neither labeled nor unlabeled) from the target domain.
PROBLEM SETUP
We consider the data generating mechanism as described by the causal graph Figure T and the equivalent structural equation model (or structural causal model SCM (equation [ Pearl (2010)).
One particular setting where this graph applies is medicine where we are often interested in predicting conditions from potential causes and symptoms of the condition. Additionally, these features may be influenced by demographic factors that may vary across hospitals (Schrouff et al. 2022). Additionally, in physical processes where measurement is slow, observes both upstream (causal) and downstream (anticausal) features of the events of interest. An example is in task-fMRI where the BOLD -oaedd-ye-e---el--eeeeeepe
signal in task-fMRI (functional Magnetic Resonance
2

---

Under review as a conference paper at ICLR 2023
Z
X
7.e
Figure 1: Graphical model depicting the structure of our data-generating process shaded nodes indicate observed variables. X represents the observed features, Y represents observed targets, and represents domain influences. There is explicit separation of domain-general (causal) Ze and domain-specific (anticausal) Ze features, which are combined to generate observed X.
Imaging) much slower than the neural processes in the brain associated with the task (Glover 2011). Many other real-world problems fall under this causal and anticausal setting: note that this graph is also assumed by previous work Arjovsky et al. (2019). We also assume that the observed data drawn from set of F,r training domains E (e; 1, ..... } all generated from Equation | thereby fixing the mechanisms by which the observed distribution is generated:
(es) p(e.)
P2,
SCM((e:
yla)) fy (es) (26)))))))
where 1 (04)
(e)) = fr. (y)es), sp(rs) where s7(44) T y(s))
where Pz, is the causal covariate distribution, fyr fz. are generative mechanisms of y and &er respectively, and v, are exogenous variables. These mechanisms are assumed to hold for any domain generated by this generative process, i.e., Me, (3/ =z) n4 zz) and Ve, (ze ly) =<(2, (y) Ve, & for some distributions μ and v, where E is the set of all possible domains. Under the Markov assumption, we can immediately read off some properties of any distribution induced by the data- generating process shown in Figure I (i) e IL Y Zc. (ii) Z. 1 Ze Y.e, and (iii) Y y e X. However, as shown Figure [ we only observe an unknown function of latent variables ze and I h(ze; ze) and we would like predictions that do not depend on domains (e). Ideally, we want to map x > ze such that the mechanism fy can be learned, as this would suffice for domain generalization.
In contrast, though we have that the mechanism =e - is preserved, we have no such guarantee on the inverse ze & y it may not exist be unique and, therefore, does not satisfy domain generalization. This becomes a problem when considering mappings that include ze:
Pe, (y ze) A Me, (1) ze) and
Me, (y) Zes Re) * Mey (y) Zes Zey ze) for +
The latter implies that Me; (y) x) * Pey (y x)) for * j, and therefore, the original observed features will be comain-general. Note that the generative process and its implicates are vital for our approach, as we assume that it is preserved across domains.
Assumption 3.1. We assume that all distributions, train, and test (observed and unobserved at train), are generated by the generative process described in Equation]
In the following sections. will introduce proposed algorithm to learn a feature extractor that maps to Z, and show the generative propertics that are necessary and sufficient to do so.
TARGET-CONDITION
REPRESENTATION INDEPENDENCE OBJECTIVE
We first define two distinct types of representations of observed features that will need henceforth domain-invariant and domain domain-general.
Definition 4.1. A domain-invariant representation &(X), with respect to set of observed domains E,,, is one that satisfies the intervention properties μr, (XX) = x²) Pe, (v/((X) = x) Ve,, ej e Ex, for any fixed v, where e,, éj domain identifiers and fa is probability distribution.
3

---

Under review as a conference paper at ICLR 2023
Ze
Oc
De
X
F
X ei
Occ
Des
Ze
We first generate a feature representation via featurizer F. During training, both representa- tions, & and V, generate predictions domain- and domain-specine predictions, respectively. However, only the domain-invariant by the solid red arrows. b indicates concatenation.
Figure 2:
pppresentatios//rreiittoon are used during test time - indicated
In other words, under
representation, the output conditional distribution for a given input is necessarily the same across the reference (typically observed) domains. This is consistent with the existence of μ s.t. μ (x) x) = fu (y (o(@((()))))) x²), where do(-) is the "do-operator" Peearl (2010), denoting an intervention (arbitrary assignment of value). Conversely, for speciffc representation, the output conditional distribution for given input need not be the
same across domains.
However, by definition, this representation is the set of domains that it is learned on. So, it may not be domain-invariant with respect to test domains without additional assumptions on the set of training domains, e-g-. their convex hull (Rosenfeld et al.] 2020). So, we need to refine this property to better specify the representations we would like to learn. This motivates the following definition which ties domain-invariance to a specific generative process, opposed to set of the observed distributions, which, along with Assumption 3.1 connects causality to domain genaainngnrraltt..
up specific of domains, often
Definition 4.2. A representation is domain- general with respect DAG G if it domain-invariant with respect to E, where is the set all possible domains generated by
By Assumption 3.1 causal mechanisms from features Z. to target Y are domain-gentt general, so natural strategy is to extract Z. from the observed features. order to understand when we can recover Ze from the observed X, we will consider two conditional independence properties implied by the assumed causal graph: Y I e Ze. which we call the domain invariance property, and Ze IL Ze Y, e, which we capture in the following target conditioned representation independence property (TCRI; definition 4.3).
Definition 4.3 (Target Conditioned Representation Independence). Two functions, ④. V, are said to satisfy TCRI with respect to random variables X, Y, e, if I(D(X), (X):Y) = I[Ze. Ze: and (X) 1 ((X) Vez.
We will show that these properties together identify Z, from X, Y to give a domain- repre- sentation in Section 5 Based our results, design algorithm to learn feature mapping that recovers Ze, i.c., the domain -enuinal representation (mechanisms). Figure illustrates the learning framework.
In practice, we propose TCRI objective containing four terms, each related to the properties desired of the learned representations, follows,
LTCRI
Σ laLt + +1 (1 + ALTRMOP + 3Ler),
(2)
Er
e.EEtr
where a E [0,
1].11,800 B
and À > 0 hyperparameters- - 2 shows the full framework.
In detail, we let La = RO0,- +)) represent the domain-general predictor accuracy and let LiRMer be penalty from & and the linear predictor 0. that enforces that & has the same optimal predictor across training domains, capturing the domain invariance property, where db : X' + H4. 0. H. + y (Arjovsky et al. 2019)
LiRMEr oo, oo.77 R (0c 1)112
4

---

Under review as a conference paper at ICLR 2023
where Rd denotes the empirical risk achieved on domain e,. The Lo and LIRMeL' together implement a domain- invariance property. however, we know that this is not sufficient for domain generalization (Rosenfeld et al.) 2020). We will show later that the addition of the TCRI property suffices for domain generalization (TTheorem 5.4).
To implement the TCRI property (definition
also learn domain-specifer representation V, which is constrained to be (i) conditionally independent of the domain- -emainl representation given the target and domain and (ii) yield a predictor good one from X when combined with Q. We first address (ii); given a domain-specific representation, V/ X + Hv. we define set of domain domain-specific predictors {fe, H4 x Hy + y i = L.... Eir}. We then add term in the
objective that minimizes the loss of these domain domain-specifer specific predictors:
Loev R (fe. o ( + V))
This term aims to enforce the total-chain maal--hain-informationn minimize a nomain-specinf loss together, where non-domain-ao domain-general general information about the target can be used to improve performance. Since we have that both the domain-general and domain-specift have unique information about the target, the optimal model will use both types of information. We allow the Oe, for each domain to be different since, by definition of the problem, we expect these mechanisms to vary across domains.
of TCRI, by allowing t and V to
We detine Lor to be the conditional independence part of the TCRI property and use the V-statistic- based Hilbert-Schmide Independence Criterion (HSIC) estimate (more details on HSIC can be found in Gretton et al. (2007)). For the two representations (()), V (X) for which want determine independence (+) (X) To (X)), define
1 Σ HSTC(((X)) ((X))) ((X))) LC1
1
C trace trace(Ko K KnH,,,
C
k=1
where k, indicates which class the examples the estimate correspond to, C. is the number of classes, Kg E TR Ky E Xme are Gram matrices, K k( X x), t (X))), K. w(V(X): (X),) with kernels K, w are radial basis functions, H., Ing 11 is a centering matrix, I, is the ny X n& dimensional identity matrix, 1n, is the nl--dimensional vector whose elements are all 1, and T denotes the transpose. We condition on a label and domain by taking only examples of each label-domain pair and computing the empirical HSIC; then we take the average across all labels. We note that any criterion for conditional independence can be used as LCr, e.g.. partial covariance.
Altogether, we have the following task:
min
Σ e.
JaCs + (1 (1- al)-aoooo + LIRMor + BCerl.
(3)
c M N...0.0..00 02 BES
Eir
We compute the complete objective for each domain separately to condition the domains e, and after minimizing this objective, only the invariant representation and its predictor, 0₂ Q are used.
5 CONDITIONS FOR DOMAIN-GENERAL REPRESENTATIONS
Now provide some analysis to justify our method. Consider a feature extractor & X Z, where X, are input and latent features spaces, respectively. We first show that a & that captures strict subset of the causal features satisfies the domain invariance property Y I e ((X) while not necessarily being domain- -amainngeneral.
Lemma 5.1 (Insufficiency of Causal Subsets for domain generalization). Conditioning on a subset of causal variables (invariant mechanisms) does not imply domain generalization (definition 4.2).
Z Z. + Me, (v) Z 2) Mey (vl Z 2)ve, + cj. E 2
where Z, is the causal feature space.
5

---

Under review as a conference paper at ICLR 2023
domain
domain 2
domain
0
-1 0
X1
x1
X1
Figure 3: Visualization of Lemma 5.1 where ø1 and D2 22. The large circle is the true decision boundary, where x distribution (Uniform), which differs across domains. The length of the solid line indicates the m 0 2 0). Clearly the length of each line changes across domains, hence the causal subset 21 does not yield comain-geneete predictor.
0, o : = 1. The dotted square indicates the covariate
Proof. We provide a simple counterexample. Suppose have the following generative process de..1 + 21 + f + dei2 with æ1, E R, de,,1; de, E Rzo. r E 1u and
0 if 27 + 22 <=r
1 if of + +2 >T.
Suppose we observe domains fe; : 1..... Err where 21 ~ Uniform(-re + dea.1, dea.) and 22 ~ Uniform( 2,, 2e..2) for domain e₂ and de, a domain quantity. Now, suppose we condition on 21. The optimal predictor is
Me. (y 1 21) 2
dey.2
2r
In this case, 21, causal subset, does not yield domain domain-general general representation since its optimal predictor depends on de,, which changes across domains.
provide a visualization of Lemma 5.T in Figure 3 Additionally, we show that subsets of causal features satisfy the
property.
Lemma 5.2. A representation & that maps to a strictly causal subset can be
(definition 4.7
Proof. This proof follows a similar argument as Lemma 5.1 We replace the mechanism for 21 with 21 Uniform(da,,1 -r, de,,1 r), i.e., the domain is scaled along 21. Suppose de,, for all training domains {e: : 1,... Ex, } and ((21, 22) = 21. There is still a covariate distribution shift due varying values (scaling) of de.. Then & is domain invariant on the training domain but will no longer be domain-genera -ennral with respect to any domain ej where de, 2 0.
In other words, one failure case that illustrates a limitation of strictly considering the domain- invariance property (#(X) L Y e) is when the shifts observed in the training set are different than those the test set. Specifically, an incomplete causal representation may not be domain- causal features that not captured in the incomplete
general if the shifts at test-time causal representation. We now show that the
property and Target Conditioned Representation Independence (TCRI) property are sufficient for recovering the complete set of invariant mechanisms.
Lemma 5.3. (Sufficiency of TCRI for Causal Aggregation). Recall, X. Ze. Ze, Y from Figure 7 Let Ze, Ze be direct causes and direct effects of Y, respectively, and recall that X is a function of Zn and Ze the representations induced by feature extractors & V satisfy TCRI, then wlog T(o(()) Y) 2 T(Z.;
Lemma 5.3 shows that the TCRI property directly solves the problem identified in Lemmal 5.1 that is, the TCRI property implies that causal information about the target is aggregated into one representation &. Now, identify Z.,. only need show that & is strictly causal.

---

Under review as a conference paper at ICLR 2023
properties for identifying Ze). Recall that in the assumed generative model and X are the observed features Figure [ If Φ V satisfy TCRI and domain-invariann property, then & recovers Ze and is therefore domain- -eneralgeneral.
Theorem 5.4 (Sufficiency of TCRI Ze. Ze are the true latent features features
Proof. By Lemma 5.3 we have that when D, V satisfy TCRI, 4(X) contains all of the causal (necessarily nomain-generee information in X. However, ((X) may also contain non-causal Clomain-specifie information, spoiling the domain- -eeuaialtee of ④b It remains to show that (XX) is strictly causal when we add the invariinne property.
If d satisfies the domain- invariance property, then Y T e (xX). Clearly, this cannot be the case if & (X) contains features of Ze, since e and Y are colliders on Ze and therefore conditioning on Ze opens path between e and Y, making them dependent. Thus (xX) can only contain causal features.
Therefore a representation that satisfies TCRI and the domain-invariane property wholly and strictly causal and thus domain-general. The latter follows from the Ze having invariant mechanisms.
Theorem 5.4 suggests that by learning two representations that together capture the mutual informa- tion between the observed X, Y, where one satisfies the TCRI, one can recover the strictly and complete causal feature extractor and Nomain-genere predictor.
property and both satisfy
Remark 5.5. (Revisiting Lemma 5.1 (s counterexample) Given two representations 4, V that satisfy TCRI, 4 necessarily captures 21, #2. By definition, Φ v must capture all of the information in #1, 22 about y, and we know from the graph that they are conditionally dependent given y. i.e, 21, 22 are common causes of y (colliders), dependent. So, 2 and 22 must be captured the same feature extractor.
conditioning
renders the marginally independent variables
Remark 5.6. One limitation TCRI is a failure mode when the strictly anticausal representation gives domain invariant predictor. In this case, either representation may be Q. However, one of the benefits of having a domain-spectie predictor for each observed domain that we can check those classifiers are mapped to the domain-specific feature extractor, we will see that the domain-specific classifiers give similar results when applied domain that they invariant causal mechanisms. This, however, gives a test not a fix for this setting we leave a fix for future work.
interchangeable. Specifically, if we in this scenario where the causal features
not trained since they are based
6 EXPERIMENTS
Model
L
$1.0
ERM
0.84
0.18
IRM
0.83
0.18
(e)) 26
Exp (ax))
TCRI (HSIC)
1.11
0.01
Oracle
1.13
0.0
SCM((e) =
y(ei)= ze + Exp (0.25),
2e (es)
= ((a)) + Exp (a
Table 1: Continuous Simulated Results Fea- ture Extractor with dummy predictor 0, = 1., i.c.. 9 x x w, where E g 4
R2x w E IR. Oracle indicates the coefficients achieved by regressing y on 2, directly.
To evaluate our method in setting that exactly matches our assumptions and we know the ground truth mechanisms, we use Equation to generate our AinerrGGausiie data, with domain parameters gf of code provided in the supplemental materials.
We observe 2 domains with parameters df of 0.1, .-: of 1. each with 1000 samples, and use linear feature extractors and predictors. Minimizing the TCRI objective (Equation 3 recovers linear feature representation that maps back to s, (Table T Note that for
7

---

Under review as a conference paper at ICLR 2023
Table 2: Colored MNIST. 'acc" indicates model selection via validation accuracy, and 'cov' indicates model selection via validation conditional independence. 'ci' indicates a conditional cross-covariance penalty, and 'HSIC' indicates Hilbert-See Sehbert-Schhidt Independence Criterion penalty. Columns [+90%, +80%, -90% indicate domains {0.1, 0.2, 0.9) digit label and color correlation, respectively. üú, o indicate the mean and standard deviation of the average domain accuracies, over 3 trials each, respectively.
Domains
Domain Accuracy Statistics
Algorithm
+90% %
+80% %
-90%
mean
std
min
ERM
71.8 = 0.1
72.8 + 0.2
10.0 0.1
51.5
36.0
10.0
IRM
72.4 0.5
72.8 + 0.2
10.0 + 0.3
51.7
36.1
10.0
VREx
7..3
73.2 0.5
0.1
51.8
36.2
10.0
GroupDRO
0.2
73.1 + 0.2
0.2
51.8
36.2
10.0
MLDG
73.0 0.1 7.0
10.2 0.0
51.6
35.9
10.2
ARM
0.6
74.5 1.2
10.2 0.0
55.6
39.4
10.2
TCRI (cov) ace
1.4
1.9
10.0 0.1
50.2
34.8
10.0
TCRI (HSIC)- acc
71.9 0.5
72.6 0.4
10.3 + 0.1
51.6
35.8
10.3
TCRI (cov) ci
54.7 1.7:1, 1.1
56.4 4 2.4
50.6
53.9
3.0
50.6
TCRI (HSIC) ci
54.7 = 1.6
60.1 4.1
5.1
55.9
3.6
53.0
ERM, À = 3 = 1 0, 0, a= and TCRI, 0.1, 8 and 0.75;
additional details can be found in Appendix
6.1 DATASETS
Algorithms: We compare our method to the following baselines: Empirical Risk Minimization: Empirical Risk Minimization (ERM, Vapnik (199T)), Invariant Risk Minimization (IRM Arjovsky et al. (2019)), Variance Risk Extrapolation (V-REx, Krueger et al. (202T), Meta-Learning for Domain Generalization (MLDG, [i et al. (2018)) Group Distributionally Robust Optimization (GroupDRO, Sagawa et al. (2019)), and Adaptive Risk Minimization (ARM [Zhang et al. (202T)). Additional discussion on the algorithms can be found in the appendix.
We evaluate our proposed method on real-world datasets. Given observed domains &,, = (ez :i= 1, 2..... }. we train on Exr \ e; and evaluate the model on the unseen domain e,, for each e;,
Typically, ML practitioners use within-domain hold-out validation set for model selection. However, this strategy is biased towards the empirical risk minimizer, i.e., the one with the lowest error the validation set from the training domains, however, we know that the model that achieves the highest validation accuracy may not be -eneral This same is true if we out-of-domain validation set that is not from the target domain. Alternatively, we propose leverage the generative assumptions for model selection. Specifically, we consider other properties of our desired model for model selection; specifically, low Lcr. To do this, we follow the practice of hold-out within-domain validation set, however, we compute Les for the validation set and choose the example with the lowest CI score instead of the highest validation accuracy. We compare this strategy with validation accuracy in our results. Additional details can be found in Appendix
Model Selection:
We evaluate our method on the ColoredMNIST dataset [Arjovsky al. (2019) which is composed of 7000 (2 x 28 x 28, 1) images of hand hand-written digit and binary-label pairs. There are three domains with different correlations between image color and label, i.e., the image color is spuriously related to the label by assigning a color to each of the two classes (0: digits 0-4, digits 5-9). The color is then flipped with probabilities {0.1, 0.2, 0.9} to create three domains, making the color-label relationship domain-specific, as it changes across domains. There is also label flip noise of 0.25, so we expect that the best we can do is 75% accuracy. As in Figure [ Z, corresponds to the original image, Ze the color, the label-color correlation, Y the image label, and X the observed colored image. Code (a variant httttp//gghhho.coo///coooeeeeeae can be found at
CNISTedMNIST:
8

---

Under review as a conference paper at ICLR 2023
We use MNNISTCCONNNN Guirajani and Lopez- Paz (2020) backbones for the MNIST datasets and parameterize our experiments with the DomainBed hyperparameters with three trials to select the best model Gulrajani and Lopez- Paz (2220). The MNIST-ConvNet backbone corresponds to the generic featurizer F in Figure 2 and both + and V are linear layers of size 128 x 128 that are appended to the backhone. The predictors 0z, 01:...; Dg.. the & and V layers, respectively.
also parameterized to be linear and appended to
6.2 RESULTS AND DISCUSSION
We observe that TCRI competitive in average accuracy with the baseline methods. However, it is significantly more stable when using conditional independence for model selection, i.e., the worst-domain accuracy is highest and variance across domains is lowest for TCRI both by large margin. We note that domain -90%, which has color-label relationship flip probability of 0.9, has majority color-label pairing that is opposite of domains +80% and +90%, with flip probabilities of 0.1 and 0.2, respectively. Hence, we observe that the baseline algorithms generalize poorly to domain -90%, relative to the other two domains. This is a clear indication that, unlike in TCRI, the baselines using spurious information (color) for prediction. While TCRI does not obtain the expected best accuracy of 75%, it is evident that the information being used for prediction is general across the three domains, given the low variance in cross-domain accuracy.
An important implication of a domain-generee is stability - rrobustness in worst-domain performance, up domain difficulty. While average accuracy across domains, provides some insight into an algorithm's 's ability to generalize to new domains, it is susceptible to being dominated by the performance of subsets of observed domains. For example, ARM outperforms the baselines in average accuracy, but this improvement is driven primarily by the first domain (+90%), while the worst-domain accuracy stays the same. In the context of real-world challenges such as algorithmic fairness, comparable worst-domain accuracy is necessary (Hardt et al.) 2016). TCRI achieves 5x (53.0 vs. 10.2) the worst-domain accuracy of the best baseline while maintaining the competitive average accuracy outperforming most of the baselines on average. Additionally, the variance of the domain accuracies is over 10x lower than that of the baselines, showing further evidence of the cross-domain stability would expect from domain domain-general general algorithm. The baselines also include V-REx, which implements regularizers on risk variance across observed domains.
Worst-domain Accuracy:
Limitations: The strength of TCRI is also its limitation; TCRI is very conservative, so as to be robust to worst-domain shifts. While many important real-world problems require robustness to worst- domain shifts, this is not always the case, and in this setting, TCRI sacrifices performance gains from non-domain-geet -general information that may be domain-invariant with respect the expected domains. The practitioner should apply this method when it is appropriate, i.e., when domain generality is critical and the target domains may be sufficiently different than the source domain. is. however, important to note that in many settings where one is happy with domain-invariane. as opposed to Nomain-generalle ERM may be sufficient (Gulrajani and Lopez-Pa7 2020, Rosenfeld et al. 2022).
CONCLUSION AND FUTURE WORK
We address the limitations of state-of-hhe-as art algorithms' inability to learn Nomain-geneers predictors by developing an objective that enforces DAG properties that are sufficient to disentangle causal (domain- -eneral) and anticausal mechanisms. We compare domain generality to showing that our method is competitive with other state-of-the-art domain-generalizan on real-world datasets terms of average across domains. Moreover, TCRI outperforms all baseline algorithms in worst-domain accuracy, indicating desired stability across domains. We also find that using conditional independence metrics for model selection outperforms the typical validation accuracy strategy. Future work includes further investigating other model selection strategies that preserve the desired domain -eenrallyy properties and curating more benchmark real-worke datasets that exhibit worst-case behavior.
algorithms
outperfor

---

Under review as a conference paper at ICLR 2023
REFERENCES
Amodei, D., Olah, C., Steinhardt, J., Christiano, P. F., Schulman, J., and Mané, D. (2016). Concrete problems in ai safety. ArXiv, abs/1666.06666.
Arjovsky, M., Bottou, L.,, Gulrajani, I., and Lopez-Paz, D. (2019). Invariant risk minimization. ArXiv, abs/1907.02899 02893.
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F. C., and Vaughan, J. W. (2009). A theory of learning from different domains. Machine Learning, 19:151-175.
Courty, N., Flamary, R., Habrard, A., and Rakotomamonjy, A. (2017). Joint distribution optimal transportation for domain adaptation. Advances in Neural Information Processing Systems, 30.
Filos, A., Tigas, P., McAllister, R., Rhinehart, N., Levine, S., and Gal, Y. (2020).. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? In ICMI..
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):009-20300.
Glover, G. H. (2011). Overview of functional magnetic resonance imaging. Neurosurgery clinics of North America, 22 2:133-9, vii.
Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., and Smola, A. (2007). A kernel statistical test of independence. NIPS.
Gulrajani, I. and Lopez-Paz, D. (2020). In search of lost domain generalization. CoRR, abs/2007.01434.
Hardt, M. Price, E., and Srebro, N. (2016). Equality of opportunity in supervised learning. Advances in neural information processing systems, 29.
Kpotufe, S. and Martinet, G. (2018). Marginal singularity, and the benefits of labels in covariate- shift. In Bubeck, S., Perchet, V., and Rigollet, P., editors, Proceedings of the 31st Conference Learning Theory, volume 75 of Proceedings of Machine Learning Research, pages 1882-1886. PMLR.
Krueger, D., Caballero, F., Jacobsen, J., Zhang, A., Binas, J., Priol, R. I.,, and Courville, A. C. (2021). distriitittiiitto generalization via risk extrapolation (rex). ICMI..
Li, D., Yang, Y., Song. Y.-Z., and Hospedales, T. M. (2018). Learning to generalize: Meta-lcarning for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence.
Long, M., Zhu, H.. Wang, J., and Jordan, M. I. (2016). Unsupervised domain adaptation with residual transfer networks. Advances neural information processing systems, 29.
Pearl, J. (2010). Causal inference. In NIPS Causality: Objectives and Assessment.
Peters, J., Bühlmann, P., and Meinshausen, N. (2016). Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society, Series B (Statistical Methodology), pages 9012
Rosenfeld, E., Ravikumar, P., and Risteski, A. (2020). The risks of invariant risk minimization. arXiv preprint arXiv:2010.05761.
Rosenfeld, E., Ravikumar, P., and Risteski, A. (2022). An online learning approach to interpolation and extrapolation in domain generalization. In International Conference Artificial Intelligence and Statistics, pages 2641-2657. PMLR.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. (2019). Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: /9/ 1.8731..
10

---

Under review as a conference paper at ICLR 2023
Schrouff, J., Harris, N.. Koyejo, O., Alabdulmohsin, I., Schnider, E., Opsahl-Ong, K., Brown, A., Roy, S., Mincu, D., Chen, C., et al. (2022). Maintaining fairness across distribution shift: do we have viable solutions for real-world upplications? arXiv preprint arXiv:2202.01033.
Shen, Z., Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. (2021). Towards -ittof-distribution generalization: A survey, arXiv preprini arXiv: 13624.
Singh, H., Singh, R., Mhasawade, V., and Chunara, R. (2021). Faimesss violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 3-13.
Vapnik, V. (1991). Principles of risk minimization for learning theory. In NIPS, volume 91, pages 831-840.
Wang, H., Si. H., Li, B., and Zhao, H. (2022). Provable domain generalization via invariant-featurr subspace recovery. In ICML.
Zhang, M., Marklund, H., Dhawan, N., Gupta, A., Levine, S., and Finn, C. (2021). Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems, 34.
Zhao, H., des Combes, R. T., Zhang, K., and Gordon, G. J. (2019). On learning invariant representa- tions for domain adaptation. In ICML.

---

Under review as a conference paper at ICLR 2023
A SIMULATED DATA
We observe 2 domain with parameters o===0 = o = 0.1, o = = 1, each with 1000 samples. We let, and use linear feature extractors and predictors. Minimizing the TCRI objective (Equation 3) recovers a linear feature representation that maps back to 2 (Table I Note that for ERM, À = 3 a IRM, À 0.1, 3 a and TCRI, 0.1, 8 10 and 0.75; additional details can be found in Appendix
In addition letting 0. 1. be dummy variable, we also solve the OLS (Ordinary Least Squares (X xx) y) problem to compute the Legv term in the loss. Each backward pass takes in all examples from a domain batch.
B HYPERPARAMETER SELECTION
We do random search over hyperparameters for our method six randomly selected hyper- parameter sets in total. Additionally, for each set, we run three trials to generate standard er- rors. Additional sampling details can be found in We use the default values for the baseline algorithms since the results closely match those reported by Gulrajani and Lopez-Paz (2020). The hyperparameters used for each trial are also provided in the supplemental material.
C MODEL SELECTION
Across the hyperparameter sweep, we select the model with the lowest average conditional indepen- dence score between the two TCRI representations (Definition 4.3) to evaluate test set. This is in lieu of selecting the model with the highest validation accuracy on the training domains.
Additionally, show results based oracle selection, that is, selection based held-out target domain data. We observe that TCRI still outperforms the baseline methods and has accuracies in the source domains that closer to the baselines. This suggests that the regularizers harsh that the TCRI models are not able to learn good predictors in practice. The results, however, do suggest that there is room for improvement in model selection. We leave this for future work.
Table 3: Colored MNIST. Columns (+90%, +80%, -90% } indicate domains {0.1, 0.2, 0.3, 0.9) digit label and color correlation, respectively. ğ. & indicate the mean and standard deviation of the average domain accuracies, over trials each, respectively. Using the oracle selection method held out target data.
Domains
Domain Accuracy Statistics
Algorithm
+90%
+80% %
-90 %
std
min
ERM
61.9
66.3
26.5
51.6
17.8
26.5
IRM
73.0
72.2
51.0
65.4
10.2
51.0
GroupDRO
64.8
68.0
26.0
53.0
19.1
26.0
MIDG
68.8
72.7
28.6
56.7
19.9
28.6
ARM
81.6
73.4
24.2
59.7
25.3
24.2
VREx
70.2
70.8
49.7
63.6
9.8
49.7
TCRI (cov)
61.6
66.4
53.0
60.3
5.5
53.0
TCRI (HSIC)
68.2
67.7
56.4
64.1
5.4
56.4
D ON BENCHMARK DATASETS FOR EVALUATING DOMAIN GENERALIZATION WORST-CASE
We show some results below that illustrate the challenge of accurately evaluating the efficacy of an algorithm in domain generalization. We first note that we expect ERM (naive) to perform poorly in domain generalization tasks, certainly so when observe worst-case shifts test time. However, like other works (Gulrajani and Lopez-Pazi 2020), we observe that ERM performs as well as other baselines during transfer various benchmark datasets. Previous theoretical results (Rosenfeld et al.
12

---

Under review as a conference paper at ICLR 2023
2022) suggest that this observation is indicative of properties of the benchmark domains that may be sufficient for domain generalization with ERM specifically that the distribution (and equivalently loss) of the target domain can be written convex combination of the those in the source domains.
To further investigate this, we develop additional experiments motivated by the ColoredMNIST (Arjovsky al. 2019) which seems to not fall into the scenario in (Rosenfeld et al. 2022). We note that the +90%, +80%, and -90% domains of ColoredMNIST, the -90% domain has the opposite relationship between the spurious correlation and the label, so the use of spurious correlation generalizes catastrophically in the -90% domain. In the setting, the baseline algorithms we present achieve poor accuracy in the -90% domain while maintaining high accuracy in the +90% and +80% domains. Consequently, we investigate two settings, setting a: +90%, +80%, +70%, -90% domains and setting b: +90%, +80%, -80%, -90% domains. setting a, we add another domain with the majority direction in the relationship between spurious correlation and labels, and in setting b, we add another domain with the minority direction.
We use Oracle model selection (held-out target data) to remove the effect of model selection for all methods the results. We find that in setting a where we add a domain (+70%) that has spurious correlations that do not generalize the -90% domain, we observe worst-case accuracy across baselines still very different from the median-case
Table 4: Colored MNIST setting a. Columns (+90%, +80%, +70%, -90%) indicate domains
{0.1, 0.2, 0.3, 0.9) digit label and color correlation, respectively, A, indicate the mean and standard deviation of the average domain accuracies, over 3 trials each, respectively. Using the oracle selection method held out target data.
Domains
Domain Accuracy Statistics
Algorithm
+90%
+80%
+70%
-90%
mean
std
min
ERM
72.8 = 0.3
74.7 + 0.3
73.3 0.1.
16.3 + 1.5
59.3
24.8
16.3
IRM
49.0 = 0.1
54.2 + 2.0
50.3 0.3
43.8 + 2.8
49.3
3.7
43.8
GroupDRO
71.0 0.6
72.2 0.3
7..7
36.4 4.2
62.6
15.1
36.4
MLDG
72.8 = 0.9
74.8 0.3
13.6 0.3
58.5
26.0
13.6
ARM
74.7 = 0.4
0.2
14.0 0.4
59.0
26.0
14.0
VREx
74.1 = 1.3
0.5
72.1 1..1
19.5 19.5:555
59.6
23.2
19.5
TCRI (cov)
68.5 4.4
66.4 1..4-65
67.8 + 2.9
53.6 2.3
64.
6.1
53.6
TCRI (HSIC)
72.1 = 1.5
73.6 + 0.4
72.6 + 0.4
49.9
4 0.3
67.0
9.9
49.9
However, in setting b, where add a domain (-80%) that has spurious correlations that generalize to the -90% domain, we observe that the worst-case accuracy is much closer than the median-case single digit standard deviation across domains
Table 5: Colored MNIST setting b. Columns (+90% +80%, -80%, -90% indicate domains
{0.1, 0.2, 0.8, 0.9} digit label and color correlation, respectively. A, à indicate the mean and standard deviation of the average domain accuracies, over trials each, respectively. Using the oracle selection method held out target data.
Domains
Domain Accuracy Statistics
Algorithm
+90%
+80%
-80 %
-90 %
mean
std
min
ERM
58,4 = 1.3
67.0 + 0.5
64.2 + 2.0
52.6 3.2
60.6
5.5
52.6
IRM
56.7 = 3.3
56.6 2.8
51.6 0.7
51.7 0.7
54.2
2.5
51.6
GroupDRO
69.7 = 0.8
71.7 0.3
72.0
71.4 1.9
71.2
0.9
69.7
MILG
60.6 = 0.3
64.6 + 1.0
66.7 0.5
55.6 2,4
61.9
4.2
55.6
ARM
67.5 = 0.4
65.5 + 1.6
66.7
64.7
66.
64.7
VREx
67.4 1.9
7..4
7.22
5.
67.1
4.7
59.4
TCRI (cov)
67.6 0.8
14.05..4
5.5
6.65
64.0
2.3
61.5
TCRI (HSIC)
62.2 = 4.4
70.0 + 1.3
67.9 + 1.4
65.4 + 2.8
66.4
2.9
62.2
13

---

Under review as a conference paper at ICLR 2023
E THEORETICAL RESULTS
Lemma E.1. (Sufficiency of TCRI for Causal Aggregation). Recall, X, Ze, Ze, Y from Figure 7 Let Zo Le be direct causes and direct effects Yf,, respectively, and recall that X is function of Zo and Ze- If the two representations induced by feature extractors Q V satisfy TCRI, then wlog I(((X); Y))
Proof.
(i) First we define K Z to be random variables with non-zero mutual information with Ze marginally and conditioned on Y: [( Ze) > 0, ( Z. >
(ii) Furthermore, we have from (i.) that for any pair Z Z E iZ...... z% I(:: Z Y) 2 O, since neither be made conditionally independent of Ze given Y.
(iii) Given the whlll-hhaainnnaaaia criterion, we have that there exist a set of K Z 1 across (X),, (X) s.t. ( k Y) 2 I (Za; Y') for some K.
(iv) Combining (ii) and (iii), we have that all Z are aggregated in of the representa- tions, say ④. since for any Z that satisfies (i.), (ii.) = z ((X), and therefore (ii.. I(*(X):Y) 2 (Ze: Y).
14


Reviews:
[{"review_id": "dtoFvYIPOA9", "reviewer": "Reviewer_ZJFn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to learn domain-general representation by achieving the target conditioned representation independence (TCRI). Specifically, the proposed TCRI method not only addresses the domain invariant property but also assumes that the domain-general feature and domain-specific feature are conditionally independent of each other when given class label $Y$ and domain label $e$. By introducing two constraints to the general loss function which ensure 1) a good predictor, and 2) conditional independence, the TCRI method is demonstrated to achieve domain generality compared to other OOD generalization methods. Theoretical analysis and experimental comparisons are provided to validate TCRI.", "summary_of_the_review": "I have carefully read the whole paper. This paper is well-motivated, however, there are still some concerns (see weaknesses). If the authors can address my concerns, I will consider raising my score.", "strength_and_weaknesses": "Strength:\n- This paper is based on a reasonable SCM to analyze the data-generating process. As a result, the existing method only solves the domain invariant constraints, failing to address the target conditional independence. Hence, TCRI can reasonably achieve better performance than other methods by addressing the target conditional independence.\n- This paper has good theoretical support, making TCRI a solid research work.\n\nWeakness:\n- It is unclear how the target conditioned representation independence property can be break down into the proposed total-chain-information-criterion $I(\\Phi(X), \\Psi(X); Y)=I(Z_c, Z_e; Y)$ and $\\Phi(X)\\indep\\Psi(X)|Y \\forall e_i$. There is no detailed derivation.\n- Furthermore, how the two properties $I(\\Phi(X), \\Psi(X); Y)=I(Z_c, Z_e; Y)$ and $\\Phi(X)\\indep\\Psi(X)|Y \\forall e_i$ can be solved by the proposed $L_{\\Phi\\oplus\\Psi}$ and $L_{\\text{CI}}$. Please give more details.\n- How to choose the hyper-parameter $\\alpha$ and $\\beta$ are not discussed. Additionally, there is no experimental analysis of the importance of each loss term.\n- The experiments are quite insufficient. All experiments are conducted on coloredMNIST which is a very small dataset synthesized artificially, not a real-world dataset as indicated by the title of Section 6.1. So the realistic applicability is quite doubtful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GijwUqaVOA8", "reviewer": "Reviewer_xK43", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a Target Conditioned Representation Independence objective for domain generalization. It claims that a domain-invariant representation may not extend to test domains and refine it to a domain-general representation. Based on this, this paper proposed the TCRI objective and evaluate it on synthetic and real-world data.", "summary_of_the_review": "The proposed method is interesting. It lacks evaluation results.", "strength_and_weaknesses": "Strengths:\n- This paper is well written.\n- This paper gives a new attempt to explain and solve domain generalization in the view of causality. \n- This paper contains detailed proof and designs the method based on the analysis.\n\nWeaknesses:\n- The method requires a classifier for each domain, and it is evaluated on a three-domain setting {0.1, 0.2, 0.9}. How the results would be if there are more domains? Why choose {0.1, 0.2, 0.9} in the experiment?\n- This paper focuses on methodology. Only one real-world dataset is analyzed. It would be better to have more results.\n- The method achieves much better results under “-90%” setting, but for the other settings, the results are not as strong as expected, showing the limitation.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FiyYBU7Td9", "reviewer": "Reviewer_N8XZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides necessary and sufficient conditions for domain generalization. Experimental results on a few benchmark datasets are provided.", "summary_of_the_review": "The paper presents an approach for domain generalization. The writing style is harsh. Experimental results are not convincing.", "strength_and_weaknesses": "Strengths: A theoretical foundation for generating representations that are domain-invariant and domain generalizable are provided.\nWeaknesses: Poorly written paper. Experimental results using synthetic data or variations of MNIST are not convincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CIudRdGyQB", "reviewer": "Reviewer_FowA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an objective for domain generalization. The main findings involve two types of representations: domain invariant - the distribution of the target conditioned on such a representation is the same across training environments; and domain domain-general - the representation is domain invariant across all possible environments. The paper proves, by construction that a representation that captures only a subset of causal features could be domain-invariant but not necessarily domain-general. \n\nA criterion (Target Conditioned Representation Independence - TCRI ) helpful for domain generality is proposed, using two representations, one domain invariant $\\Phi(X)$ and one domain-specific $\\Psi(X)$. Informally, this criterion is satisfied when the two representations jointly have enough information while being independent when conditioned on the target. The paper proves that if $\\Phi(X)$ and $\\Psi(X)$ satisfy TCRI and $\\Phi(X)$ satisfies domain independence then $\\Phi(X)$ is domain-general.\n\nThen, a loss function is proposed to satisfy this criterion and experimentally it is shown that it can achieve better word-case performance. An important aspect is that the loss used for conditional independence (based on HSIC criterion) can be used for model selection using only training domain data.\n", "summary_of_the_review": "The paper was some good observations and it is useful for domain generalization, especially in cases where the worst-case is crucial and model selection is hard.\n", "strength_and_weaknesses": "**Strong points**\n\nS1. The paper tackles an important problem in domain generalization and proposes a sound method for it. The presented downsides of using representations that are domain-invariant but not necessary domain-general are valid and the proposed method solves them.\n\nS2. By improving on other invariant predictors, like IRM, the proposed method has a large potential for domain generalisation and represents a good contribution.\n\nS3. In general, model selection for domain generalization is a hard problem, and the proposed method seems to achieve a good selection using only the training data.\n\n**Weak Points**:\n\nW1. In practice, even when the proposed model selection is used, the performance for all environments is quite low. This hints that the constraints could be too hard to achieve and the model is hard to optimize. What is the performance of the model if the model selection is using testing domain data (oracle selection)? \n\nW2. Some additional explanations in the proofs are needed.  In step iv of the proof of Lemma 5.3, the variables $Z_c^{i′}$ are aggregated in one of the two representations, but how to choose which one? Is it possible that the variables be aggregated into the representation $\\Psi(X)$ that is *not* optimized to be domain-invariant? \n\nW3. Is it necessary to have the two representations? Could an assumption such as \n$I(\\Phi(X); Y) = I(Z_c, Z_e; Y)$, together with domain-invariance assumption be enough to get the same result as Lemma 5.3 and Theorem 5.4? These assumptions would be satisfied when optimizing $ \\mathcal{L_\\Phi} + \\lambda \\mathcal{L}_{IRMv1’}$, i.e. the usual way of training IRM. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject