Title: Partial Transportability for Domain Generalization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causal inference

Domain generalization

Transportability
Abstract: A fundamental task in AI is providing performance guarantees for predictions made in unseen domains. In practice, there can be substantial uncertainty about the distribution of new data, and corresponding variability in the performance of existing predictors. Building on the theory of partial identification and transportability, this paper introduces new results for bounding the value of a functional of the target distribution, such as the generalization error of a classifiers, given data from source domains and assumptions about the data generating mechanisms, encoded in causal diagrams. Our contribution is to provide the first general estimation technique for transportability problems, adapting existing parameterization schemes such Neural Causal Models to encode the structural constraints necessary for cross-population inference. We demonstrate the expressiveness and consistency of this procedure and further propose a gradient-based optimization scheme for making scalable inferences in practice. Our results are corroborated with experiments.

Partial Transportability for Domain Generalization

Abstract

Afundamental task in AI is providing performance guarantees for predictions made in unseen domains. In practice, there can be substantial uncertainty about the distribution of new data, and corresponding variability in the performance of existing predictors. Building on the theory of partial identification and transportability, this paper introduces new results for bounding the value of a functional of the target distribution, such as the generalization error of a classifier, given data from source domains and assumptions about the data generating mechanisms, encoded in causal diagrams. Our contribution is to provide the first general estimation technique for transportability problems, adapting existing parameterization schemes such Neural Causal Models to encode the structural constraints necessary for cross-population inference. We demonstrate the expressiveness and consistency of this procedure and further propose a gradient-based optimization scheme for making scalable inferences in practice. Our results are corroborated with experiments.

1 Introduction

In the empirical sciences, the value of scientific theories arguably depends on their ability to make predictions in a domain different from where the theory was initially learned. Understanding when and how a conclusion in one domain, such as a statistical association, can be generalized to a novel, unseen domain has taken a fundamental role in the philosophy of biological and social sciences in the early 21st century. As Campbell and Stanley [8, p. 17] observed in an early discussion on the interpretation of statistical inferences, 'Generalization always turns out to involve generalization into a realm not represented in one's sample' where, in particular, statistical associations and distributions might differ, presenting a fundamental challenge.

As society transitions to become more AI centric, many of the every-day tasks based on predictions are increasingly delegated to automated systems. Such developments make various parts of society more efficient, but also require a notion of performance guarantee that is critical for the safety of AI, in which the problem of generalization appears under different forms. For instance, one critical task in the field is domain generalization, where one tries to learn a model (e.g. classifier, regressor) on data sampled from a distribution that differs in several aspects from that expected when deploying the model in practice. In this context, generalization guarantees must build on knowledge or assumptions on the 'relatedness' of different training and testing domains; for instance, if training and testing domains are arbitrarily different, no generalization guarantees can be expected from any predictor [12, 40]. The question becomes how to link the domains of data that are used to train a model (a.k.a., the source domains) to the domain where this model is deployed in practice (a.k.a., the target domain).

Source domains M1, M2

Target domainM*

UwY

M

UWY

M

S2

WY

W

S2

S

S

Z

Z

D1~Pl,D²~P2

D*=

Given D',D2 and S: Rp*(h) = ?

To begin to answer this question, a popular type of assumption that relates source and target domains is statistical in nature: invariances in the marginal or conditional distribution of some variables across the source and target distributions. Examples include assumptions of covariate shift and label shift (among others) [35, 34]. Notably, generalization is justified by the stability and invariance of the causal mechanisms shared across the domains [14, 21], since the distributional/statistical invariances across the domains are consequences of mechanistic/structural invariances governing the underlying data generating process. Although the induced statistical invariances, once exploited correctly, can be used as bases for generalizability. Broadly, invariance-based approaches to domain generalization [27, 29, 2, 40, 24, 20, 7, 6, 13] search for predictors that not only achieves small error on the source data but also maintain certain notions of distributional invariance across the source domains. Since these statistical invariances can be viewed as proxies to structural invariances, in certain instances generalization guarantees can be provided through causal reasoning [17, 31, 39]. This idea can be illustrated in Fig. 1. The value of variables t C, Y, W, Z u are determined as a stochastic function of variables pointing to it, while these functions may differ across domains. The challenge is to evaluate the generalization risk of a model, e.g. R P ˚ p h q : ' E P ˚ rp Y ´ h q 2 s for h : ' h p C, W, Z q ' E P 1 r Y | C, W, Z s , without observations from the target P ˚ . General instances of this challenge have been studied under the rubric of the theory of causal transportability, where qualitative assumptions regarding the underlying structural causal models are encoded in a graphical object, and algorithms are designed to leverage these assumptions and compute certain statistical queries in the target domain in terms of the existing source data [26, 4, 5, 19, 11, 17].

Despite these advances, in practice, the combination of source data and graphical assumptions is not always sufficient to identify (uniquely evaluate) the desired statistical query, e.g., the average loss of a given predictor in the target domain. In this case, the query is said to be non-transportable 3 . For example, given Fig. 1, R P ˚ p h q is non-transportable for the classifier h : ' h p C, W, Z q . In this paper, we study the fundamental task of computing tight upper-bounds for statistical queries in a new unseen domain. This allows us to assess worst-case performance of prediction models for the domain generalization task. Our contributions are as follows:

Sections 2 & 3. We develop the first general estimation technique for bounding the value of queries across multiple domains (e.g., the generalization risk) in non-transportable settings (Def. 4). Specifically, we extend the formulation of canonical models [3, 42] to encode the constraints necessary for solving the transportability task, and demonstrate their expressiveness for generating distributions entailed by the underlying Structural Causal Models (SCMs) (Thm. 1).

Section 4. Weadapt Neural Causal Models (NCMs) [41] for the transportability task via a parameter sharing scheme (Thm. 2), similarly demonstrating their expressiveness and consistency for solving the partial transportability task. We then leverage the theoretical findings in sections 2 & 3 to implement a gradient-based optimization algorithm for making scalable inferences (Alg. 1), as well as a Bayesian inference procedure. Finally, we introduce Causal Robust Optimization (CRO) (Alg. 2), an iterative method to find a predictor with the best worst-case risk.

Preliminaries. We use capital letters to denote variables ( X ), small letters for their values ( x ), bold letters for sets of variables ( X ) and their values ( x ), and use supp to denote their domains of definition ( x P supp X ). A conditional independence statement in distribution P is written as p X | ù Y | Z q P . A d -separation statement in some graph G is written as p X | ù d Y | Z q . To denote P p Y ' y | X ' x q , we use the shorthand P p y | x q . The basic semantic framework of our analysis relies on Structural Causal Models (SCMs) [25, Definition 7.1.1], which are defined below.

Definition 1. An SCM M is a tuple M ' x V , U , F , P y where each observed variable V P V is a deterministic function of a subset of variables Pa V Ă V and latent variables U V Ă U , i.e., v : ' f V p pa V , u V q , f V P F . Each latent variable U P U is distributed according to a probability measure P p u q . We assume the model to be recursive, i.e. that there are no cyclic dependencies among the variables. ˝

SCM M entails a probability distribution P M p v q over the set of observed variables V such that

ż

where the term P p v | pa V , u V q corresponds to the function f V P F in the underlying structural causal model M . It also induces a causal diagram G M in which each V P V is associated with a vertex, and we draw a directed edge between two variables V i Ñ V j if V i appears as an argument of f V j in the SCM, and a bi-directed edge V i Ø V j if U V i X U V j ‰ H , that is V i and V j share an unobserved confounder. Throughout this paper, we assume the observational distributions entailed by the SCMs satisfy the positivity assumption, that is, P M p v q ą 0 , for every v . We will also operate non-parametrically, i.e., making no assumption about the particular functional form or the distribution of the unobserved variables.

2 Risk Evaluation through Partial Transportability

In this section, we focus on challenges of the domain generalization problem through a causal lens, in particular regarding assessment of average loss of a given classifier in the target domain. We study a system of variables V where Y P V is a categorical outcome variable and consider a classifier h : supp X Ñ supp Y mapping a set of covariates X Ă V to the domain of the outcome. The setting of domain generalization is characterized by multiple domains, defined by SCMs M : t M 1 , . . . , M K , M ˚ u that entail distributions P ' t P M 1 , . . . , P M K u and P M ˚ over V . We are given a classifier h : supp X Ñ supp Y , and the objective is to evaluate its risk in the domain M ˚ which is defined as,

where L : supp Y ˆ supp Y Ñ R ` is a loss function. The following example illustrates these notions. Example 1 (Covariate shift) . A common instance of the domain generalization problem considers source and target domains M : t M 1 , M ˚ u over V ' t X , Y u and U ' t U X , U Y u defined by

Here, because P 1 p U X q ‰ P 1 p U X q , this implies via Eq. 1 that the covariate distributions are different, i.e., P 1 p X q ‰ P ˚ p X q . Still, the label distribution conditional on covariates is invariant, i.e., P 1 p Y | X q ' P ˚ p Y | X q , also known as the covariate shift setting. Accordingly, the risk of a classifier h : ' h p x q can be written as,

We will consider the problem of quantifying the variation in R P ˚ p h q subject to variation in P ˚ p x q that would be consistent with partial observations from these domains, e.g. samples from P 1 p X , Y q , and assumptions about the commonalities and discrepancies across the domains. ˝

To describe more general discrepancies in the mechanisms between the SCMs, we adapt the notion of domain discrepancy and selection diagram introduced in [19].

Definition 2 (Domain discrepancy) . For SCMs M i , M j ( i, j P t˚ , 1 , 2 , . . . , K u ) defined over V , the domain discrepancy set ∆ ij Ď V is defined such that for every V P ∆ ij there might exist a discrepancy f M i V ‰ f M j V , or P M i p u V q ‰ P M j p u V q . For abbreviation, we denote ∆ i ˚ as ∆ i . ˝

In words, if an endogenous variable V is not in ∆ ij , this means that the mechanism for V (i.e., the function f V and the distribution of exogenous variables P p u V q ) are structurally invariant across M i , M j . What follows integrates the domain discrepancy sets into a generalization of causal diagrams to express qualitative assumptions about multiple SCMs [26, 11].

Definition 3 (Selection diagram) . The selection diagram G ∆ i is constructed from G i ( i P t 1 , 2 , . . . , T u ) by adding the selection node S i to the vertex set, and adding the edge S i Ñ V for every V P ∆ i . The collection G ∆ ' t G ˚ u Ť t G ∆ i u i Pt 1 , 2 ,...,T u encodes the graphical assumptions. Whenever the causal diagram is shared across the domains, a single diagram can depict G ∆ . ˝

Selection diagrams extend causal diagrams and provide a parsimonious graphical representation of the commonalities and disparities across a collection of SCMs. The following example illustrates these notions and highlights various subtleties in the generalization error of different predictors.

Example 2 (Generalization performance of classifiers) . Consider the SCMs M i ( i P t 1 , 2 , ˚u ) over the binary variables X ' t C 1 , C 2 , . . . , C 10 u Ť t W,Z u and Y , defined as follows:

À denotes the xor operator, i.e. , A À B evaluates to 1 if A ‰ B and evaluates to 0 if A ' B . Notice that the distribution of exogenous noise associated with C 1:10 and t W u differs across the domains. Consider three baseline classifiers h 1 p c , w q : ' w ' À c P c c, h 2 p c q : ' À c P c c, h 3 p z q : ' z evaluated on data from P 1 , P 2 , P ˚ with the symmetric loss function L p Y, h p X qq ' ✶ t Y ‰ h p X qu . Their errors are given in Table 1. Notice that h 1 has almost perfect accuracy on both source distributions, but does not generalize to M 1 as it uses the unstable feature W , incurring 50% loss. This observation indicates that mere minimization of the empirical risk might yield arbitrarily large risk in the unseen target domain. h 2 uses the features C that are the direct causes of Y , also known as the causal predictor [27, 2], and yields a stable loss of 20% across all domains. On the other hand, h 3 uses only Z that is a descendant of Y , yet achieves a small loss across all domains as the mechanism of Z is assumed to be invariant. This observation is surprising, because h 3 is neither a causal predictor nor the minimizer of the empirical risk, yet it performs nearly optimally on all domains. ˝

Example 2 illustrates potential challenges of the domain generalization problem, particularly regarding the variation of the risk of classifiers across the source and target domains. The following definition introduces the problem of 'partial transportability' which is the main conceptual contribution of our paper. The objective is bounding a statistic of the target distribution using the data and assumptions available about related domains.

Definition 4 (Partial Transportability) . Consider a system of SCMs M : t M 1 , M 2 , . . . , M K , M ˚ u that induces the selection diagram G ∆ over the variables V and entails the distributions P : t P 1 p v q , P 2 p v q , . . . , P K p v qu and P ˚ p v q . A functional ψ : Ω V Ñ R is partially transportable from P given G ∆ if,

where q max P R is a constant that can be obtained from P given G ∆ . ˝

For instance, finding the worst-case performance of a classifier based on the source distributions given the selection diagram is a special case of partial transportability with ψ p x , y q : ' L p y, h p x qq . In principle, this task is challenging as the exogenous distribution P ˚ p U V q and structural assignments f ˚ V of variables V P V that do not match with any of the source domains could be arbitrary. In the following section, we will define tractable parameterization of t P p U q , F u to derive a systematic approach to solving partial transportability tasks.

3 Canonical Models for Partial Transportability

We begin with an example to illustrate how one might approach parameterizing a query such as E P M ˚ r ψ p V qs , e.g. , the generalization error, to consistently solve the partial transportability task.

Example 3 (The bow model) . Let X : ' t X u be a single binary variable, and Y be a binary label. Consider two source domains defined by the following SCMs:

The task is to evaluate the generalization error of the classifier h p x q ' ␣ x . h can be shown optimal in both source domains: achieving R P 1 p h q « 0 . 11 and R P 2 p h q « 0 . 06 . However, it is unclear whether it generalizes well to a target domain M ˚ , given the domain discrepancy sets ∆ 1 ' t X u , ∆ 2 ' t Y u . ˝

Balke and Pearl [3] derived a canonical parameterization of SCMs such as t M 1 , M 2 , M ˚ u in Example 3. They showed that it is sufficient to parameterize P p U q with correlated discrete latent variables R X , R Y , where R X determines the value of X , and R Y determines the functional that decides Y based on X . The causal diagrams are shown in Figure

(a)

G

∆

(b)

G

N

2. Canonical SCMs entails the same set of distributions as the true underlying SCMs, i.e. are equally expressive. In particular, Zhang and Bareinboim [42] showed that for every SCM M , there exists an SCM of the described form specified with only a distribution P p r X , r Y q , where, supp R X ' t 0 , 1 u , supp R Y ' t y ' 0 , y ' 1 , y ' x, y ' ␣ x u . The joint distribution P p r X , r Y q can be parameterized by a vector in 8-dimensional simplex, and entails all observational, interventional and counterfactual variables generated by the original SCM.

The following definition by Zhang et al. [42] provides a general formulation of canonical models.

Definition 5 (Canonical SCM) . A canonical SCM is an SCM N ' x U , V , F , P p U qy defined as follows. The set of endogenous variables V is discrete. The set of exogenous variables U ' t R V : V P V u , where supp R V ' t 1 , . . . , m V u and m V ' |t h V : supp pa V Ñ supp V u| . For each V P V , f V P F is defined as f V p pa , r V q ' h p r V q p pa q .

Example 3 (continued) . Consider extending the canonical parameterization to to solve the partial transportability task by optimization. Each SCM M 1 , M 2 , M ˚ is associated with a canonical SCM N 1 , N 2 , N ˚ . with exogenous variables t R X , R Y u as above. The domain discrepancy sets ∆ indicate that certain causal mechanisms need to match across pairs of the SCMs. For example, ∆ 1 ' t X u , which does not contain Y , and this means that (1) the function f Y is the same across M 1 , M ˚ , and (2) the distribution of unobserved variables that are arguments of f Y , namely, U XY , U Y remains the same across M 1 , M ˚ . Imposing these equalities on the canonical parameterization is straightforward as (1) the function f Y is the same across all canonical SCMs by construction, and (2) the only unobserved variable pointing to variable V is R V (for V P t X,Y u ). Following the selection diagram shown in Fig. 2a, M 1 , M ˚ agree on the mechanism of Y , which translates to the constraint P N 1 p r Y q ' P N ˚ p r Y q . Similarly, M 2 , M ˚ agree on the mechanism of X that translates to the constraint P N 2 p r X q ' P N ˚ p r X q . Putting these together, the optimization problem below finds the upper-bound for the risk R P ˚ p h q for the classifier h p x q ' ␣ x :

Notably, the above optimization has a linear objective with linear equality constraints. ˝

This example illustrates a more general strategy, in which probabilities induced by an SCM over discrete endogenous variables V may be generated by a canonical model. What follows is the main result of this section, and provides a systematic approach to partial transportability using the canonical models.

Theorem 1 (Partial-TR with canonical models) . Consider the tuple of SCMs M that induces the selection diagram G ∆ over the variables V , and entails the source distributions P , and the target distribution P ˚ . Let ψ : Ω V Ñ R be a functional of interest. Consider the following optimization scheme:

where each N i is a canonical model characterized by a joint distribution over t R V u V P V . The value of the above optimization is a tight upper-bound for the quantity E P ˚ r ψ p V qs among all tuples of SCMs that induce G ∆ and entail P . ˝

In words, this Theorem states that one may tightly bound the value of a target quantity E P ˚ r ψ p V qs by optimizing over the space of canonical models subject to the proposed constraints, without any loss of information. An implementation of Thm. 1 approximating the worst-case error, by making inference on the posterior distribution of the target quantity, is provided in Appendix A.

4 Neural Causal Models for Partial Transportability

In this section we consider inferences in more general settings by using neural networks as a generative model, acting as a proxy for the underlying SCMs M with the potential to scale to real-world, high-dimensional settings while preserving the validity and tightness of bounds. For this purpose, we consider Neural Causal Models [41] and adapt them for the partial transportability task. What follows is an instantiation of [41, Definition 7].

Definition 6 (Neural Causal Model) . A Neural Causal Model (NCM) corresponding to the causal diagram G over the discrete variables V is is an SCM defined by the exogenous variables:

and the functional assignments V Ð f θ V p Pa V , U V q , where U V ' t U W P U : V P W u . The function f θ V is a feed-forward neural network parameterized with θ V that outputs in supp V . The distribution entailed by an NCM is denoted by P p v ; θ q , where θ ' t θ V u V P V .

To illustrate how one might leverage this parameterization to define an instance of partial transportability task consider the following example.

Example 4. Let SCMs M 1 , M 2 , M ˚ induce G ∆ shown in Fig. 3 over the binary variables X , Y , where X ' t C 1 , C 2 , Z 1 , Z 2 , W 1 , . . . , W 5 u . Let θ 1 , θ 2 , θ ˚ be the parameters of NCMs constructed based on the causal diagram in Fig. 3 (without the s-nodes). The objective is to constrain these parameters to simulate a compatible tuple of NCMs M θ 1 , M θ 2 , M θ ˚ that equivalently entail P 1 p x , y q , P 2 p x , y q and induce G ∆ .

For instance, the fact that S 2 is not pointing to Y suggests the invariance f ˚ Y ' f 2 Y and P ˚ p u Y q ' P 2 p u Y q for the true underlying SCMs. That same invariance may be enforced in the corresponding NCMs by relating the parameterization of M θ 2 , M θ ˚ , i.e., imposing that θ ˚ Y ' θ 2 Y for the NN generating Y . Similarly, the observed data D 1 , D 2 from the source distributions P 1 p x , y q , P 2 p x , y q , respectively, impose constraints on the parameterization of NCMs as plausible models must satisfy P p x , y ; θ 1 q ' P 1 p x , y q and P p x , y ; θ 2 q ' P 2 p x , y q . This may be enforced, for instance, by maximizing the likelihood of data w.r.t. the NCM parameters: θ i P arg max θ ř x ,y P D i log P p x , y ; θ i q , for i P t 1 , 2 u . By extending this intuition for all constraints imposed by the selection diagram and data, we narrow the set of NCMs M θ 1 , M θ 2 , M θ ˚ to a set that is compatible with our assumptions and data. Maximizing the risk of some prediction function R P ˚ p h q in this class of constrained NCMs might then achieve an informative upper-bound. ˝

Motivated by the observation in Example 4, we now show a more formal result (analogous to Thm. 1) that guarantees that the solution to the partial transportability task in the space of constrained NCMs achieves a tight bound on a given target quantity E P ˚ r ψ p V qs .

Theorem 2 (Partial-TR with NCMs) . Consider a tuple of SCMs M that induces G ∆ , P and P ˚ over the variables V . Let D i ' P i p x, y q denote the samples drawn from the i -th source domain. Let θ i denote the parameters of NCM corresponding to M i P M . Let E P ˚ r ψ p V qs be the target quantity. The solution to the optimization problem,

is a tuple of NCMs that induce G ∆ , entails P . In the large sample limit, the solution yields a tight upper-bound for E P ˚ r ψ p V qs . ˝

Theorem 2 establishes the expressive power of NCMs for solving partial transportability tasks. This formulation is powerful because it enables the use of gradient-based optimization of neural networks for learning and, in principle, might scale to large number of variables.

4.1 Neural-TR: An Efficient Implementation

We could further explore the efficient optimization of parameters by exploiting the separation between variables in the selection diagram. Rahman et al. [28], for instance, show that the NCM parameterization is modular w.r.t. the c-components of the causal diagram. We can similarly elaborate on this property, and leverage it for more efficient partial transportability.

In the following, we build towards an efficient algorithm for partial transportability using NCMs by first showing an example that describes how a given target quantity E P ˚ r ψ p V qs might be decomposed for learning more efficiently.

Example 4 (continued) . P p x , y ; θ ˚ q in the objective in Eq. (8) may be decomposed as follows:

where the subsets A 1 , A 2 , A 3 are the c -components of G ˚ . Notice, S 2 is not pointing to any of the variables A 2 , which means that their mechanism is shared across M 2 , M ˚ , and therefore,

This property is the basis of transportability algorithms [4, 10], and is known as the s-admissibility criterion [26], which allows us to deduce distributional invariances from structural invariances. By Eq. (9), we can replace the term P p a 2 | b 2 ; θ ˚ A 2 q in the objective with the probabilistic model P p a 2 | b 2 ; η 2 q that is trained with D 2 to approximate P 2 p a 2 | b 2 q and plug it into the objective Eq. (8) as a constant.

As a consequence, we do not need to optimize over the parameters θ 1 A 2 , θ 2 A 2 , θ ˚ A 2 from the partial transportability optimization problem. Similarly, since S 1 does not point to A 1 , we can substitute P p a 1 ; θ ˚ q with P p a 1 ; η 1 q , and pre-train it with data D 1 . In the context of Example 4 and the evaluation of R P ˚ p h q , the objective in Eq. (8) may be simplified to the substantially lighter optimization task:

In general, the parameter space of NCMs can be cleverly decoupled and the computational cost of the optimization problem can be significantly improved since only a subset of the conditional distributions need to be parameterized and optimized. This observation motivates Alg. 1 designed to exploit these insights. It proceeds by first, decomposing the query, second, computing the identifiable components, and third, parameterizing the components that are not point identifiable and running the NCM optimization routine. The following proposition demonstrates the correctness of this procedure.

Algorithm 1 Neural-TR

This result may be understood as an enhancement of Thm. 2 in which the factors that are readily transportable from source data are taken care of in a pre-processing step. The hybrid approach is especially useful in case researchers have pre-trained probabilistic models with arbitrary architecture that they can use off-the-shelf and avoid unnecessary computation.

4.2 Neural-TR for the Optimization of Classifiers

The Neural-TR algorithm can be viewed as an adversarial domain generator that takes a classifier h p z q as the input, and then parameterizes a collection of SCMs to find a plausible target domain that yields the worst-case risk for the given classifier, namely, ˆ θ ˚ . By flipping h p z q for some z P Ω we can reduce the risk of h under ˆ θ ˚ .

Interestingly, we can exploit Neural-TR to generate adversarial data for a given classifier and introduce an iterative procedure to progressively train classifiers with with minimum risk upper-bound. Algorithm 2 describes this approach. At each iteration, CRO uses Neural-TR as a subroutine to obtain an adversarially designed NCM ˆ θ ˚ that yields the worst-case risk for the classifier at hand. Next, it collects data D ˚ from this NCM and adds it to a collection of datasets D ˚ . Finally, it updates the classifier to be robust to the collection D ˚ by minimizing

Algorithm 2 CRO (Causal Robust Optimization)

the maximum of the empirical risk R D p h q : ' ř x ,y P D L p y, h p x qq across all D P D ˚ . We repeat this process until convergence of the upper-bound for risk. The following result justifies optimality of CRO for domain generalization; more discussion is provided in Appendix C.2.

Theorem 3 (Domain generalization with CRO) . Algorithm 2 returns a worst-case optimal solution;

In words, Thm. 3 states that the classifier returned by CRO, in the large sample limit, minimizes worst-case risk in the target domain subject to the constraints entailed by the available data and induced by the structural assumptions.

(a) Example 2

(b) Example 3

(c) CMNIST

(d) Example 2

(e) Example 3

h（x)=h(x)=h（x)=h（x)

hERM=hIRMhDRO

1.0

0.8

Rp*

Rp:

Rp:

1.0

8°0

Rp*

1.0

0.8

Rp*

Rp

Rp:

0.8

1.0

Rp*

1.0

0.8

Rp*

0.6

0.6

Rp

Rp

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0.0

0

Training epochs (Neural-TR)

25

50

75

100

0.0

Training epochs (Neural-TR)

0

25

50

75

100

0.0

Training epochs (Neural-TR)

0

25

50

75

100

0.0

Training epochs (Neural-TR)

0

25

50

75

100

0'0

Training epochs (Neural-TR)

0

25

50

75

100

5 Experiments

This section illustrates Algs. 1 and 2 for the evaluation and optimization of the generalization error on several tasks, ranging from simulated examples to semi-synthetic image datasets. The details of the experimental set-up and examples not fully described below, along with additional experiments, can be found in the Appendix.

5.1 Simulations

Worst-case risk evaluation Our first experiment revisits Examples 2 and 3 for the evaluation of the worst-case risk R P ˚ of various classifiers with Neural-TR (Alg. 1).

In Example 2 we had made (anecdotal) performance observations for the classifiers h 1 p c , w q : ' w ' À c P c c, h 2 p c q : ' À c P c c, h 3 p z q : ' z in a selected target domain M ˚ . We now consider providing a worst-case risk guarantee with Neural-TR for any (compatible) target domain. The main panel in Fig. 4a shows the convergence of the worst-case risk evaluator over successive training iterations (line 15, Alg. 1), repeated 10 times with different model seeds and solid lines denoting the mean worst-case risk. The source performances R P 1 , R P 2 are given in the two right-most panels for reference. We observe that the good source performance of h 2 p c q and h 3 p z q generalizes to all possible target domains consistent with our assumptions, while the classifier h 1 p c , w q diverges, with an error of 90% in the worst target domain. In Example 3, we consider the evaluation of binary classifiers h P t h 1 p x q : ' x, h 2 p x q : ' ␣ x, h 3 p x q : ' 0 , h 4 p x q : ' 1 u . h 2 p x q ' ␣ x . Our results are given in Fig. 4b, highlighting the extent to which source performance need not be indicative of target performance. With these results, we are now in a position to confirm the desirable performance profile of h 2 , even in the worst-case, as hypothesized in Example 3.

Worst-case risk optimization For each one of the examples above, we implement CRO (Alg. 2) to uncover the theoretically optimal classifier in the worst-case. The worst-case risks of the classifiers learned by CRO, denoted h CRO, are given by 0 . 05 for Example 2 and 0 . 18 for Example 3. The worst-case risk evaluation results (with Neural-TR, as above) are given in Figs. 4d and 4e. It is interesting to note that these errors coincide with the best performing classifiers considered in the previous experiment, i.e. h 3 p z q : ' z for Example 2 and h 2 p x q ' ␣ x for Example 3. In fact, by comparing the outputs of CRO h CRO with these classifiers, we can verify that the classifiers learned by CRO in these examples are precisely the mappings h CRO p z q : ' z and h CRO p x q ' ␣ x which is remarkable. By Thm. 3, h 3 p z q : ' z and h 2 p x q ' ␣ x are the theoretically best worst-case classifiers among all possible functions given the data and assumptions.

5.2 Colored MNIST

Our second experiment considers the colored MNIST (CMNIST) dataset that is used in the literature to highlight the robustness of classifiers to spurious correlations, e.g. see [2]. The goal of the classifier is to predict a binary label Y P t 0 , 1 u assigned to an image Z P R 28 ˆ 28 ˆ 3 based on whether the digit in the image is greater or equal to five. MNIST images W P R 28 ˆ 28 are grayscale (and latent), and color C P t red , green u correlates with the class label Y .

Following standard implementations, we construct datasets from three domains with varying correlation strength between the color and image label: set to 90% agreement between the color C ' red and label Y ' 1 in source domain M 1 , and 80% in source domain M 2 . We consider performance evaluation and optimization in a target domain M ˚ with potential discrep-

W

Y

C

Z

's

S2

(a) Iteration 1

(b) Iteration 2

(c) Iteration 3

Samples from D*

Samples from D*

Samples from D*

1.0

Rp*

Y=0

Y =1

Y = 1

1.0

Rp*

Y=0

4

Y =0

Y = 1

1.0

Rp*

Y = 1

Y =0

Y = 1

3

0.6

0.4

0.2

0.8

MM

h(z)

10

10

Y=0

19

20

Y=1

20 20

7

R

Y =0

10

29

0.8

0.6

0.4

0.2

h(z)

20

19

Y = 1

19

29

10

20

9

Y = 1

10

20

Y =1

29

0.8

0.6

0.4

0.2

h(z)

29

29

Y = 1

10

20

10

29

10

Y=1

10

20

10

Y = 1

30

20

0.0

0

Training epochs (Neural-TR)

25

50

75

100

20

150

0.0

0

Training epochs (Neural-TR)

25

50

75

100

29

29

20

10

业

0.0

0

Training epochs (Neural-TR)

25

50

75

100

10

20

10

20

29

ancies in the mechanism for C , rendering the correlation between color and label unstable. The selection diagram is given in Figure 5.

Worst-case risk evaluation Consider a setting in which we are given a classifier h : Ω Z Ñ Ω Y , and the task is to assess its generalizability with a symmetric 0-1 loss function. We use data drawn from P 1 , 2 p z , y q to train predictors using Empirical Risk Minimization (ERM) [38], Invariant Risk Minimization (IRM) [2], and group Distributionally Robust Optimization (group DRO) [32], namely h ERM p z q , h IRM p z q , and h DRO p z q respectively; more detailed discussion about the role of invariance and robustness in domain generalization is available in appendix D. Using Neural-TR, we observe in Fig. 4c that the worst-case risk of h ERM in a target domain with a discrepancy in the color assignment is approximately 0 . 95 , h DRO achieves 0 . 90 worst-case risk, and h IRM achieves 0 . 65 worst-case risk. Either method perform worse than the baseline, that is random classification with risk 0 . 5 . On this task, a classifier trained on gray-scale images W achieves a worst-case error of 0 . 25 .

Worst-case risk optimization We now ask whether we could learn a theoretically optimal classifier in the worst-case with CRO (Alg. 2). Fig. 6 illustrates the training process over several iterations. Specifically, given a randomly initialized h , we infer the NCM ˆ θ ˚ that entails worst-case performance of h (in this case, chance performance R P ˚ p h q ' 0 . 5 ) and generate data D ˚ from ˆ θ ˚ , shown in Fig. 6a. In a second iteration, a new candidate h is trained to minimize worst-case risk on D ' D ˚ . Note that in D ˚ , we observe an almost perfect association between the color C ' green and label Y ' 1 : h therefore is encouraged to exploit color for prediction. Its worst-case error (inferred with Neural-TR) is accordingly close to 1, and the corresponding worst-case NCM ˆ θ ˚ entails a distribution of data in which the correlation between color and label is flipped: with a strong association between the color C ' red and label Y ' 1 , as shown in Fig. 6b. In a third iteration, a new candidate h is trained to minimize worst-case risk on the updated D ˚ with data samples from the previous two iterations (exhibiting opposite color-label correlations). By construction, this classifier is trained to ignore the spurious association between color and label, classifying images based on the inferred digit which leads to better behavior in the worst-case: achieving a final error of approximately 0.25, as shown in Fig. 6c, which is theoretically optimal. Note, however, that the poor performance of the baseline algorithms is not directly comparable to that of CRO, since CRO has access to background information (selection diagrams) that can not be communicated with the baseline algorithms. CRO may thus be interpreted as a meta-algorithm that operates with a broader range of assumptions encoded in a certain format (i.e., the selection diagram) that enable it to find the theoretically optimal classifier for domain generalization, in contrast to the baseline algorithms.

6 Conclusion

Guaranteeing the performance of ML algorithms implemented in the wild is a critical ingredient for improving the safety of AI. In practice, evaluating the performance of a given algorithm is non-trivial. Often the performance may vary as a consequence of our uncertainty about the possible target domain, also called a non-transportable setting. In this paper, we provide the first general estimation technique for bounding an arbitrary statistic such as the classification risk across multiple domains. More specifically, we extend the formulation of canonical models and neural causal models for the transportability task, demonstrating that tight bounds may be estimated with both approaches. Building on these theoretical findings, we introduce a Bayesian inference procedure as well as a gradient-based optimization algorithm for scalable inferences in practice. Moreover, we introduce Causal Robust Optimization (CRO), an iterative learning scheme that uses partial transportability as a subroutine to find a predictor with the best worst-case risk given the data and graphical assumptions.

References

Isabela Albuquerque, João Monteiro, Tiago H Falk, and Ioannis Mitliagkas. Adversarial targetinvariant representation learning for domain generalization. arXiv preprint arXiv:1911.00804 , 2019.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Alexander Balke and Judea Pearl. Bounds on treatment effects from studies with imperfect compliance. Journal of the American Statistical Association , 92(439):1171-1176, 1997.

Elias Bareinboim, Sanghack Lee, Vasant Honavar, and Judea Pearl. Transportability from multiple environments with limited experiments. Advances in Neural Information Processing Systems , 26, 2013.

Elias Bareinboim and Judea Pearl. Transportability from multiple environments with limited experiments: Completeness results. Advances in neural information processing systems , 27, 2014.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79:151-175, 2010.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems , volume 19. MIT Press, 2006.

Donald T Campbell and Julian C Stanley. Experimental and quasi-experimental designs for research . Ravenio books, 2015.

J. Correa and E. Bareinboim. A calculus for stochastic interventions: Causal effect identification and surrogate experiments. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , New York, NY, 2020. AAAI Press.

J. Correa and E. Bareinboim. General transportability of soft interventions: Completeness results. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 10902-10912, Vancouver, Canada, Jun 2020. Curran Associates, Inc.

Juan D Correa and Elias Bareinboim. From statistical transportability to estimating the effect of stochastic interventions. In IJCAI , pages 1661-1667, 2019.

Shai Ben David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , pages 129-136. JMLR Workshop and Conference Proceedings, 2010.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research , 17(1):2096-2030, 2016.

Stuart S Glennan. Mechanisms and the nature of causation. Erkenntnis , 44(1):49-71, 1996.

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 , 2020.

Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning , 110(3):457-506, 2021.

Kasra Jalaldoust and Elias Bareinboim. Transportable representations for domain generalization. Proceedings of the AAAI Conference on Artificial Intelligence , 38(11):12790-12800, Mar. 2024.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation (rex). In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 5815-5826. PMLR, 18-24 Jul 2021.

Sanghack Lee, Juan D Correa, and Elias Bareinboim. Generalized transportability: Synthesis of experiments from heterogeneous domains. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , 2020.

Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Peter Machamer, Lindley Darden, and Carl F Craver. Thinking about mechanisms. Philosophy of science , 67(1):1-25, 2000.

Sara Magliacane, Thijs Van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. Advances in neural information processing systems , 31, 2018.

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning , pages 10-18. PMLR, 2013.

Judea Pearl. Causality . Cambridge university press, 2009.

Judea Pearl and Elias Bareinboim. Transportability of causal and statistical relations: A formal approach. In Twenty-fifth AAAI conference on artificial intelligence , 2011.

Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 78(5):947-1012, 2016.

Md Musfiqur Rahman and Murat Kocaoglu. Modular learning of deep causal generative models for high-dimensional causal inference. In Forty-first International Conference on Machine Learning , 2024.

Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. The Journal of Machine Learning Research , 19(1):1309-1342, 2018.

Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk minimization. In International Conference on Learning Representations , 2021.

Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 83(2):215-246, 2021.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Xinwei Shen, Peter Bühlmann, and Armeen Taeb. Causality-oriented robustness: exploiting general additive interventions. arXiv preprint arXiv:2307.10299 , 2023.

Amos Storkey. When training and test sets are different: characterizing learning transfer. 2008.

Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8(5), 2007.

Jin Tian and Judea Pearl. A general identification condition for causal effects . eScholarship, University of California, 2002.

US Department of Health and Human Services. The health consequences of smoking-50 years of progress: a report of the surgeon general, 2014.

Vladimir Vapnik. Principles of risk minimization for learning theory. Advances in neural information processing systems , 4, 1991.

Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering , 2022.

Kevin Xia, Kai-Zhan Lee, Yoshua Bengio, and Elias Bareinboim. The causal-neural connection: Expressiveness, learnability, and inference. Advances in Neural Information Processing Systems , 34:10823-10836, 2021.

Junzhe Zhang, Jin Tian, and Elias Bareinboim. Partial counterfactual identification from observational and experimental data. arXiv preprint arXiv:2110.05690 , 2021.

Appendix

Table of Contents

A Partial Transportability as a Bayesian Inference Task

Consider a system of multiple SCMs M 1 , M 2 , . . . , M K , M ˚ that induces the selection diagram G ∆ , and entails the source distributions P 1 , P 2 , . . . , P K , and the target distribution P ˚ over the variables V . Let ψ : Ω X Ñ R be a functional of interest. Consider the following optimization scheme:

where each N i is a canonical model characterized by a joint distribution over t R V u V P V .

This section describes an Markov Chain Monte Carlo (MCMC) algorithm to approximate the optimal scalar ˆ q max upper bounding the query ϕ N ˚ : ' E P N ˚ r ψ p X qs above from finite samples drawn from input distributions P 1 , P 2 , . . . , P K . Formally, we aim to infer the value,

where ¯ v : ' p ¯ v P 1 , . . . , ¯ v P k q , ¯ v P i ' t v p j q P i : j ' 1 , . . . , n i u denote n i independent sampled drawn from P i .

We consider a setting in which we are provided with prior distributions (possibly uninformative) over parameters of the family of compatible CMs N 1 , N 2 , . . . , N ˚ . In particular, we assume that for each CM, probabilities of P p U q , U P U are drawn from uninformative Dirichlet priors; and F are drawn uniformly from the finite class of possible structural functions. That is, for every U P U and every V P V ,

The total collection of parameters is given by the set tp θ N 1 , ξ N 1 q , . . . , p θ N ˚ , ξ N ˚ qu . Among them θ ' t θ U P r 0 , 1 s d U : U P U u define the parameterization of exogenous probabilities while ξ ' t ξ p pa V , u V q V P supp V : PA V Ă V , U V Ă U u define the structural functions, one set of each CMseparately.

We design a Gibbs sampler to evaluate posterior distributions over these parameters. For simplicity, we describe each step of the gibbs sampler for a single domain and input dataset, and consider the implementation of constraints below.

A.1 Gibbs Sampling

The Gibbs sampler iterates over the following steps, each parameter conditioned on the current values of the remaining terms in the parameter vector.

Sample u . Let u P Ω U , U P U . For each observed data example across all domains v p n q P ¯ v , n ' 1 , . . . , ř i n i , we sample corresponding exogeneous variables U P U from the conditional distribution,

Sample ξ . Parameters ξ define deterministic causal mechanisms. For a given parameter ξ p pa V , u V q V P ξ its conditional distribution is given by P p ξ p pa V , u V q V ' v | ¯ v , ¯ u q ' 1 if there exists a sample p v p n q , pa p n q V , u p n q q for some n , where n iterates over the samples of u from step 1 and v associated with the subset of domains in which exogeneous probabilities match the target domain, such that ξ p pa p n q V , u p n q V q V ' v p n q . Otherwise, P p ξ p pa V , u V q V ' v | ¯ v , ¯ u q is given by a uniform discrete distribution over its support supp V .

Sample θ . Let θ U ' p θ 1 , . . . , θ d U q P θ be the parameters that define the probability vector of possible values of variables U P U C . Its conditional distribution is given by,

where β i : ' ř n ✶ t u p n q ' u i u . Similarly, n iterates over the samples of u from step 1 associated with the subset of domains in which exogeneous probabilities match the target domain.

A.2 Implementing Constraints

Iterating this procedure forms a Markov chain with the invariant distribution P p u , ξ , θ | ¯ v q . This naturally enforces the soft constraint P N i p v q ' P i p v q , i P t 1 , 2 , . . . , K, ˚u for the CMs defined by the sampled parameters. The posterior distributions of the subset of p θ N ˚ , ξ N ˚ q for which invariances across domains are assumed are then matched with the posterior distribution inferred from source data. The constraint P N i p r V q ' P N ˚ p r V q , i P t 1 , 2 , . . . , K, ˚u , V R ∆ i, ˚ is enforced by generating θ N ˚ U from the prior such that P N ˚ p r V q : ' ř u P Ω θ N ˚ u ' ř u P Ω θ N i u : ' P N i p r V q , V R ∆ i, ˚ where Ω denotes the partition of supp U that is expressed by R V .

The query is then approximated by plugging the T MCMC samples into the query ϕ N ˚ to obtain ϕ p 1 q N ˚ , . . . , ϕ p T q N ˚ and for a chosen value of confidence value α .

Example 5 (Example 3 continued) . Consider again the evaluation of the risk R P ˚ p h q : ' P N ˚ p Y ‰ h p X qq given the classifier h p x q ' ␣ x . We are data sampled from P 1 p x, y q , P 2 p x, y q . For every SCM M , there exists an SCM of the described format specified with only a distribution P p r X , r Y q , where,

Thus, the joint distribution P p u XY q ' P p r X , r Y q can be parameterized by a vector in 8-dimensional simplex. The canonical SCMs associated with each of the SCMs M 1 , M 2 , M ˚ , are denoted N 1 , N 2 , N ˚ , for which V ' t X,Y u , U ' t U XY u and supp U XY ' t 1 , . . . , 8 u . The partial task can be translated into an optimization problem aiming to to find the upper-bound for the risk R P ˚ p h q for the classifier h p x q ' ␣ x :

With the Gibbs sampler outlined above, we obtain samples from the posterior distribution P p θ N 1 , θ N 2 , ξ N 1 , ξ N 2 | ¯ v q . θ N 1 , θ N 2 encode the probabilities P N 1 p U XY ' u q , P N 2 p U XY ' u q and are instantiated as two-dimensional arrays of shape p 2 , 4 q such that, e.g., P N 1 p r Y q ' ř dim. 1 θ N 1 , with r Y P t 1 , 2 , 3 , 4 u and similarly P N 1 p r X q ' ř dim. 0 θ N 1 , with r X P t 1 , 2 u .

To enforce the constraints P N 1 p r Y q ' P N ˚ p r Y q , P N 2 p r X q ' P N ˚ p r X q it thus suffices to sample θ N ˚ from the prior Dirichlet distribution (as it has not been updated with data) and re-scale the outcomes such that the partial row and column sums satisfy the corresponding partial row and column sums computed from the MCMC samples of P p θ N 1 , θ N 2 | ¯ v q . The resulting MCMC parameters p θ N ˚ , ξ N ˚ q are then valid samples from the posterior distribution P p θ N ˚ , ξ N ˚ | ¯ v q subject to assumed constraints, and the risk could be computed by plugging those samples into R P ˚ p h q : ' P N ˚ p Y ‰ h p X qq to obtain R P ˚ p h q p 1 q , . . . , R P ˚ p h q p T q and evaluating for a chosen value of confidence value α .

˝

h2(c)

1.0

Rp*

1.0

Rp*

1.0

Rp*

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

I

工

工

T

0.0

0.0

0.0

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

The following Theorem shows that ˆ q max converges to the true (tight) bounds q max for the unknown query R P ˚ p h q .

Theorem 4. ˆ q max defined in Eq. (19) is a valid upper bound on q max for any sample size, and coincides with q max as the sample size increases to infinity.

Proof. Let Θ denote the collection of parameters ξ , θ of discrete SCMs that generate the observed data from P 1 , P 2 , . . . . We assume that the prior distribution on ξ , θ has positive support over the domain of Θ . That is, the probability density function ρ p ξ q ą 0 and ρ p θ q ą 0 for every possible realization of ξ , θ . By the definition of Θ , for every pair of parameter p ξ , θ q P Θ , it must be compatible with the dataset ¯ v , i.e., P p ¯ v | ξ , θ q ą 0 . Similarly, given that the prior has positive support in Θ , P p ξ , θ | ¯ v q ą 0 .

Note that parameters p ξ , θ q P Θ fully determine the optimal upper bound q max for R P ˚ p h q . And so this implies that P p R P ˚ p h q ă q max | ¯ v q ą 0 , which by definition of a 100% credible interval means that R P ˚ p h q ă ˆ q max .

Next we show convergence of the posterior by way of convergence of the likelihood of the data given one SCM M . For increasing sample size the posterior will, with increasing probability, be low for any parameter configuration, i.e. for any p ξ , θ q R Θ . By the definition of the optimal upper bound q max given by the solution to the partial identification task,

Therefore if the prior on parameters p ξ , θ q defining SCMs is non-zero for any M compatible with the data and assumptions, also the posterior converges,

which is the definition of the credible value ˆ q max as the 100 th quantile of the posterior distribution, which coincides with q max asymptotically.

B Additional Experiments and Details

This section includes experimental details not covered in the main body of this paper as well as additional examples to illustrate our methods, including the Bayesian inference approach.

For the approximation of credible intervals and expectations required for the Bayesian inference approach, we draw 10,000 samples from posterior distributions P p¨ | ¯ v q after discarding 2 , 000 samples as burn-in. The results will be given a violin plots that encode the full posterior distribution of the query of interest, here the target error R P ˚ p h q of a classifier h . The worst-case target error can then be read as the upper end-point of the posterior distribution.

For completeness, we provide MCMC results for Examples 2 and 3, analyzed in the main body of this paper, in Figs. 7 and 8, respectively. One could check that the upper bounds match with the analysis in the main body of this paper.

B.1 Additional Examples

This section adds additional synthetic examples to illustrate our methods.

h(x)

h(x)

h(x)

h4(x)

1.0

Rp*

TI

1.0

Rp*

1.0

Rp*

I

I

1.0

Rp*

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

I

0.2

0.2

T

1

T

0.0

0.0

0.0

0.0

n=100

n=500

n=1000n=2000

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

(a) Example 6

(b) Example 7

h(s,t,w)=h(t,w)h(w)

1.0

Rp*

Rp1

0.8

0.6

0.4

0.2

0.0

0

25

50

75

100

Trainingepochs

h(x,x2,w)=h(w)h(z)

1.0

Rp*

Rpi

0.8

0.6

0.4

0.2

0.0

0

25

50

75

100

Trainingepochs

Example 6. This experiment is inspired by the debate around the relationship between smoking and lung cancer in the 1950's [37], and the corresponding selection diagram is shown in Figure 12a. We consider M : t M 1 , M ˚ u that describe the effect of an individual's smoking status S on lung cancer C , including related measured variables such presence of tar in the lungs T , and demographic factors W . The data generating mechanism is given by

Note that ∆ ' t S u as the mechanism for S differs across domains while the mechanisms for all other variables are assumed invariant. The quantity to upper-bound is the target mean squared error: R P ˚ p h q : ' E P ˚ rp C ´ h q 2 s of cancer prediction algorithms h P t h 1 p w,s, t q ' E P 1 r C | w,s, t s , h 2 p w,t q ' E P 1 r C | w,t s , h 3 p w q ' E P 1 r C | w su given data from P 1 and G ∆ .

The results for the NCM approach are given in Fig. 9a. We observe that despite the discrepancy in S , all methods maintain an error of close to 0.4.

The results for the Gibbs sampling approach are given in Fig. 10. The violin plots encode the full posterior distribution of the query of interest, here the target error R P ˚ p h q of a classifier h . The worst-case target error can then be read as the upper end-point of the posterior distribution. We observe that the upper-bounds from the NCM and MCMC approach approximately match. ˝

Example 7. This experiment considers the design of prediction rules for the development of Alzheimer's disease in a target hospital M ˚ in which no data could be recorded, and the corresponding selection diagram is shown in Figure 12b. The observed variables are given by V ' t X 1 , X 2 , W, Y, Z u . Among those, X 1 and X 2 are treatments for hypertension and clinical depression, respectively, both known to influence Alzheimer's disease Y , and blood pressure W . Z is a symptom of Alzheimer's. Their biological mechanisms are somewhat understood, e.g. the

h(s,t,w)

h2(t, w)

hs(w)

1.0

1.0

1.0

Rp*

Rp*

Rp*

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

H

0.2

0.2

0.2

0.0

0.0

0.0

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

h(x1,x2,w)

h2(w)

h(z)

1.0

1.0

1.0

Rp*

Rp*

Rp*

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

I

I

I

0.2

0.2

0.2

0.0

0.0

0.0

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

n=100

n=500

n=1000

n=2000

effect of hypertension is mediated by blood pressure W , although several unobserved factors, such as physical activity levels and diet patterns, are expected to simultaneously affect both conditions. We assume that hypertension and clinical depression are not known to affect each other, although it's common for patients with clinical depression to simultaneously be at risk of hypertension (expressed through the presence of an unobserved common cause). More specifically, investigators have access to data from a related study conducted in domain M 1 . SCMs M : t M 1 , M ˚ u are given as follows,

Note that ∆ ' t W u as the mechanism for W differs across domains while the mechanisms for all other variables are assumed invariant. In this example, we aim at upper-bounding the target mean squared error: R P ˚ p h q : ' E P ˚ rp C ´ h q 2 s of cancer prediction algorithms h P t h 1 p x 1 , x 2 , w q ' E P 1 r Y | x 1 , x 2 , w s , h 2 p w q ' E P 1 r Y | w s , h 3 p z, t q ' E P 1 r Y | z su given data from P 1 and G ∆ ˚ 1 .

The results for the NCM approach are given in Fig. 9b. We observe that the discrepancy in W leads to poor performance for all methods (chance level) except for h 3 that outperforms.

The results for the Gibbs sampling approach are given in Fig. 11. The violin plots encode the full posterior distribution of the query of interest, here the target error R P ˚ p h q of a classifier h . The worst-case target error can then be read as the upper end-point of the posterior distribution. We observe that the upper-bounds from the NCM and MCMC approach approximately match. ˝

(a) Example 6

(b) Example 7

B.2 More on Colored MNIST

Consider handwritten grayscale digits W P r 0 , 1 s 28 ˆ 28 that are annotated with Y P t 0 , 1 , . . . , 9 u and colored with C P t red , green u , resulting in colored images Z P r 0 , 1 s 28 ˆ 28 ˆ 3 . What follows describes the underlying SCM for domain i P t 1 , 2 , ˚u :

In words, the grayscale image of handwritten digits W is generated according to a distribution P p w q shared across all domains. The label Y is the annotation of the image with the corresponding digit through mechanism f Y shared across all domains; the variable U Y accounts for the possible error in annotation. Next, the color is chosen based on the digit Y following some stochastic policy f i C p¨ , U C q that changes across the source and target domains. Finally, the colored image Z is produced by product of the grayscale image W and the color C ; exogenous variable U Z accounts for possible noise in coloring.

We have a classifier h : Ω Z Ñ Ω Y at hand, and the task is to assess its generalizability. Consider the following derivation:

Motivated by the above derivation, we use the source data drawn from P 1 , P 2 and train the generative models P p y ; η Y q , P p z | y, c ; η Z q to approximate sampling from the distributions P 1 , 2 p y q , P 1 , 2 p z | y, c q , respectively. The former generates a random digit Y according to the distribution of label in the source domain, and the latter generates a colored picture Z by taking color C and digit Y as the input. Also, we use an NCM with parameter θ ˚ C to model the c-factor P ˚ p c | do p y qq ' P ˚ p c | y q . We can now rewrite the risk as follows:

By maximizing the above w.r.t. the free parameter θ ˚ C , we achieve the worst-case risk of the classifier.

B.3 Reproducibility

For the synthetic experiments, we used feed-forward neural networks with 7 layers and 128 ˆ 128 neurons in each layer. The activation for all layers is ReLu, but for the last layer which is a sigmoid since f θ V outputs the probability of V ' 1 . For evaluation, at each epoch, we used 1000 samples from the joint distribution. The data generative process for all experiments is provided in the corresponding

example. We used Adam optimizer for training the Neural networks. In CMNIST example, we used a standard implementation of a conditional GAN [23] trained over 200 epochs with a batch-size of 64. The learning rate of Adam was set to 0 . 0002 . The architecture of the generator is given by a 5 layer feed-forward neural network with Batch normalization and Leaky-ReLu activations.

C Extended Discussion on Algorithms

In this section, we elaborate more on the algorithms presented in the paper.

C.1 Examples of Neural-TR (Algorithm 1)

In the next examples, we follow Algorithm 1 to compute the worst-case risk of a classifier.

Example 8 (Simplify) . Consider a system of SCMs M 1 , M 2 M ˚ over X ' t C, Z, W u and Y that induces the selection diagram shown in Figure 13. Suppose we would like to assess the risk of a classifier h p z q . Following Theorem 2, the naive approach requires us to parameterize three NCMs θ 1 , θ 2 , θ ˚ over the variables X , Y , and then proceed with the maximization of the target quantity R P N ˚ p h q ' E P ˚ r ✶ t Y ' h p Z qus . Notably, the latter depends only on P ˚ p y, z q . We can rewrite the risk of h as follows:

This new expression for the objective function depends only on the unknown P ˚ p y, c q , a so-called ancestral c-factor, that can generally be expressed as P ˚ p a | do p pa A qq , A ' t C, Y u . In the following, we argue that to partially transport the risk we only need to parameterize the SCMs over ancestral c-factors that are not transportable. Specifically, the partial transportation problem can be restated as follows:

In the above, D i ' P i p c, y, z, w q denotes the source data, and P p z | y ; η 2 q is a probabilistic model of P 2 p z | y q learned using the data D 2 . ˝

Example 9 (Partial-TR illustrated) . Consider a system of SCMs M 1 , M 2 , M ˚ over the binary variables X ' t X 1 , X 2 , . . . , X 9 u and Y that induces the selection diagram shown in Figure 14. Consider the classifier h p x 1 , x 4 q ' x 1 _ x 4 . The objective is partial transportation of the risk of h , expressed as follows:

X8

X

X4

X3

X9

X6

X5

X2

X1

S$1

S2

The latter indicates that ψ p X 1 , X 4 , Y q : ' ✶ t Y ‰ X 1 _ X 4 u must be passed to the algorithm. The objective function is then expressed as:

Next, we focus on transporting P ˚ p x 1 , x 4 , y q . First, we compute the ancestral set using the selection diagram;

and we decompose this set into c-components:

Next, we form the expression below:

Notice,

Thus, we use the source data D 1 , D 2 to learn the generative model P p a 2 | x 6 ; η 1 A 2 q , P p a 3 ; η 2 A 3 q to approximate sampling from P 1 p a 2 | x 6 q , P 2 p a 3 q respectively. We plug these models as constants into Eq. 35.

Since S ˚ 1 , S ˚ 2 are pointing to the variables X 2 , X 1 , respectively, the first term P ˚ p x 1 , x 2 q | do p y qq in Eq. 38 can not be directly transported from neither of the source domains. Thus, we need to parameterize this c-factor using NCMs across all domains. We require the following properties:

Parameter sharing : Since X 4 , Y, X 5 , X 7 , X 8 , X 9 are not pointed by S 1 , we share their mechanisms across all domains. Also, since X 2 , X 6 are not pointed by S 2 , we set θ ˚ t X 2 ,X 6 u ' θ 2 t X 2 ,X 6 u . These constraints are stored in C expert in the Algorithm.

Source data : To enforce θ 1 , θ 2 to be compatible with the source data D 1 , D 2 , we compute the likelihood of the data w.r.t. the parameters, as follows:

We plug P p x 1 , x 2 | do p y q ; θ ˚ A 1 q into Eq. 38. Finally, we use stochastic gradient ascent to maximize the objective function in Eq. 35 regularized by an additive term Λ ¨ L likelihood that encourages the likelihood of the data w.r.t. the parameters of the source NCMs. ˝

C.2 Illustration of CRO (Algorithm 2)

First, we initialize with a random classifier. One may also warm start with a reasonable guess such as empirical risk minimizer defined as,

Throughout the runtime of the algorithm we accumulate instances of distributions that we obtain via Neural-TR (Alg. 1). At each step, these distribution are aimed to maximize the risk of the classifier at hand. In this sense, Neural-TR can be viewed as an adversary, and the CRO can be viewed as a game between two players:

Neural-TR. Searches over the spaces of plausible target domains that are characterized by the source data and the domain relatedness encoded in the selection diagram, to find a distribution that is hard to generalize to using the classifier at hand.

group DRO [32] Updates the classifier at hand by minimizing the maximum risk over the distributions produced by Neural-TR so far, that is,

For more information about group DRO, see Appendix D.2.

The equilibrium of the above happens if the worst-case risk obtained by Neural-TR almost coincides with the risk obtained by group DRO, i.e.,

Once this is achieved, we stop the search and return the classifier at hand. When the game is not at equilibrium, we would have a difference larger than δ , meaning that the new target domain ˆ θ ˚ has enough novelty to forces the classifier at hand to perform at least δ worse than what it achieves over the existing distributions in D ˚ . Therefore, we draw samples D ˚ ' P p x , y ; ˆ θ q and add them to our collection D ˚ . As shown in Theorem 3 this game reaches the equilibrium in finitely many steps, and the classifier that we return has the best worst-case risk w.r.t. the selection diagram G ∆ and the source distributions P . The conceptual Figure 15 shows the process of convergence of CRO.

It is important to note that although we employ group DRO as a subroutine in our CRO algorithm, we do not use the source distributions directly. Instead, we use group DRO on the distributions obtained from Neural-TR. Note that under the assumptions encoded in the selection diagram, the target distribution distribution may be geometrically unrelated to the source distributions; the reason is that mechanistic relatedness of the target domain to the source domains (as indicated by the graph) do not translate directly to closeness of the entailed distributions under known distributional distance measures.

D Extended Related Work

In this section, we discuss some learning schemes based on invariant and robust learning that are proposed for domain generalization, including IRM and group DRO that are discussed in the experiments.

D.1 Invariant Learning for Domain Generalization

Several common invariance criteria are extensively studied in the literature and proposed for the domain generalization task. A prominent idea is label conditional distribution invariance that seeks a representation ϕ such that P i p Y | ϕ p X qq is equal across the source domains [29, 1, 13, 22]. These notions do not explicitly rely on an underlying structural causal model (SCM), although invariances are often justified by an underlying causal model [27, 2, 39, 31, 33]. Jalaldoust & Bareinboim

(a) Iteration 1

(b) Iteration 2

(c) Iteration 3

(d) Equilibrium

1

[17] studied the implicit assumptions that license generalizability of representations that satisfy the probabilistic relation P i p Y | ϕ p X qq . Although searching for such representation is practically challenging and in cases theoretically intractable. Thus, one may resort to achieving an approximate notion that serve as a proxy to invariance of P i p Y | ϕ p X qq ; A well-known instance of such effort is invariant risk minimization [2], discussed below.

The paper [2] studies a constrained optimization problem called invariant risk minimization (IRM) in the context of domain generalization. In the notation of our paper, the IRM problem can be written as follows:

Where ϕ : Ω X Ñ Ω R is a representation, and h : Ω R Ñt 0 , 1 u is a classifier defined based on it. In words, a pair h, ϕ satisfies the invariant risk minimization property if h ˝ ϕ attains the minimum risk across all classifiers defined based on ϕ , across all source domains. The search procedure suggests choosing the classifier that satisfies the mentioned constraint, and achieves minimum risk on the pooled source data. The constrained optimization program above is highly non-convex and hard to solve in practice. To approximate the solution, the paper considers the Langrangian form below:

In this program, θ parametrizes the classifier h , and the penalty term λ accounts for how restrictive one wants to enforce the IRM constraint. In the extreme λ ' 0 the objective equates to the vanilla ERM with all data pooled; on the other extreme, for λ Ñ8 ascertains that the solution is guaranteed to satisfy the IRM constraint.

Consider a representation that satisfies the original IRM constraint in Eq. 45. The optimal classifier defined over this representation is the bayes classifier, that uses 1 2 level set of P i p Y ' 1 | ϕ p X qq as the decision boundary. This means that satisfying the IRM constraint implies a match between 1 2 level-sets of P i p Y ' 1 | ϕ p X qq across all source domains. On the other hand, invariance of

P i p Y | ϕ p X qq requires coincidence of every level-set across the source domains, and in this sense, the IRM constraint can be viewed as a proxy to the invariance property of P i p Y | ϕ p X qq . One can speculate that since IRM yields a proxy to invariance of P i p Y | ϕ p X qq , it might still exhibit generalization, though slightly weaker than what is derived from invariance of P i p Y | ϕ p X qq . However, IRM is shown to have poor domain generalizability, both theoretically (e.g., [30]) and empirically (e.g., [15]). Still, due to popularity of this method in the literature, we find it insightful to use the Neural-TR algorithm to find out what would be the worst-case risk of IRM. As shown in Fig. 4c, the worst-case performance of IRM is much worse than what is reported by [2] and [15]; the reason is that Neural-TR does not commit to one held-out domain, and instead it constructs an SCMs that is tailored to yield the poorest performance subject to the graph and source distributions.

D.2 Group Robustness for Domain Generalization

Group Distributionally robust optimization (group DRO) [32] has been employed in the broad context of learning under uncertainty. In group DRO one seeks a single classifier that minimizes the risk on multiple distributions simultaneously. More specifically, the objective is minimizing the maximum risk among the source distributions, i.e.,

This approach ensures that the learned classifier is optimal w.r.t. an unknown target domain that lies in the convex hull of the source distributions. In this sense, group DRO objective interpolates the perturbations that are represented in the source data to define an uncertainty set for the target distribution. On the other hand, in invariant learning the objective is to extrapolate the perturbations that are observed among the source domain by learning a representation that shields the label from these changes. In particular, [18, 31, 33] highlight the invariant-robust spectrum, and propose methods that have a free parameter which allows interpolating the two. In our experiments, we considered group DRO as a representative of methods in this category, and evaluated its worst-case performance in the Colored MNIST task, as shown in Figure 4c. Once again, we emphasize that this worst-case risk is much larger than what is shown in the benchmarks, e.g., by [15]. The reason is that the worst-case performance is obtained by Neural-TR that operates as an adversary, seeking a plausible target domain that is hardest to generalize to, subject to the assumptions encoded in the graph and the source data.

E Proofs

Proof of Theorem 1

Our results rely on the expressiveness of discrete SCMs, i.e. defined over variables t V , U u with finite cardinalities. Discrete SCMs, introduced first in [3] and then in [42] have been shown to be 'canonical' in the sense that they could represent all counterfactual distributions entailed by any SCM with the same induced causal diagram defined over finite V . The following example illustrates this observation.

Example 10 (The double bow) . Let t X,Y,Z u be binary variables. Consider two source domains defined based on the following SCMs:

$

$

✶ ✶ The SCM M 1 induces a counterfactual probabilities, e.g. P M 1 p x, y x , z y q for outcomes x, y x , z y P t 0 , 1 u . [3] observed that such probabilities, defined over a finite set of events, may be generated with an equivalent model with a potentially large but finite set of discrete exogenous variables. [3] derived a canonical parameterization for the SCMs that induces the same graph but instead involves possibly correlated discrete latent variables R X , R Y , R Z , where R X determines the functional that decides

X , R Y determines the functional that decides Y based on X , and R Z determines the functional that decides Z based on Y . [3] showed that for every SCM M with the same induced graph as M 1 there exists an SCM of the described format specified with only a distribution P p r X , r Y , r Z q , where,

Thus, the joint distribution P p r X , r Y , r Z q can be parameterized by an 32-dimensional vector. ˝

This example illustrates a more general procedure, in which probabilities induced by an SCM over discrete endogenous variables V may be generated by a canonical model. This is formalized in the following lemma.

Definition 7 (Canonical SCM) . A canonical SCM is an SCM N ' x U , V , F , P p U qy defined as follows. The set of endogenous variables V is discrete. The set of exogenous variables U ' t R V : V P V u , where supp R V ' t 1 , . . . , m V u (where m V ' |t h V : supp pa V Ñ supp V u| ) for each V P V . For each V P V , f V P F is defined as f V p pa V , r V q ' h p r V q V p pa V q .

The following lemma establishes the expressiveness of canonical SCMs.

Lemma 1 (Thm. 2.4 [42]) . For an arbitrary SCM M ' x U , V , F , P p U qy , there exists a canonical SCM N such that 1. M and N are associated with the same causal diagram, i.e., G M ' G N . 2. For any set of counterfactual variables Y x , . . . , Z w , P M p Y x , . . . , Z w q ' P N p Y x , . . . , Z w q .

In words, finite exogenous domains in canonical SCMs are sufficient for capturing all the uncertainties and randomness introduced by the (potentially) continuous latent variables in SCMs. Our goal will be to adapt the canonical parameterization of SCMs such that they entail the equality constraints specified by G ∆ . The next example illustrates the implication of the constraints induced by G ∆ on the construction of canonical SCMs.

Example 11 (Example 10 continued.) . Consider M 1 and M ˚ given in Example 10. The domain discrepancy set ∆ indicates that certain causal mechanisms need to match across pairs of the SCMs. For example, ∆ 1 ˚ ' t Y u , which does not contain t X,Z u , and this implies that the functions f X , f Z are invariant across M 1 , M ˚ , and that the distribution of unobserved variables that are arguments of f Y , f Z , namely, t U X , U XY , U Y Z u are invariant across M 1 , M ˚ . The canonical parameterization of M 1 is given by

Analogously, the canonical parameterization of M ˚ is given by

With these definitions, the restrictions in ∆ 1 ˚ impose straightforward constraints on the parameterization of the canonical models given directly from the definition of discrepancy set:

for any input x, y, r Y , r X , r Z .

˝

The next lemma formalizes the observation made in the example above, showing that if a pair SCMs and a pair of associated canonical models induce the same distributions and causal diagram, their discrepancies must also agree.

Lemma 2. For a pair of SCMs M i , M j ( i, j P t˚ , 1 , 2 , . . . , T u ) defined over V with discrepancy set ∆ ij Ď V , let N i , N j be associated canonical SCMs that induce the same causal graphs and entail the same distributions over V . Then the discrepancy sets of the pairs of SCMs and canonical SCMs must agree, i.e. V P ∆ ij if and only if either f N i V ‰ f N j V , or P N i p u V q ‰ P N j p u V q .

Proof. Let V P ∆ ij , and fix M i , M j such that P M i p v | do p pa V qq ‰ P M j p v | do p pa V qq . This is possible since the interventional probabilities are parameterized by the mechanism of V which could vary across M i , M j . Assume for a contradiction that f N i V ' f N j V and P N i p u V q ' P N j p u V q for two canonical models N i , N j constructed to match all L 3 statements induced by M i , M j . This implies in particular that P N i p v | do p pa V qq ' P N j p v | do p pa V qq and therefore N i , N j do not induce the same probabilities as M i , M j . This contradicts the assumption that the pair of canonical SCMs matches the pair of SCMs in all L 3 statements.

For the converse, we proceed similarly. For fixed M i , M j , assume for a contradiction that f N i V ‰ f N j V , or P N i p u V q ‰ P N j p u V q such that P N i p v | do p pa V qq ‰ P N j p v | do p pa V qq for two canonical models N i , N j constructed to match all L 3 statements induced by M i , M j , but nevertheless V R ∆ ij . The discrepancy set ensures that P M i p v | do p pa V qq ' P M i p v | do p pa V qq but the same relation is not true for N i , N j as P N i p v | do p pa V qq ‰ P N j p v | do p pa V qq by assumption and therefore N i , N j do not induce the same probabilities as M i , M j . This contradicts the assumption that the pair of canonical SCMs matches the pair of SCMs in all L 3 statements.

Lemma 3. Consider a system of multiple SCMs M : t M 1 , M 2 , . . . , M K , M ˚ u that induces a selection diagram and entails the source distributions P : t P 1 , P 2 , . . . , P K , P ˚ u over the variables V . Then there exists a system of canonical SCM N : t N 1 , N 2 , . . . , N K , N ˚ u such that

1. M and N are associated with the same set of causal diagrams and selection diagrams.

For any set of counterfactual variables Y x , . . . , Z w , P M ˚ p Y x , . . . , Z w q ' P N ˚ p Y x , . . . , Z w q .

Proof. For (1), Thm. 2.4 [42] gives that SCMs M : t M 1 , M 2 , . . . , M K , M ˚ u and canonical SCMs N : t N 1 , N 2 , . . . , N K , N ˚ u induce the same causal diagrams. Lem. 2 gives that for every pair of SCMs M i , M j ( i, j P t˚ , 1 , 2 , . . . , T u ), their discrepancy set is the same as that of N i , N j ( i, j P t˚ , 1 , 2 , . . . , T u ). As selection diagrams are constructed deterministically from causal diagrams and discrepancy sets, M and N must share the same set of selection diagrams.

Theorem 1 (restated). Consider a system of multiple SCMs M 1 , M 2 , . . . , M K , M ˚ that induces the selection diagram G ∆ and entails the source distributions P 1 , P 2 , . . . , P K and the target distribution P ˚ over the variables V . Let ψ p P ˚ q P r 0 , 1 s be the target quantity. Consider the following optimization scheme:

where each N i is a canonical model characterized by a joint distribution over t R V u V P V . The value of the above optimization, namely ˆ q max , is a tight upper-bound for the quantity ψ p P ˚ q among all tuples of SCMs that induce the selection diagram and entail the source distributions at hand. ˝

Proof. Note that,

is a tight upper bound to the target ψ p P M ˚ q among all tuples of SCMs that induce the selection diagram and entail the source distributions at hand, by construction. It follows from Lem. 3 that for any tuple of SCMs t M 1 , M 2 , . . . , M K , M ˚ u , that induce the selection diagram and entail the source distributions, there exists a tuple of canonical SCMs N 1 , N 2 , . . . , N ˚ , that induce the selection diagram and entail the source distributions such that,

The reverse direction of the above equations also holds since a a family of canonical SCMs is an instance of a family of SCMs. This means that solutions for optimization problems in Eq. (48) and Eq. (49) must coincide.

E.1 Proof of Theorem 2

To prove this result, we need to show the following:

Necessity. Every tuple of NCMs Θ that are constraint by conditions in Eq. 8 represents a tuple of SCMs that entails P and induces G ∆ .

Sufficiency. For every tuple of SCMs M that entails P and induces G ∆ , there exists a tuple of NCMs Θ that admits the constraints in Eq. 8, and for every i P t˚ , 1 , 2 , . . . , K u , we have P p y x , z w ; θ i q ' P M i p y x , z w q , where y x , z w .

Necessity. Consider a tuple of NCMs Θ that are constraint by the conditions in Eq. 8.

G ∆ -consistency. Since these NCMs are constructed based on the common causal diagram G , they all induce G (Theorem 2 by Xia et al. [41]). Moreover, the parameter sharing constraint states that V R ∆ ij if and only if θ i V ' θ j V . This implies that the NCMs parameterized by Θ induce the same domain discrepancy sets as G ∆ . Thus, the selection diagram induced by the NCMs parameterized by Θ is exactly G ∆ .

P -expressivity. The data likelihood condition for source distribution P i p v q states the following:

For large enough samples size | D i | ' P i p v q , and enough model complexity in θ , Theorem 1 by Xia et al. [41] shows that there exists a G -constrained NCM θ that induces the distribution entailed by the true SCM M i . Thus, by imposing Eq. (50) we assure that P p v ; θ i q ' P i p v q . By imposing all data likelihood conditions, in the limit of sample size and model complexity, we ensure that the source NCMs induce the source distributions.

In conclusion, the tuple of NCMs are necessarily representing a plausible target domain since (1) they induce G ∆ and (2) they entail P .

Sufficiency. Consider a tuple of SCMs M ' x M 1 , M 2 , . . . , M K , M ˚ y that induce G ∆ and entail P . Theorem 1 by Xia et al. [41] shows that for every SCM M that induces G , there exists a G -constraint NCMparameterized by θ such that P M p v q ' P p v ; θ q (as a consequence of L3-consistency). The proof is constructive, and for every V P V the construction of the neural network θ V depends on (1) the function f V and (2) the distribution P M p u V q .

Consider two SCMs M i , M j ( i, j P t˚ , 1 , 2 , . . . , K u ) that induce domain discrepancy set ∆ ij . Follow the construction by Xia et al. [41] to obtain the corresponding NCMs parameterized by θ i , θ j . For every V R ∆ i,j , we have, θ i V ' θ j V since the construction depends on f i V ' f j V and P M i p u V q ' P M j p u V q . Thus, the domain discrepancy set induced by θ i , θ j matches with ∆ i,j induced by the

SCMs M i , M j . Therefore, By constructing the NCM θ i from M i ( i P t˚ , 1 , 2 , . . . , K u ), we are guaranteed to have a tuple of NCMs N that (1) induce G ∆ and (2) entails P .

Partial-TR via NCMs. Due to necessity and sufficiency above, we conclude that a tuples of NCMs satisfies the parameter sharing and data likelihood conditions stated in Eq. 8, if and only if there exists a tuple of SCMs M that induce G ∆ and entail P such that P p v ; θ i q ' P M i p v q for all i P t˚ , 1 , 2 , . . . , K u . Therefore, by solving the following optimization problem,

we achieve a tight upper-bound for the query E P ˚ r ψ p W qs w.r.t. G ∆ , P . ˝

E.2 Proof of Proposition 1

Consider the objective of Theorem 2;

No need to parameterize non-ancestors of W . Let T ' V z An G ˚ p W q . By applying Rule 3 of σ -calculus [9] we realize that,

The latter indicates that the parameters t θ ˚ T u T P T are irrelevant to the joint distribution P p w q , and therefore, can be dropped from the NCMs used for partial transportability of E P ˚ r ψ p W qs ' ř w P ˚ p w q ¨ ψ p w q .

Let A ' An G ˚ p W q . We drop the non-ancestors, and rewrite the objective as follows:

Next, we add the likelihood terms to the main objective regularized by a coefficient Λ to achieve a single-objective optimization.

For Λ Ñ 8 , the new optimization problem matches with that of Thm. 2. Now, we focus on the likelihood expression, and rewrite it following a causal order of G ˚ , namely, A 1 ă A 2 ă ¨ ¨ ¨ ă A N .

Let t A j u j m ' 1 be the c-components of G ˚ r A s , which is the graph induced by nodes A . We rewrite the above objective in terms of the c-factors:

From the last expression, we can observe that the NCM parameterization is modular w.r.t. the c-components, as Rahman et al. [28] also discusses. We rewrite the full optimization program again:

Let A j be a c-component that S i is not pointing to it in G ∆ , i.e., A j X ∆ i ' H . The latter means that the parameter sharing θ ˚ V ' θ i V is enforced for all V P A j ; we call these parameters θ i, ˚ A j . We notice that θ i, ˚ A j only appears through the term log P p a j | do p pa A j q ; θ A j q in the score function; once in the main objective as θ ˚ A j and once in the regularizer as θ i A j . For Λ Ñ8 , the regularizer enforces θ i, ˚ A j to satisfy the following criterion:

This criterion is in fact an interventional (L2) constraint [41] enforced on θ i, ˚ A j that requires θ i, ˚ A j to approximate P i p a j | do p pa A j qq using the observational data D i . Since P i p a j | do p pa A j qq is a complete c-factor, it is identifiable from P i p a j , pa A j q [36]. Therefore, by increasing the sample size | D i | Ñ 8 and the model complexity of θ i, ˚ A j , satisfying the criterion in Eq. 66 guarantees arbitrarily accurate approximation of the interventional quantities P i p a j | do p pa A j qq [41]. This implies that we can replace the terms involving the parameters θ i, ˚ A j with any consistent approximation

of P i p a j | do p pa A j qq as constants. To get the approximation, we are free to use any probabilistic model and architecture depending on the context; this includes the option to train the NCM parameters θ i, ˚ A j in the pre-training.

This adjustment gets us to the exact procedure pursued in Algorithm 1, thus proves consistency of it with what we would achieve via Theorem 2. ˝

E.3 Proof of Theorem 3

For this proof, it is useful to define the worst-case risk w.r.t. the selection diagram and the source distributions.

Definition 8 (Worst-case risk) . For selection diagram G ∆ and source distributions P , the worst-case risk of classifer h : Ω X Ñ Ω Y is denoted by R G ∆ , P p h q and defined as the solution of partial transportation task for the query E P ˚ r L p Y, h p X qqs , where L p y, ˆ y q is a loss function. Formally,

˝

Theorem 3 (restated). For discrete X , Y CRO terminates. Furthermore, for large enough data across all source domain, the worst-case risk of CRO is at most ϵ away from the worst-case optimal classifier w.r.t. selection diagram G ∆ and source data P . Formally,

where h CRO n : ' CRO p D n , G ∆ q , and D n ' x D 1 , D 2 , . . . , D K y is a collection of datasets that each contain at least n datapoints. ˝

Proof. Soundness of CRO relies on consistency of Neural-TR (Alg. 1 as a subroutine; we pick the data size large enough to satisfy this condition according to Theorem 2.

Termination. Let ˆ θ ˚ 1 , ˆ θ ˚ 2 , . . . be the sequence of target NCMs produced during the runtime of CRO, and let h 1 , h 2 , . . . be the sequence of classifiers obtained after each iteration. Let Π denote the space of all distributions over X , Y . For discrete X , Y , the space Π is a compact subspace of some Euclidean space. Thus, every sequence in Π has a convergent subsequence, especially the sequence t P p x , y ; ˆ θ ˚ m qu m Ă Π ; let t P l u l be this convergent subsequence. Every convergent subsequence is Cauchy, which means,

where d is an appropriate metric over the probability space. Choose τ small enough w.r.t. the convergence tolerance δ ą 0 to ensure,

The above is possible, since the mapping R P p h q is a bounded and continuous mapping on the space Π . Now, we are guaranteed to find an index l such that,

Notice that by definition, ˆ θ ˚ l ` 1 is obtained by Neural-TR (Alg. 1) to attain the worst-case risk of h l , i.e.,

Putting the last three equations together we have,

which invokes the termination.

Moreover,

Worst-case optimality. Suppose h CRO is returned by CRO, and let h ˚ be the true worst-case optimal classifier defined as,

Let D denote the collection of datasets collected by the algorithm before termination. We know that h CRO is robust to D ˚ , i.e.,

Moreover, every distribution in D ˚ is entailed by an NCM that represents a possible target domain. Therefore, the worst-case risk is at least as large as the worst-case empirical risk on the set of distribution D , i.e.,

Since that algorithm has terminated we have,

where δ ą 0 is the tolerance for the convergence condition in the algorithm. Putting all inequalities together, we have,

which indicates that the worst-case risk of h CRO is at most δ larger than the optimal worst-case risk.

F Broader Impact and Limitations

Our work investigates the design of algorithms and conditions under which knowledge acquired in one domain (e.g., particular setting, experimental condition, scenario) can be generalized to a different one that may be related, but is unlikely to be the same. As alluded to in this paper, under-identifiability issues and the difficulty of stating realistic assumptions that are conducive to extrapolation guarantees are pervasive throughout the data sciences. Our hope is that our analysis with a more surgical encoding of structural differences between domains that allow the empirical investigator to determine whether (and how) her/his understanding of the underlying system is sufficient to support the generalization of prediction algorithm is an important addition towards safe and reliable AI. This approach is not without limitations, however. We have shown that selection diagrams are sufficient to ensure consistent domain generalization (through bounds instead of point estimates) but arguably restrict the analysis to a narrow class of problems as graphs or super-structures need to be defined. This stands in contrast with representation learning methods that operate on higher-dimensional spaces, e.g. text, images, which are difficult to reason about in a causal framework. The trade-off is that guarantees for consistent extrapolation are difficult to define and that one-size-fits-all assumptions are difficult to justify in practice. Partial transportability may be understood as a complementary view-point on this problem, applicable in a different class of problems in which structural knowledge is available implying that non-trivial guarantees for extrapolation can be established. Pushing the boundaries of methods based on causal graphs to reach compelling real-world applications is arguably one the most important frontiers for the causal community as a whole. In this work, there is scope for improving posterior estimation and for introducing assumptions on the class of SCMs that are modelled, e.g. linear Gaussian models, etc., that could lead to efficient predictors in higher-dimensional spaces. Similarly, relaxations of selection diagrams, e.g. in the form of equivalence classes or partially-known graphs, could be developed for applications in domains where knowledge of graph structure is unrealistic.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please see the contribution bullets.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: See appendix F.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All proofs are provided in the appendix with a more detailed restatement of the theorems and needed assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [No]

Justification: The code will be accessible through git-hub after publication. We ensured that the results are reproducible; please see Appendix B.3 for some details on the used architecture.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See the corresponding examples for each experiment, and the reproducibility note in B.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: All the examples are small, so it is not applicable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were executed on a Macbook Pro M2 32 GB RAM.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We respect NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is theoretical and bears no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All related and used results are cited properly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The code is annotated and provided.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles to problem of transportability in domain generalization. Multiple datasets are provided together with graphical information stipulating which causal mechanisms are shared across domains and which mechanisms are going to remain invariant in the target domain. The goal is then to estimate the error committed by a given classifier in the target domain. More specifically, the authors propose a way to estimate the worst-case error of the classifier given the observed distributions as well as the graphical information. A first result shows how the problem can be simplified using canonical SCMs and a second result shows how Neural Causal Models (NCM) can be used to estimate this bound. Furthermore, an approach named Causal Robust Optimization (CRO) is proposed to learn a classifier that minimizes the worst-case risk. Experiments experiments with the NCM approach to estimate the upper-bound of various classifiers, but does not tackle the CRO approach to learn a robust optimizer.\n\nSTRENGTHS:\n- The paper reads nicely (even for a fairly technical paper) and gives a good background on SCMs and graph terminology for the non-expert. The choice of notation is good and remains consistent throughout the paper.\n- The problem appears to be very important, and the approach to tackle it seems principled.\n- The approach appears to be novel and interesting. However, I don't have a good knowledge of the literature on transportability and domain generalization and thus my judgment's credibility is limited.\n- Many examples are provided to make the discussion more concrete, I enjoyed that.\n\nWEAKNESSES:\n- Some theoretical results, in their current form, seems unlikely to be true. Specifically in Theorem 2, I was surprised to see that this result does not mention anything about the expressive power of the NCM, what if the neural networks do not have enough capacity to express the causal mechanisms? Clearly what you would end up with wouldn’t be an upper bound right? I was also surprised to see that the optimization problem is formulated in a finite-sample fashion, but the conclusion concerns the actual expectation (i.e. not finite-sample), this sounds unlikely, no? Don’t you need to take a limit as the dataset grows for this bound to hold?\n\n- Confusion surrounding canonical SCM: At line 181, it is mentioned that [41] showed that every SCM M can be written in a canonical form (with discrete exogenous variables)... However, after quickly skimming through the paper, I could not find something resembling that statement. What am I missing? Overall I thought the discussion on canonical SCM was confusing. I’m left unsure as to which type of SCM can be cast to a canonical form (definition 5) and which paper showed this fact.\n\n- Some section were hard to parse. Mainly lines 270 to 289, including Algorithm 1. I'm not sure I got the point. I was following up to that point. I give some minor suggestions to improve clarity below.\n\n- Algorithm 2 (Causal Robust Optimization) is interesting, but it looks quite costly. Do the authors believe it could be made applicable? I was a bit disappointed to see it was not implemented in the experimental section. It would be nice to have experiments for CRO. Could this approach be applied to the Coloured MNIST dataset?\n\n- The background is very well written, but the end of the paper felt a bit less polished. Since these are the most important part, I believe they deserve more space. For instance, I had a hard time understanding the experiments with colored MNIST. \n\nMinor:\n- Example 1: Why using arrows for assignment? The symbol “:=” was used earlier…\n- Definition 3: Should S_{ij} be in fact S_i here? (line 125)\n- Example 2: Long sequences of “if”s would benefit from some punctuation. The “circled plus sign” is an XOR right?\n- Definition 4: Might be helpful to spell out \\mathbb{M}_0 completely in (4), since \\mathcal{M}^*_0 appears in the expectation but is not defined (the reader has to deduce that this is the target SCM belonging to  \\mathbb{M}_0) \n- Definition 5: might be useful to end with “where h_V^{(r_V)} is a function from supp_{pa_V} to supp_V”. I’m actually not certain the very last sentence adds something to the definition, since it is already obvious that f_V(pa_V, r_V) is a function of pa_V for all r_V… I’m also a bit confused by this sentence: “The set of endogenous variables V is discrete.” What does it mean for a set to be discrete? Do you mean the endogenous variables are discrete? Also, I’m a bit confused by the way the word “support” is used here. Usually it means the set of values for which the random variable has positive probability (at least for discrete variables that’s the definition), but here I believe this is not how “support” is used, correct?\n- Theorem 1: I believe the first constraint should not include the case i = *, no? I thought P* was not observed. In fact, you don’t have the case i=* on line 248 and in Theorem 2\n- Figure 1 is unclear, what does the color mean?\n- Line 353: Bayesian inference procedure?\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nI didn't find any discussion of the limitations of the approach in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper extends the formulation of canonical models to encode the constraints for the transportability tasks. Then it adapts Neural Causal Models for the transportability task and introduces an iterative method Causal Robust Optimization to find a predictor with the best worst-case risk.

STRENGTHS:
- The paper presents a unified framework for addressing transportability problems using canonical models and neural causal models. These transportability problems are broad and highly relevant in the field of machine learning.
- I appreciate the examples illustrated throughout the main text, as they greatly help the reader understand the definitions and intuitions.
- The theorems are solid and the experiments align well with the theories.

WEAKNESSES:
Disclaimer: I am not familiar with this field, so my observations and suggestions are based on my general understanding on the paper.

- Some definitions are not provided, making it difficult for readers to understand certain parts of the paper. For example line 133, 134 $\bigoplus$.

Minor:
1. Page 1 line 9, “such as”.

QUESTIONS:
1. What does $\bigoplus$ mean?
2. Could you provide some experimental results for CRO and also compare it with other algorithms?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of domain generalization through the lens of partial transportability, and introduces some new results for bounding the value of a functional of the target distribution, given data from source domains and assumptions about the data generating mechanisms.  Authors adapt existing parameterization schemes such as Neural Causal Models to encode the structural constraints necessary for cross-population inference. Some experiments and examples are also provided.\n\nSTRENGTHS:\n- Introducing partial transportability and canonical SCM to domain generalization is novel.\n\n- The theoretic results are sound and have complete proofs. They are potentially useful.\n\nWEAKNESSES:\n- The paper organization and presentation seem complicated to convey the idea of the paper (see below).\n- It is not clear how this bound can in turn help improve general DG problems.\n\nQUESTIONS:\nBased on my understanding, authors consider bounding the queries of generalization errors in non-transportable settings, by obtaining a bound for the worst-case losses. Authors use the formulation of canonical  models as a way of encoding constraints to derive such a bound. \n\n1. I believe there are existing works that assume some prior information regarding target domain and then derive bounds for the target loss in DG problems. If my understanding is correct, then I would like to treat the current work as **an alternative way** of encoding information about the target domain. Then the current way of presentation and organization indeed make  it complicated to understand the paper. In particular, I feel that examples 1 and 2 somehow distract reader's attention, while it is better to directly formulate the central problem of the paper. \n\n2. Other questions: \n   1. line 125:  what does $j$ indicate in the notation $S_{ij}$?\n   2. definition 4: is there any constraint on $q_{max}$ ? there may exist trivial but useless upper bound for some loss functions.\n   3. Suggest give more introductions about canonical SCM, to make the paper more self-contained.\n3. about Colored MNIST experiment: I don't get how this experiments serves. Do you want to show the proposed bound is correct? However, there is no mention of the value of $R_{P*}(h)$ here.\n\nLIMITATIONS:\nsee above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}