An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation
Authors: UZAIR AKBAR
Niki Kilbertus
Hao Shen
Krikamol Muandet
Bo Dai
Venue: NeurIPS 2025 spotlight ()
Keywords: Causal Inference
Data Augmentation
Instrumental Variables
Out-of-distribution Generalization
Causal Regularization
Primary area: probabilistic_methods
TL;DR: We show the effectiveness of data-augmentation for mitigating bias due to unobserved confounding, and this motivates the proposal of our novel method for the same.

Abstract: The technique of data augmentation (DA) is often used in machine learning for regularization purposes to better generalize under i.i.d. settings. In this work, we present a unifying framework with topics in causal inference to make a case for the use of DA beyond just the i.i.d. setting, but for generalization across interventions as well. Specifically, we argue that when the outcome generating mechanism is invariant to our choice of DA, then such augmentations can effectively be thought of as interventions on the treatment generating mechanism itself. This can potentially help to reduce bias in causal effect estimation arising from hidden confounders. In the presence of such unobserved confounding we typically make use of instrumental variables (IVs)—sources of treatment randomization that are conditionally independent of the outcome. However, IVs may not be as readily available as DA for many applications, which is the main motivation behind this work. By appropriately regularizing IV based estimators, we introduce the concept of *IV-like (IVL)* regression for mitigating confounding bias and improving predictive performance across interventions even when certain IV properties are relaxed. Finally, we cast parameterized DA as an IVL regression problem and show that when used in composition can simulate a worst-case application of such DA, further improving performance on causal estimation and generalization tasks beyond what simple DA may offer. This is shown both theoretically for the population case and via simulation experiments for the finite sample case using a simple linear example. We also present real data experiments to support our case.

An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation
Uzair Akbar" Georgia Tech
Niki Kilbertus TU Munich Helmholtz AI
Hao Shen TU Munich Fortiss GmbH
Krikamol Muandet Rational Intelligence CISPA
Bo Dai Georgia Tech Google DeepMind
Abstract
The technique of data augmentation (DA) is often used in machine learning for regularization purposes to better generalize under i.i.d. settings. In this work, we present unifying framework with topics in causal inference make case for the use DA beyond just the i.i.d. setting, but for generalization across interventions argue that when the outcome generating mechanism is invariant to our choice of DA, then such augmentations can effectively be thought of as interventions on the treatment generating mechanism itself. This can potentially help to reduce bias in causal effect estimation arising from hidden confounders. In the presence of such unobserved confounding we typically make use of instrumental variables (IVs)-souree of treatment randomization that are conditionally independent of the outcome. However, IVs may not be readily available as DA for many applications, which is the main motivation behind this work. By appropriately regularizing IV based estimators, we introduce the concept of /V-like (IVL) regression for mitigating confounding bias and improving predictive performance across interventions even when certain IV properties are relaxed. Finally, we cast parameterized DA show that when used in composition can simulate a worst-case application such DA, further improving performance causal estimation and generalization tasks beyond what simple DA may offer. This shown both theoretically for the population case and via simulation experiments for the finite sample case using simple linear example. We also present real data experiments to support our case.
well. Specifically,
IVL regression problem and
1 Introduction
classical problem in machine learning is that of pngression-usinee i.i.d. samples from some fixed, unknown distribution Px,y, we predict outcome Y values for unlabeled treatment X values. The use of regularization techniques is crucial for this task to achieve good generalization from training to test data [1]. Data augmentation (DA) [2, 3] is one such method, where each sample is randomly perturbed multiple times to grow the dataset size. However, these regression models cannot generally be interpreted causally as the statistical relationship between X and Y may arise from shared common causes, known as confounders, rather than from X influencing Y. Removing such confounders requires independently assigning values of X during data generation, known an intervention [4, 5].
Unfortunately, we seldom have access to the data generation process to be able intervene on variables. A common workaround is to use auxiliary variables to correct for unobserved confounders [6-8]. One such approach is that of instrumental variables (IVs) that represent certain conditional in- dependences in the system which can be used to identify the causal effect of X on Y [9-11]. Alas, IVs too are generally hard to find in many popular applications such as computer vision and natural lan- guage processing, motivating the need for more accessible ways to mitigate unobserved confounding.
Part of work done while at Max Planck Institute for Intelligent Systems and TU Munich.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent work therefore seeks to leverage more commonly available auxiliary variables to reduce seafooundinginduuced bias even when the causal effect itself cannot be identified [12-15]. Collectively referred to as causal regularization, these methods aim to learn predictors that generalize out-of- distribution (OOD) by discouraging reliance on spurious (i.e., non-causal.) correlations. Since distri- bution shifts often correspond to interventions on parts of the data-generating process [16, 4J, models that fail under such shifts typically do so because they exploit confounded relationships [17]. Tackling this root cause directly, causal regularization offers principled approach for more robust prediction.
In the same vein, more ambitious works have also explored the use of common regularization techniques, such as é,, lz [18] and the min- interpolator [19], for the same purpose of causal regularization. This is in contrast to the canonical use of such regularizers for estimation variance reduction and i..dd prediction generalization [1]. Other popular regularization methods, however, re- main understudied in similar context of un-identifiable causal effect estimation, motivating our work.
Our contributions. To this end, we provide a first analysis of DA for estimating un-identifiable causal effects using only observational data for (X, Y). Our contributions, summarized in Tab. 1, include: (i) DA as a soft intervention (Sec. 4.1): We show that DA can synthesize treatment interventions when the outcome function is invariant to DA, lowering bias in causal effect estimates when the intervention acts along spurious features. (ii) Introducing IV-like regression (Sec. 3): Relaxing the properties of IVs, we introduce the concept of IV-like (TVL) variables. This generalization renders IV regression ineffective at identifying causal effects, but when regularized appropriately via our proposed IVL regression, may still reduce confounding bias and improve prediction generalization across treatment interventions. (iii) DA parameters as IVL (Sec. 4.2): By casting parameterized DA as IVL., we show that its composition DA+IVL with IVL regression further reduces confounding bias beyond just simple DA by essentially simulating worst-case or adversarial application of the DA.
We validate our approach with theoretical results in a linear setting for the infinite-sample case, and simulation and real-data experiments in the finite-sample case.
Preliminaries
Consider treatment X and outcome Y takiing values in X' S I and Y C R' respectively. Given the set of functions H. == {h: : X + )),, the canonical setting described in the literature [4, 20] deals with estimating the function E H in the structural equation model (SEM) 2i of the following form'
(I) where Z, C. Nx. Ny are exogenous (and therefore mutually independent) random variables and the residual E = Y f(x)= = e(C) + Ny is assumed to be zero mean, i.c. [<] = O. Since M7 is potentially cyclic, a priori it may entail several or no distributions at all. However, here we make the assumption that for all (Xo. yo) E X' x y the unique limits
X T(Y, Z, Cx)),
Y = f(X) + eCC) + Ny,
Tim X lim T 2, C, nx); t
y=
lim Yt lim f(x-11) + c(c) + ny
t-000
t-x00
t-
N.C.Nx .yy meaning that the unique distribution entailed by Dt is in this equilibrium state. Of course, if Dt acyelic, these limits always exist. Note that assuming the ex- istence of such an equilibrium does not violate the classic independent causal mechanism (ICM) prin- ciple [4]; we defer interested readers to Appendix B for further details on cyclic SEMs and the ICM.
exist for any (z, c, nx, ny) ~ p
Given a proper convex loss l. R x R' + R.,, empirical risk minimization (ERM) uses a dataset D {(x;, H-) of n samples from Mi to minimize an empirical version of the statistical risk
R (h) = E. (((),)()))),
(2)
* O, the ERM minimizer hmm typically yields a biased estimate of f [5, 4]. This bias arises due to the exclusion of the (unobserved) common parent C' of X and Y. i.c. confounder, in the ERM objective (hence fittingly called the mnitted-variable bias [21]) and/or the model is cyclic (simultaneity bias [20, 22], or reverse causality [5] in the degenerate case). For simplicity we shall refer to either case by saying that X and Y are confounded and the resulting bias as the confounding bias [5].
over h H. However, since the residual & in Eq. (1) generally correlated with X, i.e. E [ XX]
'hhooughout this work we shall borrow and overload notation from [4]. See Appendix for of symbols. 2 'Pearl [5, p.78.184 184] similarly uses the term for any bias causing observational vs. interventional deviation; this also aligns with econometries [23, 20J. where both classified as sources of endogeneity (i.e., X Tf ④).

---

Table 1: A picture summary of our contributions. -> represents composition of operations or trans- formations, and < represents equivalence.
C
C
X
Type of Data
Topics in Causal Inference
Augmentation
%
Z
Lower confounding bius in causal effect estimate
None;
Data generating structural model
observational data
(a) Graph of
(b) Graph of Mi; (X))..
Figure I: Graph of D depicting an instrument Z that satisfies treatment relevance, exclusion restriction, un-confouuneedess and outcome rel- evance properties. An intervention on X gives the graph in (b). IV regression simulates such an intervention using only observational data.
Outcome
Treatment (soft) intervention
invariant DA
Worst-case or adversarial DA
Regularized IV regression (i)
2.1 Intervention for causal effect estimation
We can make X and the residual & uncorrelated via an intervention' do(X := X'), where we explicitly set X to some independently sampled X in Eq. (1) irrespective of parents, resulting now in the new SEM Mi; do(X == X') or Dt do(() as shorthand for when X' ~ P The distribution induced by this modified SEM called an interventional distribution (with respect to W.) under which the ERM objective from Eq. (2) now defines the following causal risk (CR) [12, 19, 24] as
R (h) = (X (b)
PERM
(h)
RERM
(h),
s.t.
X'
P
(3)
Minimizing Eq. (3) is meaningful in two important cases where ERM fails: (i) Causal effect estima- tion: The minimizer 1:20 of Eq. (3) gives us an unbiased estimate of the average trealment effect (ATE) [6] [==;))o(x==x(>>>>)) X x] f(x) that measures the causal influence of X on Y. (īi) Robust prediction: ATE based prediction Y values for unlabeled values is robust in the sense that can generalize across arbitrary OOD treatment interventions shifts in the treatment distribution [25]. Consequently, the causal risk minimizer i.2 is also robust predictor
the support of T Specifically, h:m minimizes the worst-case ERM objective over the set P of all possible intervention distributions xx over the support of pat [25], i.e. for P (Px: supp((xx)) ( S supp
2idd (x::)(X:=X)) (h).
hCR E argmin max NeW
RERM ExIEP
To better isolate the estimation error due confounding, define the causal excess risk (CER) [19]
CER (h) == (C) 12 (f).
This removes the irreducible noise from Eq. (3) (see Appendix A) and directly measures how far hypothesis h deviates from the true causal function f under interventions, that CER 0.
Since interventions are often inaccessible for computing the risk Eq. (3), we usually rely on observa- tional data/ distribution and additional variables to approximate them, as outlined in the next section.
2.2 Instrumental variable regression
One way to get an unbiased estimate of from the observational distribution of 21 is to use so- called instrumental variables Z with the properties [5, 4, 10, 9, 26] of: (i) Treatment Relevance: Z 1f X. (ii) Exclusion Restriction: Z enters Y only through X. i... Z IL R
(ii) Un- confoundedness: II g. (iv) Outcome Relevance: Z carries information about Y, i.e. Y 4 Z.
'A soft intervention replaces the mechanism Eq. (1) with alternative r' [4. p. 344.. This may potentially reduce confounding hetween X and
'Counterfactual definition of the exclusion restriction property [S. 248].

---

Conditioning Eq. (1) on Z and using
[| Z = - = 0 from the unconfoundedness property gives
E T Y 2] = [f(x) Z].
(4)
IV regression therefore entails solving Eq. (4) for f. which can be done by minimizing the risk [26]
RiV (h) = E [<(Y, ( [x( 2
(5)
For linear = = f() = h (-) with f. h E ! and squared loss E(y.y') == lly y'l. this gives the two-stage-laatt-suanne (2SLS) [27] solution where the first stage regresses from Z, and the second stage regresses Y from predictions [ 2 of the first stage to get the estimate hiy.
2.3 Data augmentation
In this work we restrict ourselves to data augmentation with respect to which f is invariant [3, 28]. The action of a group G is mapping 8: X' + X' which is compatible with the group operation. For convenience we shall write gx == 8(x, g). We say that f is invariant under G (or G-invariant) if
/ [gx) f(x),
vo (g. x) E 9 X.
Less formally, we say that the map gx, henceforth assumed to be continuous in x, is valid outcome- invariant DA transformation parameterized by the vector g E G. Let 9 have (unique) normalized Haar measure and Fg be the corresponding distribution defined over it. For some G ~ Pc. the canonical application of DA seeks minimize an empirical version of the following risk.
(6) Note that it is sufficient to have some prior information about the symmetries of f in order to be able construct such DA. For example, when classifying images of cats and dogs we already know that whatever the true labeling function may be, it would certainly be invariant to rotations on the images. G would then represent the random rotation angle, whereas Gx would be the rotated image x.
1 +ERM+ (h) (E(Y, (G
We wish to contrast the use of DA in this work with the canonical setting- mitigate overfitting, DA is used grow the sample size by generating multiple augmentations (Gx, y) for each data sample y) ~ 1 X.Y [3, 28, 29J. Such regularization, overfitting mitigation, estimation variance reduction, or i.i.d. prediction generalization is not the focus of this work and we intentionally provide Eq. (6) along with theoretical results that follow in the population case to emphasize that DA is not being used as a conventional regularizer. Instead, our goal is to improve causal effect estimation and robust prediction by re- purposing DA to mitigate hidden confounding bias in the data.
3 Faithfulness and Outcome Relevance in IVs
The distribution 12t X.Y.Z.c is faithful to the graph of 21 if only exhibits independences implied by the graph [4, 30.3 This standard assumption in IV settings renders outcome-relevance implicit and therefore rarely mentioned. In this section we discuss the case where only the first three IV properties are satisfied, i.e. outcome-relevance may not hold. Since such a Z may not be a valid IV, therefore identifiability of ATE is not possible in general as the problem in Eq. (4) can now be misspecified, having multiple, potentially infinitely many solutions when Y IL Z. Nevertheless, we shall refer to such a Z as /V-like (IVL) to emphasize that while Z may not be an IV, it may still be 'instrumental' for reducing confounding bias when estimating the ATE compared to the standard ERM baseline.
ERM regularized IV regression. Despite problem misspecification for IVL Z. the target function remains a minimizer for the IV risk in Eq. (5). Albeit, potentially not unique- for example, a linear h with squared loss leads to an under-determined problem in Eq. (5). We therefore propose the following regularized version of the IV risk for such an IVL setting,
RR (h) = 1e (h) + a1
(7)
where a > is the regularization parameter. The ERM risk as a penalty allows our estimations to have good predictive performance while the IV risk encourages solution search within the subspace where we know ff to be present. We refer to minimizing the risk in Eq. (7) as IVL regression.
Note that the motivation behind IVL regression is not the identifiability of f. but rather potentially better estimations of f with lower confounding bias. The next section provides a concrete example.
Also known stability in some texts [5. p. 48J.

---

C
X
C
X
GX
G
(a) Graph of a post DA. (b) Graph of 3; do(r = Gr)
Figure 2: The observational distribution of C, and YXYY, G, C") for graphs (a) and (b) respectively are the same. The former applies DA on X, whereas the later applies a (soft) intervention on X. Furthermore, for the graph in (b), G IVL.
Figure The ground truth function f in Example 2. The DA applied here corre- sponds to randomly translating the data sam- ples along their level-set by adding random noise sampled from the null-space of f.
Example (a linear Gaussian IVL example). For scalar 0, non-zero matrices r, T R and vectors T f, e E R" such that f T * so that the following SEM 2N solvable in (X
X T + r Z + T C +oNNx
y=
+ oNy,
where Z, C, Nx, Ny are conformable, centered Gaussian random vectors and Z is IVL w.r.t (X2
Now, the task to improve our estimation of compared standard ERM. We evaluate an estimate h using the CER, which for squared loss and covariance EOR Example simply comes out be
CERam hi))
(8)
Prior works use this form to quantify the error in ATE estimation [19, 12] or measure notion of
strength of confounding [18, 31, 24]. Similarly, we use it to measure confounding bias of population estimates 2t (Appendix A) and estimation error in finite sample experiments. The next results follow. Theorem (robust prediction with IVL regression). For SEM M in Example 1, the following holds:
(r ()==)) (h),
s.t. Pa = { <s & r
IVL, E argmin max RERM CEPa
Proof. See Appendix F.3 for the proof.
Theorem (causal estimation with IVL regression). SEM M of Example 1, for a < 00, we have
CER [V
< CER
2R [ERM)
equality iff E
E
X
Tdd X [.. 2] La.s
Proof. See Appendix F.4 for the proof.
Theorem 1 shows that IVL regression achieves optimal predictive performance across treatment interventions within the perturbation set Pa defined by a. Theorem 2 further states that this strictly reduces confounding bias in ATE estimates iff the perturbations align with spurious features of X, as indicated by the equality condition (also necessary for identifiability in linear IV settings [32, 25]).
4 Causal Effect Estimation using Data Augmentation
We dedicate this section to the main topic and point of this work discussing the potential of data
augmentation for improving predictive performance across interventions and reducing confounding bias ATE estimates. To that effect, for the rest of this work we shall consider the following SEM 21
X = TYY, C, Nx),
Y = f(X) + e(C) + NNy
(9)
which is assumed to have unique stationary distribution with exogenous C, Nx, Ny and the residual Ç = Y - f(X) is zero-mean, i.e. [[[] = O. We also have access to DA transformations G.X of X
parameterized by G ~ IP such as described in Sec. 2.3. Figure 2a shows the graph of 인 post DA.
Given samples for only (X, Y) and some valid DA parameterized by G, the task is to improve predictive performance across interventions and reduce confounding bias in ATE estimates. We now make two observations in the following sections and state the respective results that follow thereof.
Sce Appendix and Lemma details on solving for and sampling of (X, Y) in such linear, cyclic SEMs. All examples assume correlated X and residual Ç. L... E0N [X 0. otherwise there is confounding.
5

---

4.1 Data augmentation as soft intervention
Consider (soft) intervention on 2 where we substitute the mechanism 7 of X with Gr. With some abuse of notation, we shall represent this SEM by 2; do = GT) the graph of which is shown in Fig. 2b. Note that this SEM also has a unique stationary distribution (proof in Appendix F.2). Comparing the DA mechanism in 21 (Fig. 2a) and the intervention A; do(T = Gr) (Fig. 2b), we see:
Observation 1 (soft intervention with DA). Distributions P G.YY,G,C and pa:do(r==mr) identical. X,Y,G,C
We can hence treat samples generated from 21 via DA as if they were instead generated from 21; do(( = GT) by intervening on X. This allows us to re-write the DA+ERM risk from Eq. (6) as,
R +AG+ (h) ERM (
to emphasize that DA equivalent to a (soft) intervention and as such can be used to reduce confounding bias when estimating f. as we will show with the following example.
Example 2 (a linear Gaussian DA example), For scalars K, >> O, non-zero matrices r, T R and vectors T f, E RT such that f T * xx= so that the following SEM A is solvable in (X,Y)
X x where G. Nx. .y are conformable, centered Gaussian random vectors, x determines how much that GX is valid outcome invariant DA
K TT + oNx, Y =f= + K T ooy, GX x + -
(X,Y) are confounded and range T S mull(f) transformation of X parameterized by G with strength y > O. This transformation can be viewed as translating X along its level-set as shown in Fig. 3 and represents our prior knowledge about the symmetries for the purposes of this example.
Theorem (causal estimation with DA+ERM), For SEM 21 in Example 2, the following holds:
CERa
g1 ERM
a I CERx (bERM)
equality iff I= G] Las. E [g]
Proof. See Appendix F.5 for the proof.
That is. DA strictly reduces confounding bias in ATE estimate iff the induced intervention perturbs X along spurious features. Importantly, Theorem 3 suggests that lower confounding bias is not a 'free lunch' with outcome invariance of DA and practitioners may need domain knowledge to construct DA that targets spurious features. Fortunately however, Theorem 3 also suggests that with outcome invariance, DA should not perform worse than ERM. We say that DA+ERM dominates ERM on causal estimation [33, p. 48). Practitioners may therefore be advised to generously use such DA, as achieves regularization in the worst case, and mitigates confounding bias a 'bonus' in the best case.
4.2 Worst-case data augmentation with IVL regression
We once again point attention to the graph of A; do(r = Gr) from Fig. 2b to observe that: Observation 2 (IV-like DA parameters). In SEM 2; do(r == GT). the DA parameters G are IVL. In light of this we can now re- re-write the IV and IVI. risks for 21; d = GT) to respectively read
D
RTy
A:do(r== Gr) (h),
R + (h) RivL.
2lddo(r:--1 GT) (N).
Corollary (worst-case DA with DA+IVL regression). For SEM A in Example holds that -
IV E argmin max R (h), s.t. G. = "
r g'ggTT T x
+
h gEg.
Proof. The result follows from Observation 1, Observation and Theorem 1.
Corollary 2 (causal estimation with DA+VL regression). For a, 7 < co in SEM 2 from Example 2,
CER at (b...t ) CCERa (DDAO
a
equality iff E |G G] Las. E [
Proof. The result follows directly from Theorem and Observation 2.
Using DA parameters as IVL therefore simulates a worst-case, or adversarial application of DA within a set of transforms Ga. Of course Corollary can also be viewed as a predictor that generalizes to treatment interventions encoded by G.. As is intuitive, such worst-case intervention improves our ATE estimation so long as the features of X intervened along include some that are spurious (Corollary 2). DA and IVL regression may therefore be used in composition if the application can benefit from regularization and/ or better prediction generalization across DA-induced interventions, with 'bonus' of lower confounding bias if the DA also augments any spurious features of X.

---

everage DA-SINL
DA IVL,
0.1
DA IVVE2
DA-FV
0.0
6
7 K E [0, 1]
(b) 7: K [10-5, 10
(c) [10-2.5 2.5 10), 8 ==
Figure 4: Simulation experiment for linear Gaussian SEM. represents the amount of confounding, is the strength of DA and a is the IVL regularization parameter. Each data-poin represents the average nCER 25 trials with 95% confidence interval (CI).
Related Work
Causal regularization is perhaps the most appropriate classification for this work. These methods aim for more robust prediction by mitigating the upstream problem confounding bias in a more accessible way than is required for full identification. This is done, for example, by relaxing properties of auxiliary variables [12-15], as we have done via our IVL. approach. Most relevant, however, are methods that re- re-purpose common regularizers, canonically used for estimation variance reduction and i.i.d. prediction generalization, for confounding bias mitigation. Of note is [18], where a certain linear modeling assumption allows the estimation of ||f] from observational (X,Y) data, which then used to develop
pros-validation scheme for f l2 regularization. [19] conducted a similar theoretical analysis for the min-norm interpolator. To the best of our knowledge, we are the first to study the same for DA-rrepurposing re-purposing yet another ubiquitous regularizer mitigate confounding bias.
Domain generalization (DG) [34] methods aim for prediction generalization to unseen test domains via robust optimization (RO) [35] over perturbation set P of possible test domains p as
R (h)
max RE (h). pEP
Since generalizing to arbitrary test domains is impossible, the choice perturbation set encodes one's assumptions about which test domains might be encountered. Instead of making such assumptions priori, it is often assumed to have access to data from multiple training domains which can inform one's choice of perturbation set. This setting is explored in group distributionally robust optimization (DRO) [36]. Variations have been used to mitigate confounding bias and subsequently generalize to treatment interventions when used with interventional data [16, 37], confounder information (i.c. entire graph) [38-40] or some proxy thereof in the form of environments [41-43, 38]. We, however, do not assume access to any of these and instead synthesize interventions via DA.
Counterfactual DA strategies have been the primary lens for causal analyses DA [44-50]. These aim for prediction robustness to treatment interventions via DA simulated counterfactuals? As as access to full SEM [45, 46], auxiliary variables [44, 46, 49, 50], causal graphs [47, 48]. By contrast, show that outcome invariance of DA suffices for treatment intervention robustness without invoking counterfactuals. Moreover, prior work has largely overlooked causal effect estimation, often assuming reverse-causal settings where the ATE becomes trivial [44, 46, 45]. Ours is the first framework to study ATE estimation via DA with minimal structural assumptions.
with counterfactual reasoning more broadly, this requires strong assumptions
Invariant prediction based methods aim to make predictions based on statistical relationships that remain stable across all domains in P. A common assumption, for instance, is that IPy x is invariant across P. with only the marginal Fx being allowed to vary. Invariance also closely linked to causal discovery-following the classic ICM principle [4], causal mechanisms remain stable under interven- tions on inputs [25, 17]. This connection has inspired approaches that enforce invariance conditions
"Representing an SEM with exogenous noise distribution conditioned on some variable Y = y by My the counterfactual SEM -y dX X : x) is an intervention do(X = x) on ly The resulting counterfactual distribution then captures questions like: "After observing Y y. what would have been had X been true."

---

to recover causal structures [16, 37]. IV regression can also be viewed as one such method, where the goal is to learn predictors whose residuals are invariant to the instruments [10, 9, 26, 51, 7]. More broadly, the principle of invariance, whether motivated by causality or otherwise, has proven useful for improving prediction generalization across heterogeneous settings [15, 41, 52, 14, 53-56, 34).
6 Experiments
We began by presenting results in the infinite-sample setting to emphasize that mitigating confounding bias is fundamentally not sample size issue, f.e., not solvable through traditional regularization alone. In this section, we turn to the finite-sample regime and empirically evaluate the effectiveness of DA in reducing hidden confounding bias. Importantly, we do not use DA for its conventional purpose of augmenting data to improve i.idd generalization or reduce estimation variance. Throughout all experiments, we therefore fix the number of samples in the augmented dataset match that of the original dataset since our focus lies squarely on robust prediction via confounding bias mitigation.
Finding baselines for evaluating our results is however challenge the problem of mitigating confounding bias given only observational (X, Y") data and symmetry knowledge via DA is quite underexplored. Nevertheless, for the sake of completeness methods from domain generalization, invariance learning and causal inference literature to be used as baselines. These methods often require access additional variables (e.g. IVs, confounders, domains/environmentss etc.), and to maintain fairness we will replace these with DA parameters G. Such comparison is conceptually valid since by virtue of being DG methods, they are essentially solving a robust loss of a similar form as in Corollary 1, giving us meaningful baselines for DA+IVL.
make an effort re-purpose existing
In addition to standard ERM, DA and IV regression, our baselines include DRO [36], invariant risk minimization (IRM) [41], invariant causal prediction (ICP) [16], regularization with invariance on causal essential set (RICE) [56], variance risk extrapolation (V-REx) and minimax risk extrapolation (MM-REx) [38]. We also include the causal regularization method by Kania and Wit [12] and the l f2 approaches by Janzing [18]. We discretize G if the method accepts only discrete variables. For IVL regression, we select the regularization parameter a in a variety of ways, including vanilla cross validation (CV), level-based CV (LCV) and confounder correction (CC) as described in Appendix D. Other implementation details are provided in Appendix E, and the code to reproduce our results is publicly released at https: / gi thub. com wwt/eooooaaaiaaeea /tw.//uzaaaarrrraaaa caattbarrraauaaaaaaeaa cawsall data- aetrraauuueenttiio...
To make CER based evaluation more interpretable for our experiments, we propose the normalization
CER
nCERar (h) CER + ( [0, 1],
ho(-) (x [Y],
where ho represents the null treatment effect, i.e. when X has no causal influence on Y, then E X] = [x) The normalized CER (nCER) can be considered a generalization of the metries used by [18, 24, 31] in linear settings and similarly has the interesting property that it is 0 for the ground-truth causal solution h = f ho but if there is pure confounding for h # f f= hg- Janzing argues in [24, 31] that using an Euclidean norm instead of the weighted norm in Eq. (8) is more relevant for causal settings, which also motivates our choice when evaluating results of the simulation and optical-device experiments described below. Conceptually, this is equivalent to evaluation based on the causal risk in Eq. (3) under the interventional distribution X" ~ *N(0 Im).
6.1 Simulation experiment
For the finite sample results of the linear SEM 인 from Example 2, by taking
32, k 31 9 we sample a new f, e and T E m from a standard normal distribution for each of the 32 experiments for every combination of и. and n. Each time we construct a := Vo with & rows as orthonormal basis of null(f), such that the SVD of f is
(dimension of G), a = 0.1 and fixing - = Om.?
0 01x(m-1)
v
f= [u
Upl
0 (m-1))x
(m-
V
Although this construction of I relies on direct knowledge of which is of course unavailable in prac- tice, we include it here purely for illustrative purposes, We treat access to I as having prior knowledge about the structural symmetries of f, noting that this information alone is insufficient recover f.
Simulation results are similar under cyclic setting with non-trivial and discussed under Appendix E.
8

---

Simulation Date
Optical Device Date
Colored MNIST Data
DA B833
DA LV
DA 1V
DA IV
BR3C
ICE-
RICE
V-RES
Jaczing
Kanis Wil 22
0.2
0.5
066
0.7
0.3
0.5
0.50
nCER
nCER
CCER
Figure 5: Experiment results; common OOD generalization benchmarks compared against the ERM, DA+ERM and DA+IV baselines including DA+IVL.
We then generate n = 2048 samples of (X, Y) for each experiment. For ERM we use a closed form linear OLS solution. For DA+IV, we make use of linear 2SLS Finally, DA+IVL.,, was implemented using a closed form linear OLS solution between empirical versions (see Proposition 1) of
X' = vax + (VI+a- va) Z), Y' vay + (vI+ -al+a-va)BL
E|y EaY
2].
Our first experimental result in Fig. 4a compares the different estimation methods across varying levels of confounding E [0, 1]. As expected, ERM performance degrades with increasing confounding. Applying DA alone already brings us closer to the causal solution, while DA+IVL achieves even better performance. DA+IV regression is unstable and generally performs poorly as it is under-determined.
Next, we fix the confounding and DA strengths at x = y = 1, and sweep over the regularization parameter [10-5, 10$] for DA+IVL,. Figure 4b shows that optimal performance is achieved for intermediate values of a, confirming that arbitrarily small values of a, while beneficial in the theoretical population setting (as suggested by Eq. (27) in the proof of Theorem 2), are suboptimal for finite samples." 10 We also find that both CV and CC strategies effectively select reasonable values of a.
Lastly, Fig. 4c examines sensitivity to the DA strength y E [10-2.5, 10). for fixed confounding strength K = 1. As expected, stronger DA results in stronger interventions on X, which improves causal effect estimation. However, we also observe diminishing returns; when the variation induced by DA is either too small or too large, DA+IVL, does not yield significant improvements over the DA+ERM baseline.
For completeness, we also benchmark our approach against other baseline methods on 16 distinct simulation SEMs with 2048 samples each. Aggregated results are presented in Fig. 5 (left most).
6.2 Real data experiments
Optical device dataset. The dataset from [24] consists of 3 x 3 pixel images X displayed on a laptop screen that cause voltage readings Y across a photo-diode. A hidden confounder C controls two LEDs; one affects the webcam capturing X, the other affects the photo-diode measuring Y. The ground-truth predictor f is computed by first regressing Y on (ø(X), C), where ((X) are polynomial features of X with degree d E {1, 5} that best explains the data (degree 2 in most cases). The component corresponding to C' then removed recover f. We add Gaussian noise G G~ (0, Σ Ex/10) for DA and fit the methods from Sec. 6.1 on features (GGX) for n = 1000 samples across 12 datasets. Note that using the same ground-truth polynomial degree for é during evaluation is important here so to avoid introducing statistical bias from model-misspecification as our analysis squarely focuses on confounding bias. Figure (middle) shows the results, where DA+ERM improves over ERM, and DA+IVL performs even better, outperforming other baselines.
We evaluate on the colored MNIST dataset [41], where labels are spuriously correlated with image color during training, but this correlation is flipped at test time. We use the same neural architecture and parameters as [41] across all baselines, training with the IV-hased objective described in the Appendix C. DA is implemented via small perturbations to hue, brightness, contrast, saturation, and translation, each parameterized by G (2. Although these do not
Colored MNIST.
We conjecture that this may be due to outcome invariance not holding exactly in practice. A more rigorous investigation is deferred to future work in order to keep the current manuscript more focused.

---

directly manipulate color, the actual spurious feature, they still help reduce confounding, Results in Fig. 5 (rightmost) show that ERM underperforms, DA+ERM provides substantial gains, and DA+IVI performs competitively with the best DG baselines, with DA+IVI. achieving the best overall performance. Interested readers may also visit Appendix E.3, where we clarify the connection of the colored MNIST model with the cyclic SEM from Eq. (9).
7 Limitations
Necessity and practicality prior knowledge. As discussed Sec. 4, outcome invariance alone does not suffice to lower confounding bias and practitioners may need domain knowledge to construct DA that targets spurious features as well. Alternatively, one can also take a 'carpet bombing' approach by exhausting all available outcome invariant DA in hope that some may align with spurious features. Nevertheless, under outcome invariance, our methods should perform no worse than standard ERM.
Fundamentally, causal estimation from purely observational data is impossible without untestable assumptions. For instance, the IV (or IVL) assumptions restriction are inherently untestable and must be justified through domain knowledge. Moreover, the requirement of alignment with spurious features in Theorem is not an artifact of our IVL relaxation-it is rephrasing of the exclusion principle that underlies identifiability in IV regression. If an IV does not influence Y through the spurious features of X, the corresponding causal components of f cannot be identified [25]. IVLs, being relaxations of IVs, inherit these same untestable premises.
un-confoundednes and exclusion
Viewed through the lens of s/VLL (Observation 2), our assumptions on DA are arguably more mod- est than they may initially seem, especially since symmetry-based DA model has well-established precedent in the literature [3, 28, 53, 57 57-633. This correspondence can summarized as follows:
un-testable DA assumptions
un-testable IVIVL assumptions
outcome-invarianee + spurious targets
un-confoundedness + exclusion
popular model for DA
benign failure violated
In this light, our framework may in fact be quite practical in domains where valid IVs (or other auxil- caatme--ivarinncss
iary variables) are scarce, but plausible
data augmentations are abundant.
Finally, we recognize the hesitation in committing to strict notions of outcome invariance in practice and leave a more thorough exploration of approximate or even violated invariance to future work.
Choice of a. Selecting the IVL regularization parameter a in finite-sample settings is not straight- forward. As outlined in Appendix D, we propose several strategies that work well empirically, though some may appear less principled since is tuned via vross-validation within the same distribution, even though the task concerns OOD generalization. This challenge not unique to IVL., but rather a broader limitation common to DG methods [64].
8 Conclusion
We conclude that our proposed causal framework for data augmentation (DA) enables re- re-purposing the widely used i..dd generalization tool for OOD generalization across treatment interventions. By interpreting outcome-invaree invariant DA as interventions and IV-like variables, our approach reduces confounding bias and consequently improves both causal effect estimation and robust prediction.
Acknowledgments
To my co-authors for their patience, to Zulfigar for being my rubber duck and saving the OpenReview submissions minutes before the deadline, and to all of ML pyos for lightening the chaos with comedy. Thank you.
This work was supported by the NSF (ECCS-2401391, IIS- IIS-2403240), and ONR (N0001422 42512173).
10

---

References
[1] Vladimir Naumovich Vapnik. Statistical learning theory. Wiley, 1998.
[2] Connor Shorten and Taghi M. Khoshgoftaar, A survey on image data augmentation for deep learning. Journal of Big Data, 6:60, 2019, doi: 10.1 10.11666644033777011-999914
[3] Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Redde On the benefits of invariance in neural networks, 2020. arXiv:2005.00178.
[4] Jonas Peters, Dominik Janzing, and Berharrd Schölkopf. Elements of causal inference: Foundations and learning algoritiuns. The MIT Press, 2017.
[5] Judea Pearl. Causality. Cambridge University Press, 2009.
[6] Liyuan Xu and Arthur Gretton. neural mean embedding approach for back-door and front-door adjustment, 2022. arXiv:2210006610.
[7] Tongzheng Ren, Haotian Sun, Antoine Moulin, Arthur Gretton, and Bo Dai. Spectral representation for causal estimation with hidden confounders. In International Conference on Artificial Intelligence and Statistics, 2025.
[8] Arash Mastouri, Yuhang Zhu, Limor Gultchin, Anna Korba, Ricardo Silva, Matt Kusner, Arthur Gretton, and Krikamol Muandet. Proximal causal learning with kernels: Two-stage estimation and moment restriction. In International Conference on Machine Learning, volume 139, 2021.
191 Rahul Singh, Mancesh Sahani, and Arthur Gretton. Kernel instrumental variable regression. In Advances in Neural Information Processing Systems, volume 32, 2019.
[10] Rui Zhang, Masaaki Imaizumi, Bernhard Schölkopf, and Krikamol Muandet. Instrumental variable regression via kernel maximum moment loss. Journal of Causal Inference, I1(1), 2023. doi: 10.1515/ /0i-2022-0073.
[11] Niki Kilbertus, Matt J. Kusner, and Ricardo Silva. A class of algorithms for general instrumental variable models. In Advances in Neural Information Processing Systems, volume 33, 2020.
[12] Lucas Kania and Emst Wit. Causal regularization: On the trade-off between in-sample risk and out-of- sample risk guarantees, 2023, arXiv:2205.0155
[13] Peter Bühlmann and Dominik Cevid. Deconfounding and causal regularisation for stability and external validity. International Statistical Review,
2020. doi: 10. 1/insr. 12383.
[14] Michael Oberst, Nikolaj Thams, Jonas Peters, and David Sontag. Regularizing towards causal invariance: Linear models with proxies. In International Conference on Machine Learning, volume 139, 2021.
[15] Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: Het- erogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology. 83(2):215-246, 2021.
[16] Jonas Peters, Peter Buhlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: Identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 1865) 2016. doi: 10.1111/rsst) 12167.
[17] Abbavaram Gowtham Reddy, Celia Rubio Madrigaa, Rebekka Burkholz, and Krikamol Muandet. When shift happens- confounding is to blame, 2025. arXiv:2505.21422
[18] Dominik Janzing. Causal regularization. In Advances in Neural Information Processing Systems, volume 32, 2019.
[19] Chennuru Vankadara, Luca Rendshurg, Ulike von Luxburg, and Debarghya Ghoshdastidar, Interpolation and regularization for causal learning. In Advances in Neural Information Processing Systems, volume 35, 2022.
[20] William H. Greene. Econometric analysis. Prentice Hall, 2003. ISBN 9780130661890.
[21] Kevin A. Clarke. The Phantom Menace: Omitted variable bias in econometric research. Conflict Management and Peace Science, 22(():341 2005. doi: 10. 000000007338944500333111
[22] John Fox. Simultaneous equation models and two-stage least squares. Sociological Methodology, 10: 130-150, 1979. doi: 10.2307/270769.

---

[23] Michael R. Roberts and Toni M. Whited. Endogeneity in empirical corporate finance. In Handbook of the Economics Finance, volume 2. chapter 7. pages 493-572. Elsevier, 2013. doi: 10.1016/ 1777--0-4--43554-88000000
[24] Dominik Janzing and Bernhard Schölkopf. Detecting confounding in multivariate linear models via spectral analysis. Journal of Causal Inference, 6(1), 2018.
[25] Rune Christiansen, Niklas Pfister, Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A causal framework distribution generalization. IEEE Transactions on Pattern Analysis and Muchine Intelligence, 44(10):6614 6630, 2022. doi: 10.1 10.1000//AMMM2.221030044 3099460
[26] K. Muandet, A. Mehrjou, S. K. Lee, and A. Raj. Dual instrumental variable regression. In Advunces in Neural Information Processing Systems, volume 33, 2020.
[27] David A. Belsley. Two-or three-stage least squares? Computer Science in Economics and Management, 1: 21-30, 1988. doi: F0.1007/B004355000
[28] Shuxiao Chen, Edgar Dobriban, and Jane H. Lee. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 21(245):1-11, 2020.
[29] Artem Savkin, Thomas I Lapotre, Kevin Strauss, Uzair Akbar, and Federico Tombari. Adversarial appearance learning in augmented Cityscapes for pedestrian recognition in autonomous driving. In IEEE International Conference on Robotics and Automation, pages 3305-3311, 2020. doi: 10. 10.1109/CRRA4094522200
9197024.
[30] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[31] Dominik Janzing and Bernhard Schölkopf. Detecting non-causal artifacts in multivariate linear regression models. In International Conference Machine Learning, volume 80, 2018.
[32] Jefrey M. Wooldridge. Econometric Analysis of Cross Section and Panel Data. The MIT Press, 2010.
[33] Erich L. Lehmann and George Casella. Theory of Point Estimation. Springer, 2nd edition, 1998.
[34] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, volume 28, 2013.
[35] Aharon Ben-Tal, Laurent EI Ghaoui, and Arkadi Nemirovski. Robust Optimization. Princeton University Press, 2009, doi: 10.1515/97814000 40078884401111 1050.
[36] Shiori Sagawa. Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[37] Christina MeinzeeDemm. Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear 1115/ei-2011-0011
models. Journal of Causal Inference. 6(2), 2018, doi:
[38] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. diiir---iittrbuttoon generalization via risk extrapolation (REx). In International Conference Machine Learning. 2021.
[39] Chaochao Lu, Yuhuai Wu, José Miguel Liernto and Bernhard Schölkopf. Invariant causal representation learning for out-of-distribution generalization. In International Conference on Learning Representartinss 2022.
[40] Hugh Dance and Benjamin Bloem-Reddy Reddy. Counterfactual cocycles: A framework for robust and coherent counterfactual transports, 2025. arXiv:240S.1384.
[41] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization, 2019. Sxiv:1907.00899
[42] Ahsan J. Cheema, Katherine L. Marks, Hamzeh Ghasemzadeh, Jarrad H. Van Stan. Robert E. Hillman, and Daryush Mehta. Characterizing vocal hyperfunction using ecological momentary assessment of relative fundamental frequency. Journal of Voice, 2024. ISSN 0892-1997, 1997. doi: 202iee2/.
[43] Seunghyup Han, Osama Waqar Bhatti, Woo-Jin Na, and Madhavan Swaminathan. Reinforcement learning applied to the optimization of power delivery networks with multiple voltage domains. In 2023 IEEE MTT-S International Conference on Numerical Electromagnetic and Multiphysics Modeling and Optimization (NEMO). 2023. doi: 10.1 1000/EE005111.202333... 10222.2. 10202224.
12

---

[44] Maximilian Ilse, Jakub M. Tomezak, and Patrick Forré. Selecting data augmentation for simulating interventions. In International Conference on Machine Learning, volume 139, 2021.
[45] Jianhao Yuan, Francesco Pinto, Adam Davies, and Philip Torr. Not Just Pretty Pictures: Toward inter- ventional data augmentation using text-to-image generators. In International Conference on Machine Learning, 2024.
[46] Amir Feder, Yoav Wald, Claudia Shi, Suchi Saria, and David Blei. Data augmentations for improved (large) language model generalization. In Advances in Neural Information Processing Systems, volume 36, 2023.
[47] Silviu Pitis, Elliot Creager, Ajay Mandlekar, and Animesh Garg. MoCoDA: Model based counterfactual data augmentation. In Advances in Neural Information Processing Systems, volume 35, 2022.
[48] Núria Armengol Urpi, Marco Bagatella, Marin Vlastelica, and Georg Martius, Causal action influence aware counterfactual data augmentation. In International Conference on Machine Learning, volume 235, 2024.
[49] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International Conference on Machine Learning, 2021.
[50] Ahmed Aloui, Juncheng Dong, Cat Phuoe Le, and Vahid Tarokh. CATE estimation with potential outcome imputation from local regression. In Conference on Uncertainty in Artificial Intelligence, volume 286, 2025.
[51] Liyuan Xu, Yutian Chen. Siddarth Srinivasan, Nando de Freitas, Arnaud Doucet, and Arthur Gretton. Learning deep features in instrumental variable regression. In International Conference on Learning Representations, 2021.
[52] Bo Dai, Niao He, Yunpeng Pan, Byron Boots, and Le Song. Learning from conditional distrihutions via dual embeddings. International Conference on Artificial Intelligence and Statistics, volume 54, 2017.
[53] O. Montasser et al. Transformation-inaariae learning and theoretical guarantees for OOD generalization. In Advances in Neural Information Processing Systems, volume 37, 2024.
[54] Fanny Yang, Zuowen Wang, and Christina Heinze-Demn Invariance Invariance-inducing regularization using worst- case transformations suffices to boost accuracy and spatial robustness. In Advances in Neural Information Processing Systems, volume 32, 2019.
[55] Sravan Jayanthi, Letian Chen, Nadya Balabanska, Van Duong, Erik Scarlatescu, Ezra Ameperosa, Zul- fiqar Haider Zaidi, Daniel Martin, Taylor Keith Del Matto, Masahiro Ono, and Matthew Gombolay. DROID: Learning from offline heterogeneous demonstrations reward-policy distillation. Conference on Robot Learning, volume 229. PMLR, 2023.
[56] Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. In /EEF/CVF Conference on Computer Vision and Pattern Recognition, 2022. doi: 10.1 101110992222288.0222... 20220000040022004
[57] H. Shao et al. A theory of PAC learnability under transformation invariances. In Advances in Neural Information Processing Systems, 2022.
[58] A. Fawzi and P. Frossard, Manitest: Arc classifiers really invariant? In British Machine Vision Conference, 2015.
[59] Y. Duhois et al. Lossy compression for lossless prediction. In Advances in Neural Information Processing Systems, 2021.
[60] Petrache and Trivedi. hwapooiiinaiiiogggnraliiiaaa trade-offs under (approximate) group equivari- ance. In Advances in Neural Information Processing Systems, 2023.
[61] D. Romero and S. Lohit. Learning partial equivariances from data. In Advances Neural Information Processing Systems, 2022.
[62] S. Zhu et al. Understanding the generalization benefit of model invariance from a data perspective. In Advancex in Neurai Information Processing Systems, volume 34, 2021.
[63] S. Wong et al. Understanding data augmentation for classification: When to warp? In Digital Image Computing: Techniques and Applications, 2016.
13

---

[64] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference Learning Reprexentations, 2021.
[65] Tom Heskes, Bias-variance decompositions: The exclusive privilege of Bregman divergences, 2025. arXiv:250118581
[66] Steffen L. Lauritzen and Thomas S. Richardson. Chain graph models and their causal interpretations. Journal of the Royal Statistical Society Series B: Statistical Methodology, 64(3):321-348, 2002. doi: 10.1111/1467-9861 98300000000
[67] Gustavo Lacerda, Peter Spirtes, Joseph Ramsey, and Patrik O. Hoyer. Discovering cyclic causal models by independent components analysis. Conference on Uncertainty in Artificial Intelligence, pages 366-374. AUAI Press, 2008.
[68] Antti Hyttinen, Frederick Eberhardt, and Patrik O. Hoyer. Learning linear eyclic causal models with latent variables. Journal of Machine Learning Research, 13:3387-3439 2012.
[69]
Mooij. Dominik Janzing, Tom Heskes, and Bernhard Schölkopf. causal discovery with cyclic additive noise models. In Advances in Neural Information Processing Systems, volume 24, 2011.
[70] Stephan Bongers, Patrick Forré, Jonas Peters, and Joris M. Mooij, Foundations of structural causal models with cycles and latent variables. The Annals of Statistics, 49(5), 2021. doi: 10.121421-AOS2064,
[71] Carl Christ. Cowles Commission's contributions to econometrics at Chicago, 1939-1955. Journal of Economic Literature, 32(1):30-59 1994.
[72] Mordecai Ezekiel. The Cobweb theorem. The Quarterly Journal of Economics, 52(2), 1938. doi: 10.2307/1881734
[73] John F. Muth. Rational expectations and the theory of price movements. Econometrica, 15-3533 1961.
[74] Amold Theil, Three-stage least squares: Simultaneous estimation of simultaneous equations. Econometrica, 30(1):54-78, 1962. doi: 10.2307/1912287.
[75] Alastair Hall, Generalized method moments. Companion to Theoretical Econometrics, chapter 11, pages 230-255. Wiley, 2003. doi: 10. (00229980447996244.cc
[76] Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method moments for instru- mental variable analysis. In Advances in Neural Information Processing Systems, volume 32, 2019.
[77] Greg Lewis and Vasilis Syrgkanis. Adversarial generalized method of moments, 2018, arXiv: 1rXiv:1803.07111 180.00714.
[78] Johnston. Econometric Methods. McGraw Hill, second edition, 1971.
[79] Roger A. Horn and Charles R. Johnson. Matrix Analysis. Cambridge University Press, 1985.
[80] Dennis S. Bernstein. Matrix Mathematics: Theory, Facts, and Formulas. Princeton University Press. second edition, 2009.
14

---

Appendix-An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation
Uzair Akbar Georgia Tech
Niki Kilbertus TU Munich Helmholtz AI
Hao Shen TU Munich Fortiss GmbH
Krikamol Muandet Rational Intelligence CISPA
Bo Dai Georgia Tech Google DeepMind
Contents
A Confounding Bias
17
Simultaneity Cyclic Structures in Equilibrium
18
Regression Supplement..
20
D IVL. Regression Supplement.
22
E Experiment Supplement
23
E.I Simulation experiment
23
E.2 Optical device experiment
24
Colored-MNIST experiment
24
Proofs
27
F1 Proof of Proposition 1 IVL regression closed form solution in the linear case
27
F2 Prouf of Proposition 2 Existence of un interventional distribution given a DA
28
F3 Proof of Theorem Robust prediction with IVL regression
29
F44 Proof of Theorem 2- Causal estimation with IVL regression
31
F.5 Proof of Theorem 3 Causal estimation with DR+
33
Miscellaneous supporting lemmas
34
15

---

List of Symbols
The notation largely borrowed from [4], with some overloading where necessary.
R R
Euclidean space; dimension conformal with inferred from context.
x Scalar.
x
Vector. When x is described vector, it means x is flat x matrix.
X
Matrix.
X
Set.
Random vector.
MN
SEM.
X
Random vector X with its SEM M2 specified when unclear from context.
P
Distribution of X entailed by Mt. Superscript dropped if clear from context.
Σ
Variance-covariant covariance matrix of X under distribution "X"
DR
Cross-on coos-coova matrix of and Y under distribution pot .XYY'
X.Y
[X]
Expected value of X under distribution p Intervention XX is set to x during data generation.
do(X == x)
dx(X)
Shorthand for do X == X') where X' P is i... to X. Intervention SEM.
21; do xx)
xxx=x
SEM with mechanisms of 2t, but exogenous noise distribution pot N|X=x'
my: do == x)
Counterfactual inte-innnnnnnoon SEM My=y.
X 1 Y
Random vectors X, Y are statistically independent, i.e. pMl -
x 1 y
x, are perpendicular, i.e. x y = 0. For random vectors, X T = 0 a.s.
Population/ infinite-sample estimate based distribution 1
Finite-sample estimate based on samples in the dataset
16

---

A Confounding Bias
Statistical vs. causal inference.
The target estimand for the statistical risk in Eq. (2) is the Bayes optimal predictor E Y X = x]. And the target estimand for the causal risk in Eq. (3) is the average treatment effect (ATE) Ed X [Y X = x] = fxx). As such, statistical inference is concerned with predictions outcome Y. whereas causal inference is concerned with estimating f(x).
Statistical vs. confounding bias. Both types of inference are subject to bias. Statistical bias arises due to misspecification of the hypothesis class H. whereas confounding bias arises due to how the data are generated. The former is therefore property of the estimator while the later is property of the data itself. For an estimator h with the expected value h(.) = EE iiP (P)())
we define these as
X x] f((x), Confounding bias x = f(x) TM [Y] X x].
Statistical bias
E.n Y
Bias-variance decomposition of the causal risk. Because the treatment X and residual are not correlated under M2; do(X) in Eq. (I), for any loss function that admits 'clean' or 'additive' bias-variance decomposition [65], the causal risk in Eq. (3) also admits bias-variance decomposition. Using squared loss example, we have for some hypothesis h.P
2R (1D)
Y -i \
[(x) h (x(x)
(Structural eq. of Y.)
[|E|]_] + =DDidd(()))
[|1(x]) h2 (x)))
(Cross term is 0 & IL X.d)
[
E
[|1(x) (
(P x I do)(()) identical by construction.)
ineduciblc noise
estimation error, CER (hx)
We show by following standard procedure that
an CERan [i²)]
20 E E ((](())-(xx))))
- |
(averago) bias
variance
Since 1: T(X) for any population estimate, CER equals the average squared estimation bias
CERAT (1.m) = E
= E [1 (x))
h
-(x)))
For rich enough hypothesis class, the ERM estimate coincides with the Bayes optimal predictor FOR (+) T X .] and the CER exactly equals the (average squared) confounding bias PERM
as we define it above. For general estimate f.P. however, the CER also contains statistical bias. Nevertheless, our claims of "better causal estimation via reducing confounding bias" rest the fact that we are essentially manipulating the data via DA and/or using treatment randomization sources in the form of IVLs. And recall that confounding bias is property of the data.
17

---

B Simultaneity as Cyclic Structures in Equilibrium
Linear eyclic assignments
SEMs with cyelic structures have been well studied both in the linear case [66-68], as well as the non-linear case [69, 70]. Here briefty provide causal interpretation to linear simultaneous equations as SEMs with cyclic assignments.
Consider square matrix M E Rd and the SEM
W MW N
(10)
where random noise vector exogenous and M allows for a cyclic structure. We enforce (Ia- M)
to be invertible so that the above equation has unique solution W for any given N. Re-writing the structural form in Eq. (10) into a reduced form, the distribution over W defined by
W (La M) NN.
(11)
One way we can present a causal interpretation of the above solution is to view as a stationary point the following sequence of random vectors We
W MW;-1+M
which converges M has a spectral norm strictly smaller than one so that M'
+ op.
The structural form Eq. (10) essentially describes the iterative application of this operation. And in the limit the distribution of lim,++oe W will be the same the reduced form Eq. (11). Although equivalent, reduced form of cyclic SEM (if one exists) obscures the causal relations in the data generation process.
Furthermore, we restrict our models to not have any "self-cycles" (an edge from a vertex to itself). So, e-g., the matrix M in Eq. (10) has all diagonal entries. This only simplifies our analysis by providing a simple and intuitive interpretation for our definition of DA in Sec. 2.3, but also ensures that non-linear SEMs entail unique, well-defined distributions under mild assumptions [70, 67].
Similarly we can write the example SEM 27 from Example this (block matrix) form as
X
0 mmxm f
X y
r
T E
Nx Ny
+
Z +
C
+
01x1
0 01xk
For this simple case, (I [[m+1] M) always invertible so long f T * from Lemma 3. Or we can also restrict - T| < to ensure that the spectral norm of M is strictly smaller than 1. We sample from this SEM by first sampling all of the exogenous variables Z, Nx, Ny and then
NNNxNy solving the above system for each sample of X, Y via the reduced form Lemma
motivating example
Cyclic SEMs were first discussed in the econometrics literature [71] to model various observational phenomena, and often solved via 2SLS based IV regression [22] since it is computationally less costly compared to solving the entire system [27]. A classic example from economics [72, 73] is that of supply and demand model M where the relation of price of a good with quantity Q of demand can be thought of as a cyclic feed-back loop where producers adjust their price in response to demand of the good and consumers change their demand in response to price of good. contrast, change in consumer tastes or preferences would be an exogenous change on the demand curve and can therefore used as an IV Z.
consumer demand:
= T - No.
producer price:
p P= f- Np.
Where scalars f T are such that f < -
so that the system converges to an equilibrium. We say that the measurements made for P and Q are at the equilibrium state of the market! with zero mean measurement noise Np, NQ respectively.
In fact, such feed back model of supply and demand was initially developed to understand the irregular fluctuations of prices/quantities that observed in some markets when equilibrium [72].
18

---

Mitigating simultaneity bias for causal effect estimation. If we now want to estimate the effect con of demand on price f. standard regression will produce biased estimate JERM f (ov().N)))
())
because of the simultaneity causing Q and Np to be correlated (to this, substitute model of into the model of Q). We can now use IV regression to get an unbiased estimate of the effect of demand on price in the market as Jiv f.
Mitigating spurious correlations for robust prediction. Similarly, if the producer wants to predict the effect on demand if price is changed (i.e. intervened on), naive ERM will not be a good choice because it will also capture the spurious correlation from Q + P. We therefore use three-stage-Ieast squares (3SLS) [74, 27] (or similar methods) estimate the ATE T3LLS E [Q P
where we use the first two stages to estimate JiV followed by ERM to regress from the residuals Np = = pont Q to Q in the third stage.
Implications for independence of causal mechanisms
Here we clarify how the equilibrium assumption/nteerprcaainn of cyclic SEMs the classic independent causal mechanism (ICM) principle [4]. Note that our SEM formulation in Eq. (1) is direct instantiation of the ICM principle described by Peters et al. [4]. The two equations represent the autonomous mechanisms, and their independence is captured by the mutual independence of the exogenous noise terms Nx, .y. The simultaneity in our model not violation of ICM, but rather the equilibrium state resulting from the interaction of these two independent mechanisms. Assuming the existence this equilibrium is a statement about the scope of systems under analysis, and not about the nature of the mechanisms themselves. Indeed, surgically changing r to some r', for example, does not in itself alter and vice versa. And precisely because of the ICM, this may may make the system unstable depending on the nature of 7'. Nevertheless, in setting, Proposition (Appendix F.2) shows that soft interventions induced by mntcome-invariane DA always stable.
odds with
19

---

C IV Regression Supplement
Two-stage estimators. Minimizing the risk in Eq. (5) is known two-stage IV regression. Another two-stage IV regression approach that we use in our theoretical results is to minimize the risk [8, 15]
(h) = 2h || (Y Z
(x( -]
This can be shown to lower-bound (hence the subscript LB) the risk in Eq. (5) under squared loss [8].
Riv (h) E
| E (x() 2
E
[[ - E|y z + [EL
Z] (X) Z)) 2))
(Adding and subtracting E|Y ZJ.)
E
||Y-
211
+ E |E
Z] (|)(X) 1
(Expand squared norm.)
+ 21
(V
E[Y 2))² (
[[((X) 12)],
|| - E|Y
111
+ E
||
[ - (x) _
(12)
E
|E|Y z (X)
11
+ E (Y E 21) z z1]
(Tower rule, scalar Y.)
=E
|E z] - E (X))
1
+ [arr Y Z = RiV (h) + E Var VYr(Y Z)],
(13)
where Eq. (13) follows from the definition of conditional variance and we get Eq. (12) by setting the cross term to zero since
E
E[
Z)) (E TEE
2 ([(()) 21)
E
(Y B[y 21)
[ Z - B[A(X) Z z
(Tower rule.)
E E (Y B Z)
7 ( 2 -E((X) (T)
(14)
E (E[Y Z] Y 2 (E / - ([((X) 1)
10 (E - {[( ~ 0,
where Eq. (14) follows from the "taking out what is known" rule, i.c..
E ( B] = ((B)E[A B).
(15)
The IV regression in colored-MNIST" experiment uses the popular generalized methods of moments (GMM) [75-77], or equivalently the conditional moment restriction (CMR) [8] framework which tries directly solve for the fact that in Eq. (1) with scalar Y
Generalized method of moments.
E [<| Z = E Y f(X) Z] = 0,
which holds as a direct consequence of un-confoundedness of Z. For any g Z - R, it then follows
E [(Y f(X)) [[ = 0.
The GMM-IV estimate of therefore tries to enforce this condition [75-77] by minimizing the risk
Σ on [(Y h(x)) (x(2) = | - A(x)) q(2)]]]
AV (h)
where q(-) R represents vector form of the set of arbitrary real-valued functions qt. A more general form of the above GMM based IV risk is to weight the norm by some SPD W [78, 75, 76]
D R
(h) = E (Y - h(X)) {
which gives the most statistically efficient estimator, minimizing the asymptotic variance, for W Σう [78, 75, 76]. We use the same for our -NIITT experiments, together with the identity function (Z)) = Z. This gives us the final loss of the form
(h)
20 Z (Y ((x)))ll
CMOSEE
20

---

And the empirical version of which can be written as follows
RIy (h) == (g 15-hi (--n()) 22' 9 - h (x))
(16)
GMM
where for dataset samples (x,, Vc, 2i) E D, we construct the vector y == Lyo.... .. Va]]' matri- ces X = [x]..... ,x,]__ 2 100 zn] T with pseudo-inverse 2 and define h ((x) = (M(x)),..) .. h(xx)]].
21

---

D IVL Regression Supplement
Closed form solution in the linear case. The following result gives us a way compute closed- form solution to the IVLa regression problem in the linear Gaussian case. An empirical version of this is used for our linear experiments.
Proposition (lVLa, closed form solution). For SEM Di in Example 1, ha IVL,
is the closed form
linear OLS solution between
X' = ax + bE Z},
Y' = aY bB Z).
where
va,
b: v1+a-va
Proof. See Appendix F.1 for the proof.
For the empirical version of Proposition we fit a closed-form OLS regressor between
X' Vax (VT+a- where 2, Ż are as defined in Eq. (16).
Y" VaY + (vI+a Va)22iy,
Choice of regularization parameter. We try the following approaches to select the parameter
Cross validation (CV), any variation thereof. We specifically the following two in experiments; (i) vanilla CV with 20% samples held-out for validation (ii) level cross validation (LCV) for when Z is discrete, where hold-out data corresponding 20% of the levels of Z for validation.
Confounder correction (CC), where in linear setting we follow an approach similar [18] by estimating the length of the true solution from the observational data D. We then chose such that the length of ip , is closest to the estimated length of the ground truth solution.
22

---

0.5
ICER
avaeaee DA+IVIS
DA JTVE2
avecaee DASIVIS
DA+FV
0.0
LEVL
6
(a) 7 K E [0,
(b) 7, K
[10-8 10
(c) E [10 101.3),
Figure 6: Simulation of the linear Gaussian SEM of Example 2 with the same setting Fig. 4, but T f sampled uniformly over a unit sphere, representing a cyclic structure. Each data-point represents the average nCER over 25 trials with 95% CI.
0.5
0.4
DA
ERM
DA
DA DIVE
DA+IVE
0.0
0.2
0.4
05
08
10.10
10
10
( 1, E [0.
(b) 7, 1, [10-5, 10
(c) E [10-2.5 100,
Figure 7: Same experiment as Fig. 4, but with I constructed by randomly selecting each basis of (f with probability of 2/3, simulating the effect of knowing only symmetries of Each data-point represents the average nCER over 25 trials with 95% CI.
E Experiment Supplement
For the methods that use stochastic gradient descent (SGD), we use a learning rate of 0.01, batch size of 256 for 16 epochs. For baselines that require discrete domains/environments, discretize each dimension of G into bins. Higher discretization bins renders most baselines ineffective since each domain/environment rarely has than 1 sample. To keep the comparison fair, however, we also discretize G for IVL, regression when using LCV. For the colored MNIST experiment, all CV implementations including baselines use 5-folds for random search over an exponentially distributed regularization parameter with rate parameter of 1, Same is the case for simulation and optical device experiments, except that DA+IVL methods
uniformly
methoss use log-uniform distributed regularization parameter over [10-4, 1]. Since RICE [56] grows the dataset size by augmenting each sample times, we provide 1 1/T sub-sample
the original data for fair comparison. Similarly, the causal regularization method by Kania and Wit [12] expects two datasets, perturbed and an un-perturbed one, which we substitute with 1/2 augmented data and 1/2 original data respectively.
E.1 Simulation experiment
For the parameter sweep experiments of Fig. 4, generate treatment of dimension 32, but for the OOD baseline comparison experiment in Fig. we use m = 16. Furthermore, for the OOD baseline comparison experiment in Fig. 5, randomly pick each basis of mull(f) with probability 1/3 to construct I (i.e., we know only some, but not all symmetries of f).
We also provide additional linear simulation experiment results in Figs. and 7-the former simulates a cyclic structure with non-zero т, and the later simulates a case where only some, but not all symmetries of f are known. The results of both consistent with original experiment in Fig. 4.
23

---

Table 2: nCER one standard error (SE) across the optical-device datasets for various choices of DA. Bold and italic denote the lowest and second-lowest average nCER, respectively. Superscripts and indicate a significant improvement over ERM or both ERM and DA+ERM, respectively, beyond margin of SE. Lastly. indicates that the method was too expensive for the value to be computed.
Method
rotate > hilip > vfiip randon-permutation
gaussian-noise
all
ERM
0.827 = 0.079
0.827 + 0.079
0.827 + 0.079
0.823 + 0.083
DA+ERM
0.617 0.085*
0.513 + 0.082
0.707 0.090
0.513 + 0.075*
DA+IVL
0.623 + 0.087"
0.540 + 0.085'
0.641 + 0.092"
0.533 + 0.083'
DA+IVL
0.619 + 0.087*
0.534 + 0.082"
0.662 0.091
0.574 + 0.087*
DA+IVLS
0.623 = 0.085'
0.527 0.082*
0.639 = 0.076
0.509 1.509:000 0.078
DA+IV
0.689 + 0.065*
1.973=0.00
0.955 0.011
0.640 + 0.083*
IRM
0.972 + 0.010
0.960
0.970 = 0.009
0.953 + 0.018
ICP
0.544 0.019
0.527 0.012*
0,646 + 0.054
DRO
0.975 = 0.005
0.959
0.981 0.003
0.952 + 0.014
RICE
0.966 = 0.014
0.960
0.005
0.959 + 0.016
V-REx
0.962 + 0.024
0.957
0.979 0.005
0.925 + 0.037
MM-REx
0.978 = 0.013
+000000000
1,000 + 0.000
0.000
4 Janzing '19
0.821 = 0.081
0.821 0.081
0.821 + 0.081
0.817
f2 Janzing '19
0.823 = 0.076
0.823 + 0.076
Kania, Wit '23
0.652 + 0.084*
0.559 + 0.084*
0.727 + 0.088
0.543 0.080
E.2 Optical device experiment
In the simulation and optical device experiments, we linear function h(.) = E R" for squared loss in all of our risk metrics. For IVLa regression, from Appendix D. We also use a closed-form solution for ERM, DA+ERM and DA+IV (2SLS) baselines. The rest of the baselines (other than ICP) use SGD.
the closed-form OLS solution
In Tab. 2, we report further experiments on the optical device dataset with various DA choices. The findings continue confirm our main hypothesis: DA+IVL dominates DA+ERM, which itself dominates ERM. We never observe an opposite trend with statistical significance.
E.3 Colored-MNIST experiment
In the colored MNIST experiment, use the same 3-layer neural network (NN) architecture for h across all methods comprising of a fully-connected input layer of input dimension m, hidden layer of input/output dimension 256 and output classification layer with Sigmoid function. Each layer separated by an intermediary rectified linear unit activation function. For the risk, we use the empirical version of the GMM based risk from Eq. (16).
Colored-MNIST as cyclic SEM-From invariant prediction estimating causal effects
colored image
true label
MNIST image
color
noisy label
Figure 8: The data generation DAG for MONIST as discussed by the original authors [4]]. They aim learn predictor X' y such that it is invariant to changes P X We argue that this DAG view of Golored-MNIST does not make obvious how the true labeling function f(x is related to the ATE do X: - {) X = x]. which we believe because it is virtually equivalent to the reduced form of our structural form presented in Fig. 9.
24

---

colored image
colored image
MNIST image
X
true label
Nx
true label
flip Y?
Nc
NY
color
noisy label
color
noisy label
flip Y?
Graph generating colored- -NNIST data. (b) Augmented graph exogenous variables explicitly shown.
Figure 9: A cyclic SEM perspective of the colored-MNIST data-an MNIST image Nx is assigned color C to produce a colored-MNIST image X. This is then passed through the ground-truth labeling function f to produce the true label Y. We flip this with probability 0.25 to produce the observed label Y, which in turn is flipped with probability (at train time E {0.1, 0.2) and 0.9 at test time) to produce the color C. These assignments are iteratively applied for any joint sample of the exogenous variables Nx, Ny, No starting at arbitrary values of endogenous variables until convergence the unique stationary point X, Y, C' (and Y).
In this section give cyclic SEM perspective of the colored-M experiment from [41]. The task binary classification of colored images X from the MNIST dataset into low digits (y = 0 for digits from 0 to 4) and high digits (y for digits from 5 to 9). The difficulty of the task arises from there being a higher spurious correlation between the color of the images (c = 0 for blue and c = for green) and (noisy) labels Y compared to the correlation between the digits in the image and the label.
Consider the following cyclic SEM in Fig. 9.
nx Px By B(0.25),, ne Bee)
sample all exogenous variables apply color C' to the image generate ground-truth label with true labeling function
= color(C,,xx)
Ý = s(X)
Y = xor Y (Y,x))
flip the label with probability 0.25
C = xor(Y,nc
generate color by flipping Y with probability e,
where we first randomly sample an un-colored MNIST image nx, and some Bernoulli distributed label noise ny B(0.25) and color noise Th B(e) which is different for each environment
{0.1, 0.2}. Then for some initial arbitrary values Xo, Po- yo and co respectively for the observed colored image X, the ground-trute -tuth label Y, the observed noisy label Y and the image color C, we iteratively apply the following assignments from the SEM
X₁ color(c-11. nx)
apply color C to the image
0: = f 1xx-1)
generate ground-trum label with true labeling function flip the label with probability 0.25
3: xor cor(Wi11,1 n
= xor( nc)
generate color by flipping Y with probability e,
until they converge while keeping all sampled exogenous variables my fixed. Is straightfor- ward show that this SEM will converge after a maximum of f to the color of the image C. Furthermore, this stationary-point will be uniquely determined by our exogenous samples nx, nl, nc. And this how we generate one sample (x, y) for our colored-MNIST experiment. We repeat this process to generate sample (x, y) for each of samples nx, my,, nc.
5 iterations" due to the invariance
Note that the ground-truth labeling function f can only correctly predict the labels 75% of the time. At test time flip the correlation between the label Y and the image color C by setting e 0.9. Also, the above cyclic SEM for colored-MNIS" produces the same distribution for (X, Y) as [41].
12 Following the mechanisms (same for ÿs).
+ D2 +
Xs. we see that (X4, y4, C4) = (Xs, Co)
25

---

The above cyclic SEM perspective of golored-MNIS, is interesting because makes it clear that Gelored-MNIST is essentially causal effect estimation task. Specifically, we can estimate the true labeling function f by estimating the ATE Io [Y
X x] since
X x x] (= xor (fr(s(()) Ny. = x]; (Ny II X (Definition of xor.)
[xor( (xxr(f(x); Nr)),
[f(x) + Ny 2f(x))
fxx + Th [Ny]] 2f(x) [Ny], = (1 - -1 [Ny]) + [Ny] 0x5f(()) + 0.25.
(Ny ~ B(0.25).)
Because this is binary classification task, we have
round [=m X x=
x)) xxx])=
f(x).
This is in contrast to the original DAG perspective of colored-MNIST shown in Fig. 8, where the connection to the estimation of the causal mechanism f immediately obvious. We argue that this is because the DAG in Fig. 8 is virtually equivalent to the reduced form of our structural form presented in Fig. 9.
26

---

Proofs
F.1 Proof of Proposition 1-IVL regression closed form solution in the linear case
Proposition (IVLa closed form solution). For SEM M in Example h is the closed form linear OLS solution between
X' ax PE Z),
Y' = aY + bB 2].
where
a Va.
b b= vI+ va.
Proof.
The OLS solution for (x', Y") minimizes the following ERM risk
Y hT x- x']
E
||aa bEly 2 h (aX bE|X |
(Substitute in definitions of X', Y".)
E [la((Y-- - x'x) + b
Z h E[X 2 [1]]
(Distribute the subtraction.)
d²B diB[1l - hT xx1x + E
| YY / h E[X -
(Expand squared norm.)
(Y - h x) (E(y) Z EE 2
+ 2abE
(17)
First we note that from the definitions of a, we have
==
+ 2ob 2ab
(VI + - (01+a-vaa= +a)=+2/aa -aaa-aaa++(((( + va)= =1
(18)
Now we evaluate the cross term in Eq. (17)
E
(Y x) (E[Y Z h B[X Z])
B
E
B [Y h x)) ( 2 h E[ Z =]]
(Law of iterated expectation.)
E B [(r- h x)
T
7 [B Z h h 17)
(Taking out what is known; Eq. (15).)
LE (ELY / h TEEX E|X 21) (ELY 2 h EEI 2T)
E [| 2] h E 2
Substituting this back in Eq. (17) we get
E Y h x' x'2
aF Y h x
(b² 2ab) E E Z] h
=al Y h x |2]
|E 2J h TE =1I"].
(From Eq. (18).)
= =aR (h) + RVV (h) E[Var Z)),
(From Eq. (13).)
RIVL (h) - E[Var(> 2))
27

---

F.2 Proof of Proposition Existennce of an interventional distribution given DA
Proposition (unique stationary interventional distribution). In SEM 2 from Eq. (9), given any G.C.Nx,NNy' for all (No. E X x y the unique limits
(g. ~ Pa lim x; lim T -y--1: c, nx);
10000
t-
= lim yi lim (x <(c) + ny
1-300 t-o0
exist, then in 21; do(r :- gr) the unique limits
x (:do(r==g))) = lim X
(,do(T==m7)) lim gr yc-1
A,do(T==87) ,c,,x) = =gx",
t000
t0000
(ido(r=))) lim Yr
a.do(r==r)) lim f (x-1 x + (ccc ny =
also exist.
Proof. First we try to show that
A yr:
(19)
For the base case, we have by construction
d = yo yo
For the step case, assuming that (;do(r=g7) -i, we have 13
A;do(r==r)) 1-2
4,do(T==7)) RT) + +e(c)= + ny,
xt+1
(g Yi ( .c,ux) ,c, + <(c() + nyy
f(T Y:
( , .,...)) + {((c) + my:
(Invariance of f to g.)
= =f(T(y]] C, , c(c) + +y,
(Assumption y:
A:do(T: gr) = =y .))
= -x(x)) c(c) +y,
y
Hence, we have shown that Eq. (19) holds for all even t. For odd 6, we simply replace t = 0 with t the base case
Hrdo(r=gr)) = ( )ido(r=
y1
E(() + ny,
=x(xx) + +(c)) ny, = =f
(Definitions A:do(r:=g)) X0 xx0
X
We have now finally shown that Eq. (19) holds for all 0. Next, it is now relatively straightforward to show that for any > have
A:do(r:=gT) gr yt-1 ,c,,),
Agdo(r=gr))
gr cx
(Follows from Eq. (19).)
gx
(20)
Finally, by applying limit / to both sides of Eq. (19) and Eq. (20), we get
y Ando(r==g))
= lim yt
A,do(r:-g1 gr)
lim y: y
1
(
Sido(r==g)) lim x 10
A:do(T:=gT)
lim gx 1000
g lim x
gx
(21)
1000
where the limit can be moved past g in Eq. (21) because g assumed continuous in its domain.
Note that here the step size for proof by induction would be At 2 since precedes y242. Similar is the case for well.
28

---

F.3 Proof of Theorem 1 1-Robust prediction with IVL regression
Theorem (robust prediction with IVL regression). For SEM M in Example 1, the the following holds:
(2: (h), Pa pa= { <c" * =1+1 (
tR IVv, E arggin max ÇEP,
RERM
Proof. Write X in terms of the exogenous variables C, N,,NN,N, using the reduced form from Lemma 3 as
X = x=
(22)
where for readability we represent
T E
Nx
7 = A Z,
Č = M
C,
Ñ = a M
Ny
with
M M M M
M Mix1
Im
T
M
f
1
Now, we start by writing the ERM objective under the intervention do (TT() = 5) as
no (T" ()==)) (h)
RERM
= Mod (T ():=c)
- xx]]]]]
"T ())=c)
| (f - h) (2 (2+ Č Nv) )|]
(Y' structural form & Eq. (22).)
W (-)=-C)
( - h) T) (Mmxm + Ö+ N) 11]
(2 & intervention definition.)
5 - h)T (0)(((++)))) N + (f (h) M M//Mmm <l(j
(T ())=c)
Mo (r (-)=-)
||5 + (f h)) T ( + N)
+ +h² <[(]] (Define hT = hf- "Mmxm.)
do [ ())=c)
s (f ([-h) h) Č + 19 + do ( (==))
(Follows from exogeneity of Ç under intervention, + cross term zeros-out.)
E do (T" ())= 0m)
||| h x do ( hn c <1
(23)
W. (+) 0m)
||| TTx x ["]
h1' <||
Dt
(-)=0.))
Y T ["]
+ tur W'''T hh'''s)) s).
do (m)=m))
||| h T x|] tr(h' hchhh)
(24)
Now, note that the maximum of the trace term over Ç E Pa gives
tr th SC (r(TTTCCThT) h)),
max CEP.
+1 1
tr TT [T 2zz
( 11)
(Linearity trace and definition P...)
1
+1
[tr(h zz Ph')].
(Linearity of expectation.)
a
1
+1
r 2)).
(Cyclic property of trace.)
a
29

---

ERR
| T z 1
a
1
1
E
(f h) Mmm I 2 _
(Substitute in definition of h' T)
a
(f h) TD)22 =11
a E
+1
(Definition of Z.)
We now substitute this in while maximizing both sides of Eq. (24) over interventions Ç E Pa as
MT (0==0)))
max RRRM CEP
(h)
do (-)=0m)
Y
1 - + max
cr(h"TcTTn
<c
(First term does not have Ç.)
CEPa
( (m)=0m ||| hT x x|]
+ 1 +
Edh ([(r- --) 2 =19
1 ERR
h)-h) T 2 =1/].
Y hT
(Inverse step of Eq. (23).)
α 1 ER
h) T EEX 21 -
T
Y h T
(From conditional exp. of Eq. (22).)
CN 1
E
Y h x
E
E
f X z
h EEXX Z [|] (Linearity of expectation.)
α 1
x h T X
27
ER
E| 2
E[X
|
(Inverse step of Eq. (23).)
R (h) + (R h
E[Var(Y (Y) 2))),
(From Eq. (13).)
AiVLL (h)
E Var (Y 2)).
30

---

F.4
Proof of Theorem 2 -auual estimation with IVL regression
Theorem 2 (causal estimation with IVL regression). In SEM M of Example 1, for a < 00, we have
CE [ivv...) S
CEEgg
hERM
equality iff
E
[
2] Las. E0T0 [X E.
21 Proof. For IVL." we have from Proposition
2
f xx
R
X'X
X
=
Note that we have
E X'y
E X'aaY bElY (1)
E ('a + bE X E Z)) '
E X'aa + bf E|X Z T]:
(By definition Z 11. {.)
E X'(af' X + a + bf E[X 21) "]
X'(FTX' x' + aε) 7.
(Substituting in X' == aX P ZJ.
xxx++ f f+ f
X'x'
p aE [X f dZE [xE]] ]]
X'x'T
(Z &, therefore E [X's]]] = aE T])
f OF Xg T]
(25)
We also see that
E X' X'X
(ax + bE[X ()) 6 =
(ax + 6z) (ax + 62) |
(Set 2 = [[[ | [] for brevity.)
[' | Be [2 abE
X2T
abE 2x
L X (12 + 2ab) Ez
(Because E X²T =E =x,))
= =F XX + Σz
(26)
where we substituted Eq. (18) in Fq. (26).
Finally, now have
hiv - -
E x'
E X
2
E
18 x
f f alE (XET) -
(Substituting in Eq. (25).)
- aE X
E[x E[xe_] -
>>
31

---

aR x
x
+ 22)
E [ "1
(Substituting in Eq. (26).)
S
s DS
[[x
(Using Lemma 2.)
1STS
m + D
s B-
(S is invertible.)
TS
D
Im
S [
(Switch to & norm.)
a
S H [ II?
(27)
ss S.-SSSS [
(Substituting I = SS
S ------ S- STS
(Back to weighted norm.)
[[XX E X
(Substituting Σ
E [XXT] = STS.
E [ ] f 2
E
- E
(Adding and subtracting f.)
E
[[XX
E
Xx
f E T] - f
(Substitute I = [xx [=E[xxT"'xxxxX
E ( X + 5) '] - f | ||2
[[
(Linearity of expectation.)
E ||XX
- E
XY
(Substituting Y f X +
100 PERM
(Closed form ERM solution.)
where inequality Eq. (27) holds because D is non-negative diagonal. Furthermore, inequality Eq. (27) only holds with equality iff S- E X is in the kernel of D. Or equivalently, iff E [ is in the kernel of S DS
z. which from Lemma is true iff
T X
Z
E
X
g
32

---

E55
Proof of Theorem 3 CCusal estimation with DA+ERM
Theorem (causal estimation with DA+ERM). For SEM 2 in Example the following holds:
CER
[6a ((DAA+ERM) v CERR (bgRM):
equality iff E= G] Las. E x E).
Proof.
We have
+ ERM
"
- " (f
(Structural eq. of Y.)
Σ
+5) (F=E(cx)Gx)""""(ccxpp"
'
(Using G-invariance of f.)
E
x
b (x++))++)))
[[ E
(Let:== G - TTG.
Ex
[B[xx]] + (O0T]) B [X
(Using & 1T X,
s S DS) [X( ITT
(Lemma 2.)
s S- D) -s S [XE]] [
(S, S invertible.)
STS
= SS [In + D) 's E XXe]
(Switch to & norm.)
(Im + +D -' D) EE [
< S "E
(28)
|
(Substitute in I.. SS
- ] STS
(Back to weighted norm.)
[[xx'] []
(Substitute in Σ = E [ ST STS.)
f [+E[Xx x
(Add and subtract f.)
[[xX]]
EE x f++ E
-f
(Use I, = X "xxx"]]
x
[[XX
x (f x + +)) 7] -f
(Linearity of expectation.)
[[XX ] E
(Structural eq. of Y.)
x
FRM
f
(ERM closed form solution.)
where inequality Eq. (28) holds because D is non-negative diagonal. Furthermore, inequality Eq. (28) only holds with equality iff S E [X is in the kernel of D. Or equivalently. iff E
[ in the
kernel of S DS = Σ & which from Lemma is true iff E [GX G]
T
E
X
E a.s.
33

---

F.6
Miscellaneous supporting lemmas
Lemma 1 (Gaussian conditional orthogonality lemma). Let X, X.Y. 2 R be zero-mean jointly
Gaussian random vectors with covariance matrices Σx = E|XX" Σg = E|ZZ"), and cross-
covariance Σ = EEYZ' Define the conditional expectation
B|y 2 = (Bl (B[zz") (
B[ZYT]] = Ey,
Then the following are equivalent:
X $LEB LLYY Z = 0 a.s.
Σx Dy, =0
Proof. Since X, Y. Z jointly Gaussian, E|y Z MZ with M = Ey.2
The scalar
random variable
S S=XEE
Z = X MZ
is Gaussian with mean zero. Hence,
S ==0 0 a.s.
Var 0.
Compute the variance:
Var(S) = [[5 = E [[[X' M2) = BlZ M XX X MZ]
Using independence and zero-mean assumptions,
Var(S) tr x M)
x
Since covariance matrices are positive semidefinite, Var (S) 0
Σ MP 1/2 = Ex ExME, = 0.
Substituting M Σy.z Σz gives
Ex Zy =0,
completing the proof.
Lemma 2 (SPD and PSD simultaneous diagonalization via congruence). For any matrices
A > 0, B > 0, there exists an invertible S E RRax and non-negative diagonal D E RRxx such that
A S
S DS.
Proof. This similar to Theorem 7.6.4 in [79, p. 465] for two SPD matrices. We proceed similarly: Since A is SPD, admits a unique SPD square root 1/2, Define
C = A BA 11/2
which is SPD. By the spectral theorem, there exists an orthogonal matrix U such that
C = U DU,
where D is diagonal with non-negative entries (the eigenvalues of C). Set
: 1A
Then
S S - 42 = 1/211A1/2 = A,
and
S SDDS A D 1/ 2 B.
Since A 1/2 and U are invertible, S is invertible, completing the proof.
34

---

Lemma 3 (solvability of simultaneous SEM). The SEM D in Example is solvable iff i + 1, in which case the following solution defines the reduced form of the SEM.
x Y
Im f
r
T 6
Nx
2+
C
+
01xk
Ny
Similarly, SEM 2 in Example 2 solves for TTT
+ k=
Proof. We re-state the SEM Wt in the following block form
X Y
=
O
m X m
f
x
0
T
Y
X
+
11k
Z
-
T
E
C+
Ny
Nx
I
X
r
T
[x
=
01xk
Z
+
6
C+a.
[N
solving for (X,Y) Y) involves inverting the block matrix on the LHS. The result immediately follows from Proposition 2.8.7 in [80, p. 108], via the Schur complement formula for block matrix inversion.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims stated in the abstract are explicitly enumerated in Sec. 1, each referencing the section of the paper that contains the respective contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of our method in Secs. 4 and 7. We also explicitly state assumptions made for theoretical results Secs. 3 and 4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
36

---

Answer: [Yes]
Justification: We provide the full set of assumptions and correct proofs for each theoretical result. Observations, assumptions, examples, lemmas, theorems, all appropriately referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide the details of our experimental settings, including algorithm used and its implementation details thyper-parameters, network architecture, etc.) in Appendix All baseline methods are referenced appropriately in Sec. 6, and their parameterization (hyper- papper-prameters. network architecture, etc.) is discussed Appendix E. We also provide code for our experiments for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
37

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we provide the code including README md file with necessary instruc- tions on how to run and reproduce the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Caaaas//ddSSSubssseeee wwwsssnoollicoooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipp..c..a wttpccccppppccceeeeeccccccccclct hoppulic/guudes..../ CCodeSubeeieeeeeee
/ttiei/uuuieeeeeeSuuuuuiiiiii betbbbisssiiie wwteiiissiioooooooooooooiooo wonooniyoyyllloooooooooooiiiimll for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we provide sufficient details for our experimental settings in Sec. 6 for the readers to understand the results and additional details in Appendix E well.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide a 95% confidence interval (CI) for our stand-alone simulation experiments in Sec. 6.1 and Appendix 1, inter-quartile ranges (IQR) in comparative analysis with other DG and causal regularization baselines Fig. 5 and standard error (SE) with additional optical-device experiments in Appendix E.
Guidelines:
38

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We briefly mention the hardware used generate experimental results in the README. md file with the supplemental code. However, the results should be hardware- agnostic.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : ://e////nuueeee /neuriius....// cc wccppuulc////eceeeceeeecceaaattt hetpaiiceehcccccuueeeeee....e.ee ines?
Answer: [Yes]
Justification: The study involved no human subjects and all data sources used are publicly available.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
39

---

Answer: [NA]
Justification: The paper discusses data-augmentation, which is fairly ubiquitous and largely application agnostic technique.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and credit all assets used, including baseline models and the datasets
used.
Guidelines:
The answer NA means that the paper does not existing assets.
40

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pan re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code reproduce our experimental results is publicly released https: http//gtttub..... com www.//aaaaaaaa//aaacooooo cattaaarrccaaalaaaa augmentation, along with appro- priate documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowd-sourcing or human subjects were used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
41

---

Justification:
No human subjects were used.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: Core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tpur/eeipp///eeeee 2205 /LLM) for what should or should be described.
42


Reviews:
[{"review_id": "mS4IRKMYAj", "reviewer": "Reviewer_CVxW", "summary": "This work is concerned with estimating causal effects in the presence of unobserved confounders using data augmentation techniques. In particular, the authors consider a setting where we observe only treatment X and outcome Y, with hidden confounder C affecting both, and where the outcome function f is invariant to certain data transformations. The estimand of interest in this setting estimating the causal effect of X on Y. The authors look at a  connection between data augmentation and causal inference through the lens of instrumental variables and soft interventions. The approach consists of three key pieces: (1) showing that outcome-invariant data augmentation is mathematically equivalent to a soft intervention on the treatment generating mechanism (2) they derive IV-like (IVL) regression, which can be seen as relaxing standard instrumental variable methods by dropping the outcome relevance requirement (while maintaining bias reduction properties through a regularized objective combining IV and ERM losses), and (3) showing that the data augmentation parameters can serve as IV-like variables, leading to a worst-case augmentation formulation that further reduces confounding bias. The authors provide theoretical guarantees in the linear Gaussian case showing bias reduction under appropriate conditions, and propose practical algorithms for both DA+ERM and DA+IVL estimation. Experiments on simulated linear systems, optical device data, and Colored-MNIST demonstrate consistent bias reduction, with DA+IVL generally outperforming standard DA alone, and modest improvements on complex real-world datasets.", "questions": "While I don't think it's necessary, per se, for publication, it would be interesting to hear if the authors have thoughts or could provide a brief discussion within the paper around examining some of these assumptions required to have a valid data augmentation. In particular, are there sensitivity analyses that could be performed so that practitioners can bound the range of outcomes they would expect under violations of assumptions in model misspecification?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n* I found this to overall be a very original and compelling idea. The authors do a really nice job of describing the problem setting, and providing the intuition for the approach before introducing it. \n* I really liked the connection between data augmentation and soft interventions. This was surprising to me and I found it very interesting and a very nice addition to the causal literature. \n* The proposed method itself is both intuitive and relatively simple. Further, while finding the \"right\" set of data augmentations is non-trivial, the idea itself and it's underlying identification requirements are easily digested. \n* The provided empirical evidence shows consistent improvements over a set of candidate alternatives. \n\nWeaknesses: \n* The main weakness of this paper is much the same as the the issue that underlies a lot of IV application in the exclusion principle. It's not entirely clear to me how hard or easy it would be to find a suitable data augmentation target here for various applications. I think the authors do a nice job of trying to address this by providing very clear criteria for when the proposed method is applicable vs. not. With that being said, I think it's important to acknowledge that essentially most of the applications of this have to lie purely on assumption, which can be difficult for practitioners. \n* The theoretical results provided are pretty restricted in terms of the settings that they consider. I think this is fine because there's a lot of task novelty here. It is worth noting that some of the results, in particular Theorem 1, seem to be operating on a very restricted setting in contrast to the framing of the paper which is much more general.", "was_revised": true, "final_justification": "This paper studies an interesting and novel task. The proposed approach is interesting and leverages nice insights tying data augmentation to soft interventions.", "rebuttal": "We would like to extend our sincerest thanks to the reviewer for their thoughtful and encouraging feedback. We are particularly pleased that they found the core idea original and compelling, and appreciated the connection we draw between data augmentation and soft interventions. The reviewer's summary of our work's strengths and weaknesses is fair and very helpful. We will address the identified weaknesses and questions below.\n\n### **On the Practicality of Finding Suitable Data Augmentations and the Untestability of Assumptions**\n\nThis is the reviewer's main concern, and we agree it is a crucial point. The reviewer correctly draws a parallel between the difficulty of finding a suitable data augmentation and the challenge of satisfying the properties for valid Instrumental Variables (IV). Both, as the reviewer notes, must often \"lie purely on assumption.\"\n\nWe believe the most appropriate yardstick for evaluating our framework is indeed that of IVs. Our proposed IV-like (IVL) variables are a direct generalization of IVs, designed to exploit a specific trade-off:\n\n- **Availability vs. Identifiability:** By relaxing the \"outcome relevance\" property of a standard IV, IVLs become easier to find and are more broadly available. The price for this increased availability is that IVL regression can no longer guarantee the identification of the causal effect. However, as we show, it can still be instrumental in reducing confounding bias.\n- **Inherited Limitations:** Consequently, the limitations of IVLs regarding testability and reliance on prior knowledge are the same as those for standard IVs. The validity of an IVL (e.g., exclusion restriction, un-confoundedness) is not empirically testable from observational data alone and must be justified by domain expertise—the same standard applied to IVs for decades.\n- **Outcome invariance:** Outcome-invariant data augmentation (DA), by virtue of equivalence with IVL by construction, also inherits these properties. The outcome-invariance assumption itself now becomes the central, untestable assumption, analogous to exclusion restriction and un-confoundedness. And while we acknowledge that this is a seemingly strong assumption, we point out that it is a fairly common modeling choice in the data augmentation and invariance literature.[1-10]\n- **Perturbing spurious features:** The condition of IVLs and/or DA perturbing spurious features of the treatment $X$ is also not a new limitation introduced by our framework. In fact, this condition is just a re-framing of the exclusion principle in IVs and is also necessary for **identifiability** in IV regression. It doesn’t pop up simply because of our IVL relaxation. Indeed, if an IV does not enter $Y$ via spurious features of $X$, then the respective components of the causal function $f$ cannot be identified. Analogously, if no spurious features are perturbed, then IVLs cannot help mitigate confounding bias. The same holds for outcome-invariant DA by virtue of their equivalence with IVL.\n\n$\\overbrace{\n\\underbrace{\\text{outcome-invariance}}\\_{\\text{popular model for DA}}\n+\n\\underbrace{\\text{spurious targets}\n}\\_{\\text{benign failure if violated}}\n}^{\\text{un-testable DA assumptions}} \\iff \\overbrace{\\text{un-confoundedness}+\\text{exclusion}}^{\\text{un-testable IV/IVL assumptions}}$\n\nUltimately, our work does not propose a \"free lunch.\" Instead, it formalizes a trade-off that may be quite practical in many settings where valid IVs are scarce, but plausible outcome-invariances (i.e., data augmentations) are abundant. Consequently, we believe it would be inappropriate to expect IVLs and/or outcome-invariant DA not to come with the same limitations as IVs other than the one trade-off made — availability vs. identifiability.\n\nIn the camera-ready version, we will make sure to emphasize that these untestabe assumptions are inherent to the difficulty of causal estimation under hidden confounding, and not a limitation of our method per se.\n\n#### **Additional Reflections on untestable validity of IV vs. outcome-invariant DA:** A majority of reviewers voiced concerns over the untestable assumption on the validity of the IV and outcome-invariant DA. While this is a fair and valid criticism, it’s worth pointing out the key distinction between IV and DA. The classic IV relies on premise that the IV induces a “natural” variation on the treatment. The researchers must therefore \"look for” these variables passively and then need to justify why they act as a valid IV. In contrast, DA is typically designed by the researchers who have control over how the transformations are applied to the data. This flexibility can make DA outcome-invariant “by design”. In fact, it’s quite intuitive to distinguish between DA that are outcome-invariant and those that are not. For example, any approaches that use the labeled data to learn DA cannot be outcome-invariant.\n\nIn my opinion, this distinction makes outcome-invariant DA less susceptible to the question of validity compared to the classical IV.    \n\n### **On the Restricted Setting of the Theoretical Results**\n\nThe reviewer notes that our theoretical results, particularly Theorem 1, operate in a more restricted setting than the general framing of the paper. We thank the reviewer for their understanding that this is reasonable given other strengths of our work.\n\nWe wish to clarify that the primary reason for Assumptions 1 and 2 is our choice of evaluation metric: the **Relative Squared Error (RSE)** by Janzing.[11] Janzing argues that the structural nature of RSE makes it a more relevant estimation error for causal settings [13], which also motivates our choice. Nevertheless, there is no generally accepted metric for causal estimation since quantifying causal influence is itself a non-trivial problem [13, 14]. So if we instead use the more familiar risk-based estimation error $\\mathbb{E}^{\\mathcal{M}}[ \\ell( f(X), \\hat{h}(X) ) ]$, we can in fact drop Assumptions 1 and 2 altogether from our results.\n\nWe therefore urge the reviewers to see these assumptions as a requirement for a more appropriate measure of causal estimation error rather than a necessary crutch. And we shall explicitly provide the same disclaimer in the camera ready version of our manuscript as well.\n\n### **Question: Examining Violations of DA Assumption(s?)**\n\nWe interpret this question as asking about the robustness of our method to violations of the core **outcome-invariance** assumption. Our paper mainly studies the case of *exact* invariance to make the connection with causal inference more obvious and tractable. And although exact invariance has precedence as a modeling assumption in DA [1-10], we understand the hesitancy to commit to this in practice. And naturally, this opens the door to thinking about *approximate* notions of invariance.[1, 6]\n\nAs such, we have been looking to incorporate the same in our framework. Preliminary results seem to suggest that our main theorems can indeed be extended to this more general setting. We plan to further develop this line of work in a journal version. This allows us to present a more focused and self-contained contribution here, while laying a clear path for future work.\n\nWe hope these clarifications have addressed the reviewer's concerns. We once again thank them for their positive and constructive review, which has helped us to polish our thinking and positioning of this work.\n\n[1]: S. Chen *et al.* *A group-theoretic framework for data augmentation*. JMLR, 2020.\n\n[2]: C. Lyle *et al.* *On the benefits of invariance in neural networks*. arXiv, 2020.\n\n[3]: H. Shao *et al.* *A theory of PAC learnability under transformation invariances*. NeurIPS, 2022.\n\n[4]: A. Fawzi and P. Frossard. *Manitest: Are classifiers really invariant?* BMVC, 2015.\n\n[5]: Y. Dubois *et al.* *Lossy compression for lossless prediction*. NeurIPS, 2021.\n\n[6]: M. Petrache and S. Trivedi. *Approximation-generalization trade-offs under (approximate) group equivariance*. NeurIPS, 2023.\n\n[7]: O. Montasser *et al.* *Transformation-invariant learning and theoretical guarantees for OOD generalization*. NeurIPS, 2024.\n\n[8]: D. Romero and S. Lohit. *Learning partial equivariances from data*. NeurIPS, 2022.\n\n[9]: S. Zhu *et al.* *Understanding the generalization benefit of model invariance from a data perspective*. NeurIPS, 2021.\n\n[10]: S. Wong *et al.* *Understanding data augmentation for classification: When to warp?* DICTA, 2016.\n\n[11]: D. Janzing. *Causal regularization*. NeurIPS, 2019.\n\n[12]: C. Vankadara *et al.* *Interpolation and regularization for causal learning*. NeurIPS, 2022.\n\n[13]: D. Janzing and B. Schölkopf. *Detecting confounding in multivariate linear models via spectral analysis*. J. Causal Inference, 2018.\n\n[14]: D. Janzing *et al.* *Quantifying causal influences*. Ann. Stat., 2013."}, {"review_id": "2mtQv31ssf", "reviewer": "Reviewer_gi4s", "summary": "The paper shows a method that simulates (soft) interventional data provided that the structural equation model satisfies a certain symmetry, which can then be used to reduce bias from confounding or feedback. \nTo this end, the paper introduces a generalization of instrumental variables in which the outcome may be independent of the instrument, but nevertheless useful to mitigate confounding bias.", "questions": "- Not sure it is recommend to overload the term “confounding” even further and subsume even cyclic causal relations under confounding. \n\n- line 141: this sentence is not very clear: \"...have good predictive performance while the IV risk encourages solution search within the subspace where we know f to be present.\", what does it man that f is present?  \n\n- line 280 \"Finally, DA+IVLα was implemented using a closed form linear OLS solution between empirical versions (see Lemma 1) of the following…\" , unclear what \"between empirical versions\" means. \n\n- line 355: \"Finally, when used together with DA, IVL regression can simulate worst-case interventions that are encoded by the DA\" in what sense does regression *simulate* an intervention? \n\n- Is the method fundamentally restricted to additive noise?\n\n- Abstract: “for when treatment randomization\nsources may carry no information about the outcome and the possibility of its use\nfor improving predictive performance across treatment interventions and reducing\nconfounding bias”, this sentence is confusing, \n\n- Table 1 is a bit unclear: The arrow of lower confounding bias points downwards, but worst-case or adversarial DA is a scenario with larger confounding, right?", "limitations": "I don’t see an explicit discussion of limitations.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength: addresses an important problem in way that is novel to the best of my knowledge. I particularly like the observation that IVs can help reducing confounding bias without being relevant to the outcome. The theory is sound. \n\nWeakness: \n\n- It remains unclear how often symmetries of the structural equation are known, although the paper mentions a few cases admittedly. \n\n- It is difficult to fully understand the comparison to other methods because it is not very transparent which role exactly the domain DA parameters play in the other methods and why it is conceptually right to let them play the respective role.  \n\n- The authors could do a better job in providing intuition alongside with theoretical results: why does the soft intervention mitigate bias?", "was_revised": true, "final_justification": "I have been confirmed that this is an interesting paper which should be accepted.", "rebuttal": "We sincerely thank the reviewer for their positive assessment and insightful, constructive feedback. We are encouraged that they found our approach novel, and the core ideas sound and valuable. The reviewer's questions and suggestions for improvement are appreciated and will help us improve the clarity and impact of our work.\n\nWe have structured our response to first address the main weaknesses identified (intuition, comparisons, and practical assumptions), followed by answers to the specific clarification questions.\n\n### **On Weaknesses and General Improvements**\n\n**1. Intuition for Bias Mitigation and Additive Noise Restriction (Weakness 3 & Additive Noise Restriction Question)**\n\nThank you for the suggestion. We agree that a more direct intuitive explanation would be helpful.\n\nThe core intuition behind why the **outcome-invariant data augmentation (DA) induced soft intervention** helps reduce confounding bias is that it injects new randomness into the treatment $X$ that is, by construction, independent of the hidden confounder $C$. This process effectively \"dilutes\" the spurious correlation between $X$ and $Y$ that is mediated by $C$. By training on this augmented data, the model is discouraged from relying on the spurious path and is guided toward learning the true causal relationship, thereby mitigating confounding bias.\n\nAnother, albeit more `technical' intuition is to look at it from the perspective of the data processing inequality $I(GX;C)\\leq I(X;C)$ applied to the Markov chain $C\\rightarrow X\\rightarrow GX$, so that $GX$ now carries less information about the confounder $C$ than the original treatment $X$, unless $GX$ preserves information about $C$ (this is precisely the condition of perturbing spurious features in our linear settings). This perspective also portrays that our method is **not merely limited to additive noise**, and applies to more general transformations $G$.\n\nWe will add a similar explanation at the end of Section 4.1 to better frame our theoretical result and also emphasize that it is more general than our simple additive noise setting would suggest.\n\n**2. Availability of Symmetries and Discussion of Limitations (Weakness 1 & Limitations)**\n\nWe thank the reviewer for this crucial point. We agree that assuming known structural symmetries is a strong requirement and they are not universally available. Our work builds on a key insight inspired by this very challenge:\n\nThe widespread use of an outcome-invariant model for data augmentation in the literature seems to suggest that in many application domains, such as vision, natural language processing, etc., practitioners already possess and leverage this type of symmetry knowledge, even if implicitly.\n\nAnd the motivation behind our work is to show that these existing, domain-justified symmetries can be **repurposed** as a principled tool to mitigate confounding bias.\n\nTherefore, our framework is most valuable in application domains where such data augmentations are already established. More generally, and as with any non-experimental causal method like Instrumental Variables, justifying these symmetries ultimately relies on domain knowledge. This is a fundamental characteristic of the problem space, rather than a unique limitation of our approach.\n\nWe discuss this and other limitations of our method in Appendix E.4, but will be moving that section into the main paper with more explicit discussions of the same for better transparency.\n\nWe also make some additional comments on symmetry availability under Reviewer CVxW's thread that you may find interesting -- see under *\"Additional Reflections on untestable validity of IV vs. outcome-invariant DA\"*.\n\n**3. Clarity of Comparison to Other Methods (Weakness 2)**\n\nWe appreciate the reviewer's call for more transparency here. Our rationale for using DA parameters ($G$) as inputs for baseline domain generalization methods (e.g., V-REx, DRO, IRM, etc.) is to ensure a **fair comparison**. The fundamental challenge we address is reducing confounding bias given **only** observational data $(X, Y)$ and knowledge of some symmetries via valid augmentations. For a fair comparison, all methods must be restricted to this same information. Therefore, we use the DA parameter $G$ as a proxy for the \"environments\" or \"domains\" that these other methods typically require as input. The reason this is still a **conceptually valid comparison** is because by virtue of being domain generalizaiton methods, they are essentially solving a robust loss of a similar form as in Corollary 1. This gives us direct and correct controls to compare our DA+IVL method against.\n\nWe will explicitly state this rationale in Section 6 to make the experimental design clearer.\n\n### **Responses to Specific Questions**\n\n- **On subsuming cyclic relations under \"confounding\"**: We appreciate the nuance. Our use of “confounding bias” is the same as Pearl’s definition of the term [5]\n$\\text{Confounding Bias}\\coloneqq \\mathbb{E}[Y\\mid X] - \\mathbb{E}[Y\\mid\\operatorname{do}(X)]$ .\nWhich is to say, any bias that causes the observational to deviate from the interventional. In our context, this would also include simultaneity. Such a general framing also follows the tradition in econometrics [14, 39] where omitted-variables (i.e. confounders) are lumped together with simultaneity as sources of endogeneity (statistical dependence between the treatment $X$ and the residual $Y - f(X)$). We will add a footnote in Section 2.1 to clarify our use of the term.\n- **Clarification of line 141**: This sentence describes the two parts of our IVL risk. The IV risk term ensures the solution lies in a set that contains the true causal function $f$ (i.e., where \"$f$ is present\") (assuming $\\alpha=0$). Since this set may not be a single point, the ERM risk term acts as a regularizer, selecting a solution from this set that also has good predictive performance. We will rephrase lines 140-142 for better clarity.\n- **Clarification of line 280**: \"Between empirical versions\" was imprecise phrasing. It refers to the transformed variables $X'$ and $Y'$ defined just below in the text. We perform a closed-form OLS regression of the empirical transform $Y'$ on $X'$. We will revise this to be more precise.\n- **Clarification of line 335 (\"regression simulates an intervention\")**: By *simulation* we essentially mean *is equivalent to*:\n    1. Applying DA is equivalent to a **soft intervention** on the treatment mechanism (Observation 1).\n    2. And DA+IVL regression is equivalent to a *worst-case* DA from a given set (Corollary 1).\nIn this sense, the DA+IVL regression procedure finds a solution for a worst-case *intervention scenario* encoded by the DA. We will clarify this two-step logic in Section 4.2.\n- **Confusing sentence in the Abstract**: Thank you for highlighting this. We will revise the sentence for clarity. A proposed revision is: \"We introduce the concept of IV-like (IVL) regression, which leverages sources of treatment randomization even when they are irrelevant to the outcome. We show that this approach can still improve predictive performance across interventions and reduce confounding bias.\"\n- **Arrow in Table 1**: This is a good point to clarify. The arrow indicates that moving down the column leads to methods that yield **lower confounding bias**. \"Worst-case or adversarial DA\" refers to a method that optimizes for a solution robust to the worst possible augmentation. In our setting, this robustness forces the model to ignore spurious correlations, thereby yielding a better causal estimate with *less* bias. We will add a note to the table's caption to make this interpretation explicit.\n\nThank you once again for your thorough and valuable review. We are confident that by incorporating these changes, we can significantly improve the paper.\n\n### **References:**\n\nPlease see the manuscript."}, {"review_id": "y8NLEaDJ5h", "reviewer": "Reviewer_MsXV", "summary": "The work presents a framework for improved ATE estimation in the presence of hidden confounders, when no proper instrumental variables are available. In such a setting the causal effect is not identifiable. The authors propose two complementary approaches to reducing the estimation bias in this setting: regularising the instrumental variable (IV) risk through an additional empirical risk term, and a data augmentation technique relying on applying an outcome-invariant transformation to the data. The authors carry out theoretical analysis of these methods for linear data generating models, and further validate their results through empirical experiments on synthetic and real-world data.", "questions": "1. In the introduction the authors mention a suit of regularisation techniques relying on $\\ell_1$ and $\\ell_2$ regularisation, applicable in the similar non-identifiable setting. How do these methods compare to the presented approach, in terms of the precise settings they target, as well as int terms of empirical results?\n2. Could the authors elaborate on the meaning of the set $\\mathcal{I}_{\\alpha}$ for ICL regression? In particular, what’s the significance/intuitive meaning of the parameter $\\alpha$?\n3. Could the authors provide a more detailed justification for Observation 1? This conclusion does not seem immediate to me. Further, what is the formal definition of $G\\tau$? What is $G\\tau$ in the context of example 2?", "limitations": "As mentioned in the weaknesses section, this work operates in the setting where the causal effects are not identifiable. While the authors are open about this inherent limitation of their method, I think it could be emphasised more that the obtained estimates can be arbitrarily far away from the true causal effects. The work should emphasise more that special care should be taken before applying this method in real-world contexts, particularly as evaluating any causal method using strictly observational data, to gauge the level of resulting error, is infeasible.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The authors provide a comprehensive introduction to the topic of causal effect estimation and instrumental variable regression (section 2), grounding their work in context and making the subsequent sections more easy to follow.\n- The authors evaluate their proposed method empirically, on simulation data which respect their imposed assumptions, but also on real-world data where the assumptions might not necessarily hold.\n\nWeaknesses\n- **Limited applicability:** In their work, the authors consider the problem of causal effect estimation in the presence of hidden confounders, when the causal effect is not identifiable. As in this setting the causal effect cannot be accurately estimated, even in the infinite sample limit, the prevalent approach is that the causal effect should not be estimated, as it can be arbitrarily far away from the truth (depending on the level of confounding). However, the authors propose a method which allows to improve estimation performance in this setting (when no guarantees on identifiability can be made). This raises two significant questions which, I believe, the authors did not sufficiently address in their manuscript: 1) in what real-world applications would that be useful? 2) if there is no use for specific real-world applications, what are the implications of their results for the broader research in the area of causal effect estimation?\n- **Weak theoretical results:** The authors analyse the proposed methods (IVL-regression and outcome-invariant DA) only in the case of two simple linear data generating models. No theoretical insights beyond these two settings are provided. Even in these two linear setting, the provided results indicate only that the proposed methods, in some settings, can reduce the relative squared error (RSE) compared to the baseline of empirical risk minimisation. However, the results do not guarantee in any way that this reduction will be in any way meaningful (i.e. the RSE can still be prohibitively large).  Further, the authors themselves mention that the RSE is a “surrogate” for measuring confounding bias. It would be interesting to see how these results translate into the magnitude of the error on the actual ATE estimate.\n- **Large number of assumptions:** What further limits the applicability and insightfulness of the presented theoretical results is that they rely on numerous implicit and explicit assumptions. For the IVL regression, aside from assumption 1, the variable $Z$ still needs to satisfy properties (i)-(iii) in section 2.3, and to see tangible improvements $\\tilde{Z} \\perp \\tilde{C}$. For DA, the chosen transformation $G$ needs to be outcome-invariant but also perturb $X$ along the spurious latent features originating from C. Finding such a transformation (if one exists) seems non-trivial, and so is verifying that a proposed transformation satisfies these criteria.\n- **Empirical evaluation in the real-world settings:** While the presented method seems to offer advantages in the linear simulation setting agreeing with Examples 1 & 2 (Fig 4), the benefits of the proposed methods are more questionable in the real-world examples (Fig 5). It is not clear whether the method significantly improves upon the baselines, raising further questions about its broader applicability. Further, in the optical device dataset the ground truth label is computed by assuming linearity (which might bias the results). As mentioned above, it would also be interesting to see the results in terms of metrics other than the RSE, which is a _surrogate_ for confounding bias (e.g. error on the ATE estimate).", "was_revised": true, "final_justification": "The reviewers addressed most of my concerns in their rebuttal. Specifically, their comments about the 'broader significance' of their work made me more appreciative of the scientific value of their paper and the theoretical analysis it presents. Similarly, their detailed comments comparing the assumptions they are making in the paper with the standard assumptions made in the IV/IVL literature convinced me that this work can be of interest to the broader causal inference community. Overall, I think that the work can provide a good preliminary exploration into the direction of using DA to mitigate confounding bias (as well as providing an understanding of _why_ DA has a potential to help).\n\nThe additional experimental results that the authors provided during the rebuttal also make the empirical evaluation of the method more robust, showcasing it as a promising solution.\n\nSome (minor) issues still remain and I think should be addressed by the authors in the revised camera-ready submission:\n\n**Changes to the writing**:\n- Discussing broader significance in Section 1.\n- Connecting Assumptions 1 and 2 with the RSE metric, justifying that the equivalent results for the EE metric also hold and do not require these assumptions.\n- Including a more comprehensive section about the limitations of the work, making explicit that the proposed method can still lead to arbitrarily large error on the estimated causal effect (even though it is smaller than ERM error).", "rebuttal": "We thank Reviewer MsXV for their constructive feedback. The reviewer raises crucial points, including about the value of our work in a non-identifiable setting, which we are happy to clarify.\n\n### **Response to Limited Applicability**\n\nOur work's significance is best understood through the following lens:\n\n1. **Practical Value: A Safe, \"No-Regret\" Improvement:** Our primary response is grounded in **Theorem 3**: under the standard DA assumption of outcome-invariance, our approach is guaranteed to perform *no worse than standard ERM*. This provides a \"no-regret\" choice for practitioners. They gain a potential reduction in bias if the data augmentation targets spurious features, but risk nothing compared to the ERM baseline.\n2. **Broader Significance (1): Addressing the Upstream Problem for Robustness:** Reducing confounding bias, even in non-identifiable settings, is not a niche goal but the **upstream problem** for OOD generalization and robustness [4]. A model that fails on shifted distributions often does so because it learned spurious correlations (i.e., confounding). Our work, in line with the goals of causal regularization [10-12], tackles this root cause directly. By formalizing *why* DA can help (as a soft intervention) and *when* (by perturbing spurious features), we provide a principled path toward building inherently more robust models.\n3. **Broader Significance (2): Downstream Implications for:**\n    1. **Partial Identifiability:** Lower confounding bias in the data (e.g., via DA) can result in tighter bounds in partial identification [9].\n    2. **Sensitivity Analysis:** Lower confounding bias means reduced sensitivity to confounding, leading to more reliable conclusions from subsequent sensitivity analyses.\n\nThe value is in repurposing the already existing, cheap tool of DA to provide a **safe and causally-grounded improvement** over the ERM baseline, with direct positive implications for the above downstream tasks. We will clarify and explicitly mention applicability in section 1 of our draft.\n\n### Theoretical Results, Assumptions, and Metrics\n\nThe reviewer's concerns about the theoretical results, assumptions, and the RSE metric are interconnected, and we address them together.\n\n- **RSE is a direct ATE Error Metric.** There seems to be a slight misunderstanding here. The Relative Squared Error (RSE) is not a surrogate for the average treatment effect (ATE) error; it **is** itself a valid ATE estimation error [33,10]. Furthermore, Janzing argues that its structural nature makes RSE more relevant for causal statements than the more familiar risk based estimation error (EE) $\\mathbb{E}^{\\mathcal{M}}[ \\ell( f(X), \\hat{h}(X) ) ]$ [33,12], which also motivates our choice. The (un-normalized) RSE is simply $\\|\\hat{h}-f\\|^2$, which directly measures the squared error of the estimate $\\hat{h}$ from the true causal function $f$ (which equals the ATE in our model). The normalization is only done to make the metric more interpretable and meaningful for our experiments. We used the term \"surrogate for confounding bias\" because any choice of estimation error also contains statistical bias, but we see now how this was unclear. We will revise the phrasing to \"causal estimation error\" in the paper to avoid this confusion.\n- **Strong Assumptions are for the Stricter Metric.** Our strongest assumptions (Assumptions 1 & 2, regarding joint diagonalizability) are a direct consequence of using the stricter, structural RSE metric. We would like to emphasize that **our core results on bias reduction also hold for the standard risk-based EE, without requiring Assumptions 1 and 2.** We will make this clarification about choice of metric vs. generality of results explicit in the revised manuscript. We believe this directly addresses the concerns about the limited insightfulness of our results and choice of estimation error. Should the reviewer still find it helpful, we are also prepared to add experimental results using EE to the appendix , as our core claims hold for that metric as well.\n- **On the Practicality of Assumptions.** To avoid redundancy, we provide a more detailed discussion on limitation of assumptions in our response to Reviewer CVxW. Please refer to the same.\n\n### Empirical Evaluation\n\n- **Weakness: Benefits on real-world data are questionable.**\nWe respectfully disagree with this characterization. Our results in Fig. 5 show a consistent and often significant improvement.\n    - On the **Optical Device dataset**, our proposed DA+IVL\\textsubscript{CC} is the only method that meaningfully outperforms both the ERM and DA+ERM baselines.\n    - The reviewer mentions that DA+IVL improvement over DA+ERM is modest (though, we argue, still significant) in **Colored-MNIST**. We point out that this is still competitive with or better than established OOD generalization benchmarks. Also, we urge the reviewer that DA+IVL should not distract from giving credit on the undeniable performance of DA+ERM baseline itself, as DA as soft intervention is also a central contribution of our framework.\n- **Weakness: Linearity assumption in the optical device dataset.**\nThis is a fair point. We followed the precedent set by [33] who introduced this dataset and also analyzed it under a linearity assumption [10]. While the true mechanism may or may not be linear, this setup allows for a controlled experiment and direct comparison with prior work.\n\n### Addressing Specific Questions\n\n1. **Comparison with causal regularization methods.**\n    \n    [12] involves a purely theoretical analysis for using \\ell\\_2 to improve causal estimation in purely observational (X, Y) setting.\n    \n    [10] considers also considers purely observational (X, Y) but under special, linear modelling assumptions so that now estimating \\| f \\| is possible, even if f isn’t. This information is then used to develop a cross-validation scheme.\n    \n    In [11] authors consider data from 2 observational (X, Y) sources to develop a croos-validation scheme for \\ell\\_2 regularization based on stability of regression parameters across the 2 datasets.\n    \n    Regarding incorporating these into our experiments, we thank the reviewer for highlighting this area of improvement. Due to time limitations, however, we couldn't include additional experimental results in this rebuttal. Nevetheless, we do plan to complete and incorporate these into the camera-ready revision.\n    \n2. Si**gnificance and meaning of $\\alpha$ in $\\mathcal{I}\\_{\\alpha}$ — the degree of OOD robustness.** In Theorem 1, the set $\\mathcal{I}\\_{\\alpha}\\subseteq\\operatorname{span}(\\mathbf{K}^\\top)$ represents the interventions over which the IVL estimate has robust, or optimal ‘worst-case’ predictive performance. The parameter $\\alpha$ can be interpreted as a ‘knob’ that determines how OOD the interventions in this set would be. As $\\alpha\\rightarrow 0$, then $\\lim_{\\alpha\\rightarrow 0}\\mathcal{I}\\_\\alpha = \\operatorname{span}(\\mathbf{K}^\\top)$ such that now our estimates would have robust predictions over arbitrarily large interventions of the form $\\operatorname{do}(\\tilde{Z}\\coloneqq\\xi), \\;\\xi\\in\\operatorname{span}(\\mathbf{K}^\\top)$. So, essentially, $\\alpha$ determines the trade-off IVL makes between in-distribution risk (i.e., ERM) vs. OOD treatment-intervention risk (i.e., IV regression).\n3. **On justification and notation of Observation 1**\n    \n    **Justification of Observation 1.** Theorem 4 (Appendix G.7) shows that for SEM $\\mathcal{A}$ in Eq. (7),\n    \n    $X^{\\mathcal{A};\\operatorname{do}(\\tau\\coloneqq G\\tau)} = GX^{\\mathcal{A}}, \\qquad Y^{\\mathcal{A};\\operatorname{do}(\\tau\\coloneqq G\\tau)} = Y^{\\mathcal{A}}.\\qquad(\\dagger)$\n    \n    So it immediately follows that  $P\\_{X, Y, G, C}^{\\mathcal{A};\\operatorname{do}(\\tau\\coloneqq G\\tau)} = P\\_{GX, Y, G, C}^{\\mathcal{A}}$ given that $G$ and $C$ are exogenous.\n    \n    **Intuition for Observation 1.** The intuition behind Eq. ($\\dagger$) and, by extension, Observation 1 is that the transformation $GX$ does not change $Y$ by definition of outcome-invariance, *despite cycles*.\n    \n    **Clarification of notation for $G\\tau$.** We \n    \n    - **Definition:** $G\\tau$ is simply the composition $G\\circ\\tau$. Given that we have\n    \n    $\\tau: \\mathcal{Y}\\times\\mathcal{C}\\times\\mathcal{N}\\_X\\rightarrow \\mathcal{X}, \\qquad \\delta:\\mathcal{X}\\times\\mathcal{G}\\rightarrow \\mathcal{X}$,\n    \n    then using $\\mathbf{g}\\mathbf{x}\\coloneqq \\delta(\\mathbf{x}, \\mathbf{g})$ as shorthand for group action, $G\\tau$ is simply\n    \n    $G\\tau(\\cdot) = G\\circ\\tau(\\cdot) = \\delta( \\tau(\\cdot), G ).$\n    \n    - **Form in Example 2:** For Example 2, $\\tau$ has the block-matrix form\n    \n    $\\tau(\\cdot) = \\mathbf{T}(\\cdot) ,\\qquad \\mathbf{T}\\coloneqq[\\mathbf{0}\\_{m} \\quad \\mathbf{E}^\\top \\quad\\sigma\\mathbf{I}\\_{m\\times m}]$\n    \n    so that now for any tuple $(Y, C, N\\_X)$ we have\n    \n    $\\tau(Y, C, N\\_X) = \\mathbf{T} \\begin{bmatrix} Y \\\\ C \\\\ N\\_X \\end{bmatrix}$.\n    \n    Finally, given the definition $GX\\coloneqq X + \\gamma\\mathbf{K}^\\top G$, we have\n    \n    $G\\tau(\\cdot) = G\\circ\\tau(\\cdot) = \\mathbf{T}(\\cdot) + \\gamma\\mathbf{K}^\\top G.$\n    \n\n### **Response to Limitations**\n\nWe fully agree with the reviewer and will explicitly emphasize this context in the camera-ready version. Our goal is not to claim full identification, but to provide a principled step forward from ERM, but the residual bias still necessitates caution. Crucially, we will frame this as the reality of the problem setting, where our method still offers a robust and provable improvement over standard ERM.\n\nWe appreciate the reviewer’s feedback and hope our response addresses their concerns. We are confident the clarifications will strengthen the manuscript and better highlight its practical relevance. We also note that the reviewer’s comments and criticisms focus primarily on identifiability, whereas others reviewed our work from a broader perspective.\n\n**References:**\nSee manuscript."}, {"review_id": "IY1NUwSENi", "reviewer": "Reviewer_WchF", "summary": "This study proposes treating outcome-invariant data augmentation (DA) as a form of soft intervention on the treatment in causal effect estimation. The proposed method aims to mitigate hidden confounding bias by introducing instrumental variable like regression, which interprets data augmentation parameters as weak instruments and incorporates empirical risk minimization (ERM) regularization. The authors theoretically and empirically demonstrate that this approach outperforms standard ERM and conventional DA methods in terms of intervention generalization and estimation bias.", "questions": "Please address the weaknesses listed above.", "limitations": "The authors did not discuss the limitations of their method in the main text. In my view, while the proposed method introduces a novel perspective on data augmentation, its theoretical properties rely on some untestable assumptions, such as the inclusion of unobserved confounding information in the augmented data. If my understanding is incorrect, please feel free to correct me.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- S1: The paper addresses a valuable topic and presents an interesting and novel methodological contribution.\n- S2: The figures and notation are well-organized, and the presentation is easy to follow.\n- S3: The paper is the first to place data augmentation (DA) within a causal inference framework, showing that under the outcome invariance assumption, DA is equivalent to a soft intervention on the treatment. The introduction of IVL regression relaxes the requirements of classical IV methods and is theoretically shown to reduce confounding bias under certain conditions. The practical interpretations of the theoretical results are also insightful.\n\nWeaknesses:\n- W1: While the topic of improving predictive performance via data augmentation is important, the manuscript lacks a discussion on the scope of applicability of the proposed method and its connections to classic literature. In particular, I would like to see the authors discuss how their approach relates to and differs from the well-known Independent Causal Mechanisms (ICM) principle. Specifically, under what conditions does enriching the marginal distribution of $P(X)$ (via data augmentation) help or fail to improve the conditional distribution of $P(Y \\mid X)$?\n- W2: Although the theoretical analysis is thorough and informative, the empirical improvements are limited, especially on the Colored-MNIST dataset, where DA+ERM already accounts for most of the gains, and the additional benefit from combining DA with IVL is marginal.\n- W3: The effectiveness of the proposed DA+IVL method in strictly reducing confounding bias relies on the assumption that the augmentation perturbs latent directions aligned with unobserved confounding, which is untestable. Moreover, the theoretical properties also depend on additional assumptions, such as joint diagonalizability of covariances, which may not hold in practice.", "was_revised": true, "final_justification": "The authors provide a clear and detailed discussion of the relationship between this work and Independent Causal Mechanisms and offer a fundamental analysis of the untestable assumption I raised. I believe that the revised version incorporating these points exceeds the conference’s acceptance threshold, and therefore, I recommend accepting this paper.", "rebuttal": "We thank reviewer for their constructive feedback and comments. We have addressed their concerns and questions below.\n\n### W1: Independent Causal Mechanism (ICM) Principle\n\nWe find the reviewer’s comment about relation of our model to the ICM principle by Peters et al. [1] quite interesting, especially since cyclic SEMs have been around for quite a bit longer than the ICM principle itself, with roots in econometrics from as early as 1940s.[2] Despite this, we are not aware of any explicit reference to the principle in modern literature on cyclic SEMs, including in works by Peters et al.[1][2][3] Nevertheless, we can observe the following.\n\n**Similarity — Structural equations as ICMs:** Our SEM formulation in Eq. (1) is a direct instantiation of the ICM principle as described by Peters et al.[1] The two equations represent the autonomous mechanisms, and their independence is captured by the mutual independence of the exogenous noise terms $N_X, N_Y$. The simultaneity in our model is not a violation of ICM, but rather the equilibrium state resulting from the interaction of these two independent mechanisms. Assuming the existence of this equilibrium is a statement about the scope of systems under analysis, and not about the nature of the mechanisms themselves. Indeed, surgically changing $\\tau$ to some $\\tau’$, for example, does not in itself alter $f$ and vice versa. And precisely because of ICM this may or may not make the system unstable depending on the nature of $\\tau’$, but in our setting Theorem 4 (Appendix G.7) shows that soft interventions induced by outcome-invariant DA are *always* stable.\n\n**Difference — Markov factors as ICMs:** We speculate that this is the main motivation for reviewer’s comment about ICMs. Indeed, in most settings, ICMs are discussed in terms of more familiar Markov factorization of joint distribution of the system since such a factorization always exists in DAGs. However, such a factorization generally does not exist for cyclic models, which is precisely why they are almost exclusively represented as simultaneous structural equations.\n\nWe thank the reviewer for this very insightful and thought-provoking question, and agree that this connection to ICMs should be made explicit in the manuscript. To address this in the preliminaries Section 2 we will make clear that our model's structural equations are a direct instantiation of the ICM principle and point to a more detailed commentary in Appendix B as an appropriate place to elaborate nuances since it is exclusively dedicated to discuss cyclic SEMs.\n\n### Success and failure modes w.r.t. **ICM**\n\nGiven the above clarification about structural equations being a more appropriate choice for representing ICMs in cyclic models, **we believe our manuscript already addresses the reviewer’s wishes of success/failure modes** with respect to ICMs $\\tau$, $f$, etc. We provide a recap (and an additional comment) for completeness.\n\nThe task is to improve estimation of $f$ which corresponds to the ICM of the outcome $Y$. And we do this by changing/ intervening on the ICM $\\tau$ of the treatment $X$ while keeping all other ICMs in the system unchanged. This is done via DA, and it is successful/unsuccessful in the following cases.\n\n1. **Outcome-invariant DA:**\nObservation 1 shows that outcome-invariant DA is equivalent to surgically changing the treatment ICM $\\tau$ to $G\\tau$ while keeping all other ICMs in the system unaffected. As such, outcome-invariance complies with the ICM principle.\n    1. **Success — perturbing spurious features:** Theorems 2 shows that this reduces confounding bias when the DA aligns with spurious features.\n    2. **Benign failure — no spurious features perturbed:** Theorems 2 also shows that this does not reduce confounding bias when the DA is orthogonal to spurious features. Fortunately however, this does not perform worse than ERM, and so the failure is rather benign compared to the following case.\n2. **Malignant failure(?) — outcome-invariance violation:** Although outside the scope of our current work, we conjecture that a violation of outcome-invariance is equivalent to a violation of the ICM principle itself. Since outcome-invariance is necessary for DA equivalence to IVL, a violation would mean that the DA not only affects $X$ but also $Y$ so that it is now a confounder. We believe this to be true because violations of outcome invariance in DA has been shown to bias estimates in works that don’t necessarily study causality [6]. Although this raises many interesting questions in itself, we defer a further formal investigation to future works in order to keep the current submission focused.\n\n### Scope of applicability\n\nGiven the above classification, it is natural, as the reviewer points out, to ask whether the conditions required for our method’s success are *reasonable*. In the next section, we argue that they are—particularly when viewed through the appropriate lens of instrumental variables (IVs).\n\n### **W3 & Limitations: On Untestable Assumptions**\n\nWe agree with the reviewer that the untestability of assumptions is a central concern, and appreciate the opportunity to clarify this point more directly in the main text.\n\n### 1. IVs Are the Right Benchmark\n\nWe believe IVs are the appropriate lens for judging whether our assumptions are \"too strong.\" Our IV-like variables (IVLs) are a direct generalization of IVs—designed to be more available by relaxing the outcome relevance condition—while still aiming to reduce confounding. This generalization naturally inherits the core limitations of IVs: **assumptions like exclusion restriction and unconfoundedness are not testable from observational data and must be justified via domain knowledge.**\n\nSimilarly, outcome-invariant data augmentation (DA), which we show is equivalent to IVL by construction (Observation 2), carries the same interpretability and limitations. In this light, **we argue it is inappropriate to expect our method to be free from the very limitations of the framework it generalizes**.\n\n$\\overbrace{\n\\underbrace{\\text{outcome-invariance}}\\_{\\text{popular model for DA}}\n+\n\\underbrace{\\text{spurious targets}\n}\\_{\\text{benign failure if violated}}\n}^{\\text{un-testable DA assumptions}} \\iff \\overbrace{\\text{un-confoundedness}+\\text{exclusion}}^{\\text{un-testable IV/IVL assumptions}}$\n\n### 2. Clarifying Assumptions\n\n- **Outcome Invariance:** The central assumption is outcome-invariant DA—analogous to exclusion restriction and un-confoundedness in IVs. While untestable, this is a common and accepted modeling choice in the DA and invariance literature [6–8].\n- **Perturbing Spurious Features:** This condition is not a new limitation but a direct analogue of the exclusion principle—an IV must enter $Y$ via a spurious feature of $X$ to identify the corresponding confounded component of $f$.\n- **Joint Diagonalizability:** The reviewer raises a valid concern about Assumptions 1 and 2 behind our theoretical results. These are **only required** for our use of the **Relative Squared Error (RSE)** metric, which—as argued by Janzing [9]—is more relevant for causal estimation. Our core results on confounding bias reduction still hold under the standard risk-based estimation error $\\mathbb{E}^{\\mathcal{M}}[\\ell(f(X), \\hat{h}(X))]$, **without** requiring Assumptions 1 and 2.\n\nAcknowledging that these clarifications could be better emphasized, we will move and expand the limitations discussion from the Appendix into the main paper (as Section 7), directly addressing the concerns around untestable assumptions and theoretical scope.\n\n### **W2: On Empirical Improvements**\n\nWe acknowledge your observation that on Colored-MNIST, the additional benefit of DA+IVL over DA+ERM is modest (although, arguably still significant). We would like to offer two points of context:\n\n1. **DA+ERM's performance *is* a key result:** A central contribution of our paper is to provide a causal interpretation for why outcome-invariant DA works: it acts as a soft intervention. The fact that the simple DA+ERM baseline is already so effective is strong empirical validation of this core thesis. The performance of DA+ERM should therefore be seen as a strength of our framework, not a weakness of the DA+IVL extension.\n2. **Contextualizing the DA+IVL improvement:** While the additional gain from DA+IVL is modest on this specific task, it is still competitive with or better than several established OOD generalization methods (as seen in Figure 5). Furthermore, on the Optical Device dataset, our proposed **DA+IVLcc is the only method that provides a significant performance gain over both the ERM and DA+ERM baselines**, demonstrating its value in other settings.\n\nWe hope these clarifications have addressed your concerns. We thank you again for your insightful review, which helped us identify key areas for improvement in the presentation of our work. We are confident that by incorporating these points, we can make the paper's contributions and limitations clearer to the reader.\n\n[1]: J. Peters et al. *Elements of Causal Inference*. MIT Press, 2017.\n\n[2]: C. F. Christ. “The Cowles Commission's Contributions to Econometrics at Chicago, 1939–1955.” *J. Econ. Lit.*, 1994.\n\n[3]: S. Bongers et al. “Foundations of Structural Causal Models with Cycles and Latent Variables.” *Ann. Statist.*, 2021.\n\n[4]: J. M. Mooij et al. “On Causal Discovery with Cyclic Additive Noise Models.” *NeurIPS*, 2011.\n\n[5]: D. Rothenhäusler et al. “BACKSHIFT: Learning Causal Cyclic Graphs from Unknown Shift Interventions.” *NeurIPS*, 2015.\n\n[6]: S. Chen *et al.* *A group-theoretic framework for data augmentation*. JMLR, 2020.\n\n[7]: C. Lyle *et al.* *On the benefits of invariance in neural networks*. arXiv, 2020.\n\n[8]: H. Shao *et al.* *A theory of PAC learnability under transformation invariances*. NeurIPS, 2022.\n\n[9]: D. Janzing and B. Schölkopf. *Detecting confounding in multivariate linear models via spectral analysis*. J. Causal Inference, 2018."}]

Author rebuttal:

Meta-review: The authors propose a new approach to treatment effect estimation through data augmentation (DA) and regularized instrumental variable (IV)-like regression.  Assuming that the outcome is invariant to some properties of the data, they generate data which preserves the outcome invariance and show that this can be used to reduce confounding bias in the case where we do not have identifiability and thus standard IV approaches are not possible.  Results include theoretical guarantees (in the linear Gaussian setting), practical estimation algorithms, and experiments showing improved bias reduction on simulated, Colored-MNIST, and optical device datasets.

Strengths of the paper:
* The use of data augmentation in a causal setting (rather than just for generalizability in typical prediction settings) is novel.
* The IV-like regression, a generalization of standard IV that does not require dependence of the outcome on the IV, is of high and independent interest. The observation that IVs can help reducing confounding bias without being relevant to the outcome is highly original and useful.
* Strong and convincing experimental results.
* Well-written and clear.

Weaknesses of the paper:
* Missing a limitations section.
* Theoretical guarantees are available for fairly limited settings (but I don't think that this is a major issue).
* More clarity needed about the assumptions and their connections to (a) typical IV analysis and (b) independent causal mechanisms -- this was resolved in the discussion and the authors have promised to add this to the camera-ready version.

Acceptance decision: I am in favor of acceptance as a spotlight, primarily because of the novelty and high interest of the work.

Discussion and rebuttal: The conversation was positive and productive, with all reviewers agreeing that the paper is worthy of acceptance and that no major concerns remained after discussion.  The authors made an important point that the approach generalizes typical IV analysis and therefore depends on the same (untestable) assumptions -- since IVs are a common and useful method, this is a reasonable starting point for the work.  Additional experimental results (comparisons to two papers in the literature) were provided by the authors and should (and can easily) be included in the camera-ready version.  The authors should also add a brief limitations section to the camera-ready version and have agreed to do so.
Decision: Accept (spotlight)