Incorporating Interventional Independence Improves Robustness against Interventional Distribution Shift
Authors: Gautam Sreekumar
Vishnu Boddeti
Venue: Accepted by TMLR ()

Abstract: We consider the problem of learning robust discriminative representations of causally related latent variables given the underlying directed causal graph and a training set comprising passively collected observational data and interventional data obtained through targeted interventions on some of these latent variables. We desire to learn representations that are robust against the resulting interventional distribution shifts. Existing approaches treat interventional data like observational data and ignore the independence relations that arise from these interventions, even when the underlying causal model is known. As a result, their representations lead to large disparities in predictive performance between observational and interventional data. This performance disparity worsens when interventional training samples are scarce. In this paper, (1) we first identify a strong correlation between this performance disparity and the representations' violation of statistical independence induced during interventions. (2) For linear models, we derive sufficient conditions on the proportion of interventional training data, for which enforcing statistical independence between representations of the intervened node and its non-descendants during interventions lowers the test-time error on interventional data. Combining these insights, (3) we propose RepLIn, a training algorithm that explicitly enforces this statistical independence between representations during interventions. We demonstrate the utility of RepLIn on a synthetic dataset, and on real image and text datasets on facial attribute classification and toxicity detection, respectively, with semi-synthetic causal structures. Our experiments show that RepLIn is scalable with the number of nodes in the causal graph and is suitable to improve robustness against interventional distribution shifts of both continuous and discrete latent variables compared to the ERM baselines.

Published in Transactions on Machine Learning Research (12/2025)
Incorporating Interventional Independence Improves Ro- bustness against Interventional Distribution Shift
Gautam Sreekumar
sreekum 1mmus edu
Deportment of Computer Science and Engineering, Michigan State University
Vishmu Naresh Boddeti
Cechuu&@ms..cmm edu
Department of Computer Science and Engineering, Michigan State University
Reviewed on OpenReview: htips: // openreview. net/ forum? id= kXfcByNIrs
Abstract
We consider the problem of learning robust discrininative representations of causally related latent variables given the underlying directed causal graph and training set comprising passively collected observational data and interventional data obtained through targeted interventions on some of these latent variables. We desire to learn representations that are robust. against the resulting interventional distribution shifts. Existing approaches treat interventional data like observational data and ignore the independence relations that arise from these interventions, even when the underlying causal model is known. As result, their representations lead large disparities predictive performance between observa- tional and interventional data. This performance disparity worsens when interventional training samples scarce. In this paper, (1) we first identily strong correlation between this performance disparity and the representations' violation of statistical independence in- duced during interventions. (2) For linear models, we derive sufficient conditions on the proportion of interventional training data, for which enforcing statistical independence be- tween representations of the intervened node and its non-descendants during interventions lowers the test-time error on interventional data. Combining these insights, (3) we propose RepLIn, training algorithm that explicitly enforces this statistical independence between representations during interventions. We demonstrate the utility of RepLIn on synthetic dataset, and on real image and text datasets on facial attribute classification and toxicity detection, respectively, with semi-synthetic causal structures. Our experiments show that RepLIn is scalable with the number of nodes the causal graph and is suitable to improve robustness against interventional distribution shifts of both continuous and discrete latent variables compared to the ERM basclines.
1 Introduction
We consider the problem of learning robust discriminative representations corresponding to latent variables for downstream prediction tasks. These latent variables usually correspond to semantic concepts such as the color of an object, the level of glucose in the blood, and a person's age. The relationship between these latent variables can be modeled using directed acyclic graphs (DAGs) called cousal graphs. Causal modeling allows manually altering the causal graph and observing its effects on the data. E.g,, intervene on the amount of insulin (parent variable) in the blood by consuming an insulin inhibitor and then measure the blood glucose level (child variable). This procedure is known as a cousal intervention, and the data collected through this procedure is called interventional data. In contrast, data passively collected without intervention known observational data. Of the several types of interventions possible on causal graph, interested in

---

Published in Transactions on Machine Learning Research (12/2025)
hard interventions where manually set the value of one or more variables. Intervening node renders it statistically independent of its parent nodes in the causal grapt See (Peters et al. 2017 Chapter 6).
Consider a causal graph with latent variables A and B. During observation, A causes B (A B). An attribute-specific representation of A, denoted by FA. learned from only observational data, may contain information about its child node B due to the association between A and B, although this association may be broken during interventions on B. Example: Consider computer-aided diagnosis system that inputs chest X-ray image and outputs representations corresponding to A = "air sac inflammation" to predict pneumonia. and B "fluid accumulation around lungs" to check for pleural effusion. These representations will be used by their corresponding diagnosis-specifs prodictors. This design makes the system modular and interpretable. Here, pneumonia is one of the many unrelated causes of excess fluid accumulation. E.g., excess fluid accumulation can happen as a side effect of some medication (intervention) unrelated to pneumonia. The representation, FA. used to predict pneumonia may incorporate information about fluid accumulation (B) to aid pncumonia diagnosis, although excess fluid accumulation does not guarantee pneumonia. To avoid misdiagnosis, these representations must include only the information necessary for their diagnostie purpose, and must be robust against distribution shifts due to any interventions causally downstream variables.
To learn representations that are robust against interventional distribution shifts, interventional data samples are included in the training set. For example, in (Sauer & Geiger 2021 Gao et al. 2023). interventional data was generated to train image classification models invariant to texture and background. When interventional training data is available, existing discriminative learning approaches treat interventional data merely as data sourced from a different domain or environment, ignoring the explicit statistical independence relations arising from interventiona (Arjovsky et al. 2019 Sagawa et al. 2020 Heinze-Deen & Meinshausen, 2021) As we demonstrate, ignoring these independence relations may lead to representations that are still susceptible to interventional distribution shifts during inference. Moreover, performing interventions in the real world is often challenging and expensive. This limits the amount of interventional data available for training and thus demands a causally motivated lcarning strategy that leverages the limited interventional training data.
In this work, we first consider a simple case study where we observe that models that do not learn indepen- dent representations during interventions show a performance drop on interventional data. We then derive sufficient conditions the proportion of interventional data during training, under which enforcing lincar independence between interventional features of linear models during training can reduce test-Lime error on interventional data. Motivated by these theoretical insights, we propose "Representation Learning from Interventional Data" (RepLIn), an algorithm to learn representations with improved robustness against interventional distribution shifts. We confirm the utility of RepLIn on a variety of synthetic (Sec. 5.1 and real datasets (Secs. 5.2 and 5.3) on various modalities with semi-synthetic causal structures, and demonstrate its scalability to the number of nodes (Sec. 6.2
Our Contributions:
Observation: We demonstrate a positive correlation between the accuracy drop during interventional distribution shift and the dependence between representations corresponding to the label node and its children. We refer to this as "interventional feature dependence" (Sec. 3.3)
-heory:
We thcoretically explain why linear ERM models are susceptible to interventional distribu- tion shifts in the regime of linear causal models. In the same setting, we theoretically and empirically show that enforcing linear independence between interventional features improves robustness when suf- ficient interventional data is available during training and establish the sufficient condition (Sec. 3.4
Approach: We propose novel training algorithm that combines these insights and demonstrates that this model mininizes the drop in accuracy under interventional distribution shifts by explicitly enforcing independence between interventional features (Sec.
For case, we refer "statistical independence" "independence," and "hard interventions" "interveutions"" We will
also use "features" and "representations" interchangeably denote the vector representations the data learned by model. 2Th distribution shift. due to differing environments mare general than interventional distribution shift. See App. D
2

---

Published in Transactions on Machine Learning Research (12/2025)
2
Related Works
Identifiable Causal Representation Learning (ICRL) seeks to learn representations of the underlying
causal model under certain assumptions (Locatello et al. 2019 Schölkopf et al. 2021 Hyvärinen et al.
2024
and are important to interpretable representation learning. Several ICRL works also use interventional
data
Lippe et al. 2022b; 2023 Ahuja et al. 2023 Squires et al. 2023 von Kügelgen et al, 2023 Zhang
et al.
2023 Jiang & Aragam 2023; Buchholz el al. 2023 VarieI el al. 2024b Bing et al., 2024 Lachapelle
et al.
2024]. where known interventional targets are identifiably learned up to permutation ambiguity. In
contrast, we are interested in learning discriminative representations when some underlying causal relations are known. Instead of learning the entire causal model, we seek to exploit the known independence relations from interventions to learn discriminative representations that are robust against these interventions, and
hence do not require fully identifiable models. A detailed review of ICRL is in App. C
Interventional data key in causal discovery [Eberhardt al. 2005 Yu et al., 2019 Ke et al. 2019 Lippe
et al. 2022a, Wang et al. (022b) as one can only retrieve causal relations up to a Markov equivalent graph without interventions or assumptions on the causal model. For example, known interventional targets have interventional
been used for unsupervised causal discovery of linear causal models (Subramanian et al. 2022
and observational data have been leveraged for training supervised model Ior causal
discovery (Ke et al.
2022,, and interventions with unknown targets were used for differentiable causal discovery
Brouillard et al.
p020 Interventional data also find applications in reinforcement learning (Gasse et al.
2021 Ding et al.
2022a) and recommendation systems (Zhang et al. 2021; Krauth et al. 2025 Luo et al.
2024 While this
body of work focuses on discovering causal relations the data, our work considers how to leverage known
causal relations to learn data representations that are robust distribution shifts induced by interventions.
Domain Generalization (DG): In DG, the learning objective is predictor for an attribute of interest that is robust/ bobust/invariat 'invariant to changes in the domain/ domain/envvroonen /environment (Mahajan et al. 2021: Wang et al. 2022a; Ding et al. 2022b). Here, there is no interest in learning representations for the domain, and multiple factors could be jointly treated as a single domain. Moreover, there is also no requirement that the learned predictor for the attribute of interest is free of domain information (Rosenfeld et al.) 2022). Therefore, the learned reprosentations obtained from domain generalization may not be trustworthy for modular applications such as the medical diagnosis system described in Sec. 1 A more detailed discussion is provided in App. D
3 The Learning from Interventional Data Problem
Random variables and random vectors are denoted by regular (e.g.. A) and bold (e.g., a) serif characters, respectively. The distribution of a random variable A denoted by PA-
Notations:
We now formally define the problem of interest in this paper, namely learning attribute- specific discriménative representations that are robust against known interventional distribulion S in general terms, and examine a specific case study in Sec. 3.I. The learning problem is characterized by DAG g that causally re- lates the attributes of interest AI...., Am: and B. Let Pa(B) = {AI,..., Am) denote the parents of the altribule B. These altributes along with other unobserved exogenous variables U, generate the observable data X as X (B Ai...., Am. Um, During interventions, the variable is set to values drawn from a known distri- bution independent of Pa(B). Therefore, the post-intervention variable B (denoted by B) is statistically independent of its parents, i.e., B IL Pa(B), as shown in Fig. [ Although gx is not affected by this intervention, the distribution of X (now denoted by X') will change since it is function of B.
Setup:
During observation
During intervention
U
A
A
Abe
U
A
A
Aue
X
B
X'
B
Figure 1: Causal graph modification due to inter- vention: During observation, B is the effect if parent variables Pa(B) (Ai....,A.,, When intervene B, it becomes statistically independent of its parents.
We use "discriminative" explicitly state that the purpase of these representations is robust prediction and data generation. Information loss with improved robustness is therefore acceptable.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Assumptions: To learn representations that are robust against distribution shift due to intervention B, our setting only provides us information about B and its parents in the causal graph, and not of any causal relations between AI....,Am Am. We also do not place restrictions on the functional form of causal relations between , Am. B, and X, or on their marginal distributions. Our training set comprises both observational and interventional samples, i.e., Dirain = Dobe Diitt, where Dobe ~ P(X. B, AL...., Am) and Dint ~ P(X', B, .....,,,4 Am). However, the number of interventional training samples much less compared to the number of observational training samples, i.e., |Dint| < [Dobe], Given Dirain and g, the goal is to learn attribute-specifi discriminative representations FB (x) and FA, hA. where FA, are robust against distribution shifts due to intervention on B.
3.1 Does Accuracy Drop during Interventions Correlate with Interventional Feature Dependence?
In this section, will design synthetic dataset for case study to establish correlation between the accuracy drop interventional data and the statistical dependence between the attribute representations under intervention.
Data legend
A
B
A
B
A 0, B 0
0
&
0
A = 0, B = 1
A = 1, B 0
U
X
U
X
A B=1
-2
0
2
0
2
X1
X1
(a) Observational graph and data
(b) Interventional graph and data
Figure 2: An illustration of Windmill Dataset: A and B binary random variables that are causally linked to each other and X, as shown in (a). By intervening B as shown in (b), we make A -l B. X = 9x (A, B, U) where U denotes unobscrved noise variables. The true decision boundaries for predicting A and B from X are shown in red and blue dashed lines, respectively, See App. I for a detailed description.
Windmill Dataset: In the causal graph shown in Fig. 2a A and are binary random variables that gencrate the observed data X E RR X also affected by an unobserved noise variable U. Functionally, X = [9 B, U). A itself could be a funetion of unobserved random factors that are of no predictive interest to us. Therefore, we model A ~ Bernoulli(O.6). The distribution of B is only affected by A, denoted by the arrow between them. Analytically, B == A, where == is the causal assignment operator, following (Peters et al.) 2017). Visually, the observed data looks like windmill. The value of A determines the windmill's blade, and B determines the radial distance. The precise angle and radial distance of the data samples are determined by noise samples independent of and B. The windmill blades are also sheared as a sinusoidal function the radial distance. In Fig. 2b] we intervene on B, modeled as ß Bernoulli(O.5) This induces a change in the distribution of B, and subsequently in that of X. Since the intervention is independent of A, B is also independent of A, denoted by removing the arrow between A and B.
Note that 9x is a one-to-one mapping by construction and unaffected by this intervention. Therefore, a single model can predict A and B from X accurately during both observation and intervention. However, the true decision boundary for A is more compler than that of E Therefore, models may use information from B to predict A due to their association during observation. similar to the concept of simplicity bias (Shah et al. 2020). The exact mathematical formulation of the data-generating process provided in App-
The lcarning task is to accurately predict A and B from X test time. We have N samples for training, where 3N are interventional and (1 (1-8)N are observational with 0 < 8 < 1. For this demonstration, = 40, 000, B = 0.01 to get 39,600 observational and 400 interventional samples. We train foed-forward network with two hidden layers to learn representations FA and FB corresponding to A and B, respectively. We normalize them by dividing them by their corresponding L2 norm. Separate linear classifiers predict A
set
We informally define "complexity" as the minimum polynomial degree required to approximate the decision boundary.
4

---

Published in Transactions on Machine Learning Research (12/2025)
and B from FA and F3 respectively. As result of WINDMILL's construction, FA may contain information about B even during interventions when A II. B. Following the standard ERM framework, the cross-entropy errors in predicting A and B from FA and FB, respectively, provide the training signal. The statistical loss function can be written as Lrotal(f) = Ep..amn [Cpred []]] The training distribution is a mixture of observational and interventional distributions with (1-8) and / acting as the corresponding mixture weights. Thus, Leotal (f) = (1 - Pow [ (f: xobo + Pr. [e xl
Accuracy in predicting A
Accuracy in predicting B
ERM version
NHSIC
Obsorvation
Intervention
Relative drop
Obsorvation
Intervention
Relative drop
Vanilla
99.98 = 0.01
60.15 3.12
0.40 + 0.03
100.00 + 0.00
99.99 + 0.01
0.72 = 0.06
w/ Resampling
94.53 1.14
70.20 + 3.73
0.26 0.03
100.00 0.00
99.99 0.01
0.64 0.08
Table 1:
The relative drop in accuracy in predicting A correlates well with gap in the measure of dependence between the learned representations on interventional data.
Observations: Tab. Tab. sshoss the accuracy of ERM in predicting A and B on observational and interventional data during validation. We expect no drop in accuracy from observation to intervention if the learned representations are robust against interventional distribution shift. However, we observe that ERM performs only slightly better than random chance in predicting A on interventional data. As remedy, we modify the vanilla ERM method to sample observational and interventional data in separate batches, and thus prevent the gradients from interventional samples being obfuscated by those from observational samples, which are likely to be more in number in a given batch. This is equivalent lo sampling interventional data (47") times as observational data. We refer to this version as "ERM- "esampee, The equivalent training loss for model f in ERM-Resampled is Lrotat (/) = Era. [Epred (J, x + +Bp,.. [4pred (J, Xi"))). Note that 8 does not appear in (f) due to resampling. Although ERM- ReMM-eesampeed performs better than vanilla ERM, ERM-Rcsampled still cxhibits a large drop in predictive accuracy between observational and interventional data during inference. We also observe a drop in Resappessss ' observational accuracy of predicting A as it improves its interventional accuracy. As we will show in Sec. 3.4, this drop in observational accuracy is due to the removal of spurious information previously exploited to boost its observational accuracy.
3.2 Measuring Statistical Dependence Between Interventional Features
A key consequence of hard interventions in causal graphs is that the variable being intervened upon becomes independent of all its non-descendants. Since the predictive accuracy on the parent node is affected by inter- vention, we hypothesize that the parent node's representation remains dependent on that of the child node during intervention, even when their underlying latent variables in the causal graph become independent. To verify our hypothesis, we measure the dependence between the representations. We choose to measure the dependence between the representations instead of between the representations and the latent attributes to align with our goal of learning robust ietribute-speciffy representations.
To measure dependence between a pair of high-dimensional continuous random variables X and Y, we use HSIC (Gretton et al. 2005). Empirical HSIC between N i.i.d. samples X = {a; and y = {y ) from X and Y, respectively, is HSIC(X, = -ー Trace [ H Hy H], where H is the N x N centering matrix, and Kx, Ky N are Gram matrices whose (1, entries are 4x (æx,, ar,) and ky (yk, V,), respectively. Here, hx and ky are Mercer kernels (Schölkopf & Smola, 2002 Chapter 2). Since HSIC is unbounded, we HSTTCX.))
normalize it as NHSIC(X,)) VHSIC(X,X)) HSICQU,)) following (Cortes et al. 2012 Cristianini et al. 2001
To improve computational efficiency, we use random Fourier features RRaimi & Recht 2007).
Observations: Tab. | compares NHSIC values between the features FA and Fn learned by ERM and ERM- Resampled on interventional data from WINDMILL dataset. We observe that features learned by ERM had more statistical dependence during interventions than those by vanilla ERM, indicating a larger violation of the underlying statistical independence relations in the causal graph during interventions. Interestingly, the relative drop in accuracy also increases with the statistical dependence between interventional features.
5

---

Published in Transactions on Machine Learning Research (12/2025)
3.3 Strength of Correlation between Drop in Accuracy and Interventional Features Dependence
How strong is the observed correlation between the dependence of features and the drop in accuracy? For a given combination of prodictive task and dataset, does it hold for a variety of hyperparameter settings? To answer these questions, we train several models under the ERM-Resampled setting described in Sec. 3.1 The learned models are ford-forward networks, each with one to six hidden layers and with 20 to 200 hidden units. We also use early-stoppigg in our training, as it has been noted as an effective regularizer (Sagawa et al. 2020j. Early-stopping is executed in our experiments by choosing an arbitrary number of training epochs for each run. We measure the robust ness of a model to interventional distribut ion shift using the relative drop in accuracy between observational and interventional data: Rel.A Obe -Ic--Int scc,, Similar experiments were Obs acc.
reported in (Sreckumar& & Boddcti 2023). although their primary rescarch question concerned the effect of data and model complexities on spurious correlations. In the following experiment, we expand their setting to deeper models and more variety in hyperparameters while foregoing the variation in data complexity.
In Fig. 3 we plot the relative drop in accuracy against the interventional feature dependence. In addition to NHSTC, we also use kernel canonical correlation (KCC) (Bach & Jordan, 2002) to mea- sure the dependence. The strength of the correla- tion between the relative drop in accuracy and in- terventional feature dependence is quantificd using Spearman's 's rank correlation coefficient (p) (Spear- man 1904) and Kendall's rank correlation coefi- cient [endalll 1938). ) In Fig. 3a P = 0.81 and J 0.61 when the dependence measured us- ing NHSIC, indicating that the correlation noted in Sec. 3.2 can be observed for a wide range of hyper- parameters. When KCC is used for measuring inter- ventional feature dependence, / = 0.75 and 7 = 0.56 as shown in Fig. 3b We observe that all models with a high relative drop in accuracy also have a large in- terventional feature dependence (see top-right regions in the plots). However, the corollary is not true a large interventional feature dependence does not mean relative drop in accuracy. Therefore, we conclude for this case study that relative drop in accuracy is always accompanied by interventional feature dependence.
0.8
0.8
NHSIC
0.6
0.2
0.4
P 0,81, c 061
p: 0.75, r: 0.56
0.0
0.4
0.0
0.2
Rel.A in accuracy
Rel.5 accuracy
(a) Rel.A against NHSIC
(b) Rel.A against KCC
Figure 3: Across various model capacities and hy- perparameter settings, a relative drop in accuracy is always accompanied by interventional feature depen- dence. However, the corollary does not hold. Feature dependence is measured using NHSIC and KCC.
Note on Choice of Measure of Dependence: The correlation strength in Fig. 3 was affected by our choice of measure of dependence. A popular measure of information between two random variables is Shannon mutual information (MI). However, computing MI requires density estimation as the first step, which is challenging for high-dimensional data (Paninski 2003 McAllester & Stratos 2020). For the same reason, MI is also not suitable for training representations. We note that a variational upper bound for MI can be obtained (and minimized to enforce independence between representations) if the conditional density
of one random variable w.r.t. the other is known (Barber & Agakov 2004 Alemi et al. 2018 Poole et al.
2019), although computing this bound still requires a tractable density.
In contrast, kernel-based measures of dependence, such as HSIC and KCC, are computationally ef- ficient and well-suited for optimization. Both NHSIC and KCC satisfy the postulates for an appropriate measure of dependence in (Rényi, 1959 and measure dependence from the spectrum of the cross-covariance operator between reproducing kernel Hilbert spaces (RKHSs). However, HSIC measures the Hilbert-Schmiit norm of the covarianv operator while KCC measures its spectral norm (largest singular value). As result, KCC is more suited for independence tests where the presence of dependence is more important than its overall strength. Informally, KCC is a "harsher" measure of dependence compared to NHSIC. HSIC between two random variables is equivalent to maximum mean discrepancy (MMD) (Gretton et al. 2012) between the joint distribution and the product of marginals of these variables (Schrab) 2025). MMD was originally proposed to check whether two samples came from the same distribution or not. They have similar computational costs. HSIC is also a lower bound on MI (Sriperunbudur et al., 2012 Xu et al. 2024). For the remainder of this work, we will use NHSIC for training and analysis, and reserve KCC for evaluation.
6

---

Published in Transactions on Machine Learning Research (12/2025)
3.4 Will Minimizing Dependence between Interventional Features Improve Robustness?
Our case study in Sec. 3.3 showed that large relative drop in accuracy is always accompanied by strong interventional feature dependence. Based on this observation, we ask the following question: Will minimizing interventional feature dependence improve the robustness to interventional distribution shifts? We answer this question theoretically using a linear cansal model. The detailed proof of each step is provided in App. B
We use the causal model shown in Fig. 2a with A and B being continuous random variables. A and B are causally related during observation as B WABA. Although our analysis is valid if an external noise was added to B, we will skip such noise term in the proof for conciseness. The observed data signal X is generated from A and B as X
Causal Model:
X + U, where XA = WAA and XB = WB.. B. U =
U^
=
XB]
UB.
is exogenous noise. UA and UB are independent of A and B respectively. We intervene on B as shown in Fig. 2b severing the causal relation between A and B. The intervened variable is denoted as B' and B' T1 A.
Learning Model: Similar to Sec. 3.3 the task is to predict the latent variables A and B from observed data signal X. The training data is sampled from a training distribution Purina which we model as a mixture of observation distribution, Pohs, and interventional distribution, Pint, with mixture weights (1 8) and 3, respectively: Pirai = (1 (1-3)PPaa + Pm We use linear models to learn attribute-specific representations FA and FB, from which predictions Å and B, respoctively, are made using corresponding classifiers. The linear models are parameterized by ((A) and ( and the classifiers are parameterized by c(A) and c(B).
Statistical Risk: The parameter matrix of the linear feature extractor for A can be written in terms of
0 (A)T)
its constituent parameter vectors as (A) =
0 A Assuming zero mcan for all latent variables 5 the
0
statistical squared error of an arbitrary model in predicting A from interventional test sample X 2 T A
EA- (1 (1-WAch WAC
T)T (A)
pa + (c(4)T ((1))
2
pir. + ( o( P + (c(4TT0%a)) 2 p
(1)
F
where P3 = Em [ Pi:- = pm [ pi.. Er,.. [U]], and pùn = Er [U%] The statistical risk can be split into two components: (1) EQ" in terms of A and UA, and (2) E in terms of B and UB. E * 0 when 0(aa² + 0. A non-zero ((4) indicates that the representation FA is a function of XB, iee, it learns a spurious correlation with B. Thus, the prediction Å is susceptible to interventions on B. In contrast, a robust model will have 0(4) 0, and thus achieves E O. Derivation of Eq. (1 is provided in App. B.1
An optimal ERM model minimizes the expected training risk in predicting the latent attributes. Since we are interested in the accuracy drop in predicting A from interventional data, we consider the optimization of parameters for predicting A by minimizing the expected mcan squared error over the training distribution.
[A- (A--C4)T@(AN) x) 2
(A)+ c(A)+ argmin Ereaa @LA),c(A)
(2)
Since (A) and () can be optimized only up to a scaling factor, we can equivalently optimize &b c(A)T $1 where th (A) (A) and th2 c(1)T (())) Thus, Eq. (2) becomes
1/2
3 argmin Erren (CA-
To check whether the optimal ERM model is robust, we can verify if eb² = 0, since a robust model will have of ((4) = 0. Solving Eq. (3) by setting the gradients of the optimization objective to zero, we get:
- - 2 #0,
(4)
T
5re sero mean assumption make the calculations easier. always learn the mean of the data separately.
This will affect our conclusion from the proof, as can
7

---

Published in Transactions on Machine Learning Research (12/2025)
where T is non-zero scalar. If there was an added noise term in the causal relation A - B, Eq. (4) would have taken a different form, but would have still been non-zero. Eq. (4) implies that 0/a11 * 0, and consequently implies F * in optimal ERM models. Therefore, optimal ERM models are not robust against interventional distribution shift. The detailed derivation is provided in App. B.2
Note that a robust model cannot be a minimizer of training loss in Eq. 13 as the minimizer requires 0 * O. This means improving robustness by minimizing spurious information from B in predicting A (through 0/A41 + 0) may lead to higher prediction loss over observational data. This explains the drop in observational accuracy of ERM- BRM--Rsammled when its interventional accuracy in predicting A improved in Sec. 3.1 This phenomenon is also illustrated in App. H
Minimizing linear dependence: In Sec. 3.3, we showed that interventional feature dependence correlated positively with the drop in accuracy on interventional data. We will now verify if minimizing dependence between interventional features FΛ and F'y can reduce the accuracy drop in a linear setting.
FA (A)) X = (1O + xB0 (A) (B) F θ (B)T
(Interventional feature for A)
X x X + X (B)
(Interventional feature for B')
Since the data generation process and the learned models are linear, it is sufficient to minimize the linear interventional dependence between representations instead of the full statistical dependence. Following the definition of HSIC (Gretton et al. 2005), the linear dependence in interventional features can be defined (
Dep (FA, F/n) = |E FI
Leveraging the independence relations during interventions, we can expand Eq. 5 as,
Era. F
B _F (w>2 + pú, 10/100110000000
( - (wbpp' + pú. 1
B)
(6)
The dependence loss is thus the Frobenius norm of sum of rank-one matrices. Three classes of solutions
minimize Eq. 66: (1) g(4) = 0 0(4i = ((8) = ( = 0, (2) 0/A) = =y0 (A) and 70(3)) =
(B) for some
scalar 7 * 0, and (3) 0(4) 0 0 ( O, and B features and increase the classification errör, except two non-degenerate solutions: (S1) 0141 0, ( 0, and (S2) 0/=1) 0, (B) = 0, where (S2) corresponds to robust model. Since both (S1) and (S2) minimize Eq. the solution with lower prediction error over A and B will prevail during training.
or 0 B) O. However, all solutions produce trivial
Proposition 1. The total training error for (S1) is strictly greater than that of (S2) when the following
conditions are satisfied: (1)82 B 1211
WAM (2) 3 2 min 20 TP
AAPA
Proposition D conveys that, given a certain proportion of interventional data in the training set, explicitly enforcing independence between learned representations can provably improve robustness against interven- tional distributional shifts. Note that Proposition 1 describes sufficient conditions for (S1) to have a larger training error than (S2), and does not imply the contrary for smaller values of 3. In practice, 3 could be much smaller. For instance, in Tab. 2 explicitly enforcing independence using our proposed approach improves robustness even for 3 = 1%. Refer to App. B.3 for derivation and empirical verification of Proposition |
To experimentally verify the thcoretical results, we construct the causal model with WAA = WB = WAB 1. The random variables A, B, UA, and Up are sampled from independent normal distributions with zero mean and unit variance. We generate N 50000 data points for training with 0.5. The classifiers use 2-dimensional features learned by linear feature extractors to predict A and B. The experiment is repeated with 50 seeds. EX and E(P) the components of the statistical risk in Eq. 11,
, are plotted in
Figs. 4a and 4b respectively. As reminder, a robust model will achieve E 0. As expected, both
models have similar (1()) However, linear independence models achieve lower (2) (L A
(2)
~ 0, similar to a
A
robust model) than ERM models, and thus obtain a lower total error EA (Fig. 4c
For complete definition of the dependence, refer App. R.3
8

---

Published in Transactions on Machine Learning Research (12/2025)
ERM
Linear independence
Figure 4: Robust models achieve E = 0 in Eq. ERM models have a non-zero (A4) resulting in E Minimizing linear independence on interventional features results in orthogonal interventional feature spaces where 0/A) 0(() O. Thus, they result in robust models with E 0.
0,54
0.545
Tes
0.02
0.52
0.30
(a)
(b)
A
(c) E
4 RepLIn: Enforcing Statistical Independence between Interventional Features
The previous section demonstrated strong correlation between the accuracy drop during interventions and interventional feature dependence. We also showed theoretically that minimizing lincar dependence between interventional features can improve lest-time error on interventional data for linear models. Based on this observation, we propose "Representation Learning from Interventional data" (RepLIn), a training method to learn discriminative representations that are robust against known interventional distribution shifts.
During observations
F
Lxat
FRM
Uses only Epei for training.
Encoder
Classifier A
Eyped
Ignores distribation changes from interventions.
deep (inly during interventionss)
Rapl.In (ours)
During
Epredt
Lay! consistency interventional
Encoder B
Classifier B
causal model.
X
Lxt eucourage learning attribute-
specific information.
F
Ciat
Figure 5: Schematic illustration RepLIn for two-variable causal graph (A + B) where X gx (A, U,). Encoders for and B learn corresponding representations FA and FB, which are then used by their corresponding classifiers predict Å and B, respectively. On interventional samples, minimize Ldep between the features to ensure their independence. On all samples, minimize Lyelf to improve attribute-specifi information in the representations.
To enforce independence between interventional features, propose use dependence-guidee regularization, denoted Ldep, in addition to the prediction loss (e-g., cross-entropy for classification tasks) used in ERM. We refer to this regularization "dependence loss," and is defined for the general case in Sec. 3
as Laep Σi= NHSIC Fint). We minimize the dependence loss only for the interventional Training samples since congruent statistical independence between latent variables occurs only during interventions.
However, Ldep may encourage learning irrelevant information to lower feature dependence, Therefore, to courage lcarning relevant attributo-specific information, we use the "self-dependence loss," denoted by Luelt to maximize the dependence between feature and its corresponding label on both observational
and interventional training data. We define Caeff as Lielr 1 CHSIC(Fs. B)+ EY (HSIC(FA, Using
2(++l)
Leef in addition to Lpred ensures that the representations contain much information about the modeled latent variables possible, rather than just the information required for the train-time prediction task.
Kernel choice for Ldep and Lyeir: The nature of dependence captured by Ldop and Leeft depends on the richness of the underlying RKHS induced by the kernel used in NHSIC. To maximize a lower estimate of the dependence, we will use linear kernels, Æx (æ, x;) = x; xy, in Ceelt- Likewise, to remove all non- non-linear dependence between the represcntations, we will use RBF kernels Ldep- In general, universal kernels, such RBF and polynomial kernels, capture non-linear dependence between variables up to the resolution decided by the kernel's parameters (Micchelli et al.
2006 Fukumizu et. al. 2007). The empirical estimates of

---

Published in Transactions on Machine Learning Research (12/2025)
HSIC in our loss functions have a bias of [(n against their population estimates, where is the number of independent samples used to estimate HSIC (Gretton et al. 2005). The empirical HSIC estimates thus converge to their population estimates when the sample size is large. For computationally efficient estimation
of NHSIC, we use random Fourier features (Rahimi & Recht 2007).
In summary, RepLIn optimizes the following total loss: Liotal = Speed + Adep + Aselt where Xdep and Agel are weights that control the contribution of the respective losses. The impact of the choice of these hyperparameters is discussed in App. H A pictorial overview of the RepLIn pipeline is shown in Fig. 5
5 Experimental Evaluation
In this section, we verify the effectiveness of RepLIn compared to the baselines using the WINDMILL dataset introduced in Sec.3.11 and evaluate its broader applicability to practical scenarios through the facial attribute prediction task on the CelebA dataset (Liu et al.) 2015) and toxicity prediction on the CivilComments datasct (Borkan ct al., 2019). Since the truc causal models are not known for real datascts, we design plausible causal models for these datasets based on the variables of interest, and sample data points according to these semi-synthetic causal models. Through our experiments, we validate the following hypothesis: Does explicitly minimizing the interventional feature dependence reduce accuracy drop during interventions?
Training Setup and Baselines: We consider vanilla ERM and ERM-Resampled (Chawla et al. 2002:
Cateni et al. 2014) as our primary baselines since they are the most commonly used training algorithms training and domain generalization (Idrissi et al.] 2022; Gulrajani & Lopez-Paz 2021). On WINDMILI dataset, we also
Additionally, ERM-Resampled has been shown to be a strong baseline for group-imbalane
(Arjovsky et al., 2019), Fish (Shi et al. 2020 SAGM (Wang et al. 2023), DIWA RRme et al. 2022) and TEP Qiao & Peng 2024). The latter two are weight-avveaaging methods, for which we train 20 independent models per seed. We do not include ICRL baselines, as they learn intribute-identifaale up to permutation invariance, which cannot be used with attribute-specific classifiers. We compare two variants of our method against these baselines: RepLIn and Repllu-Ressmpled. The latter variant uses the resampling strategy from ERMMResampled. In each method, attribute-specific representations are extted from the input data and fed into the corresponding classifiers to get the final prediction. All baselines use the same architecture to learn reprosentations and linear layers to make the final prediction from these representations. The values of Adep and Amalt in RepLIn variants are tuned and kept fixed for all values of B. A detailed description of the datasets and the training settings is provided in App. A
consider the following domain generalization algorithms: IRMv1
2022), GroupDRO (Sagawa et al.
representations
Evaluation Metrics: Our primary interest is in investigating the accuracy drop during interventions for the variables unaffected by these interventions. Ideally, the learned features respect causal relations during interventions, we expect no change in the prediction accuracy of parent variables of the intervened variable between observational and interventional distributions. To measure the change, we use the relative drop in accuracy defined in Sec. 3.3 Rel.A Obe acc aec.-Int acc. Since optimize NHSIC during training, we Obe acc, use KCC from Sec. 3.3 to evaluate the interventional feature dependence during testing. We repeat each experiment with five different random seeds and report the mean and standard deviation.
5.1 Windmill dataset
We first evaluate our method on WINDMILL datasct that helped us identify the relation between the per- formance gap between observational and interventional data in predicting A in Sec. 3.1 As a reminder, the causal graph consists of two binary random variables, A and B, where A + B during observations. We intervene by setting B ~ (ernoulli(O.5). breaking the dependence between A and B. The proportion of interventional samples in the training data varies from 8 = 0.01 to 3 = 0.5.
Tab. 2 compares the interventional accuracy in predicting A for various amounts of interventional training data. We make the following observations: (1) RepLIn outperforms every bascline in interventional accuracy for all values of 3, clearly demonstrating the advantage of exploiting the underlying causal relations when learning from interventional data, in contrast to treating it a separate domain. (2) Comparing ERM and
10

---

Published in Transactions on Machine Learning Research (12/2025)
RepLIn with their resampling variants, we observe the usefulness of resampling through its large gains when B is very small (e.g., 8 < 0.05). We are also interested in the relative drop in accuracy between observational and interventional data (Rel.A). From Tab. [ we observe that GroupDRO has the lowest Rel.A among the considered methods for 3 2 0.05, and achieves its best results when more interventional data is available during training. However, this improvement comes at the cost of lower interventional accuracy - over 7 percentage points difference compared to RepLIn. Mcanwhile, Rel.A of RepLIn is comparable GroupDRO at larger values of 3 and has the least Rel.A lower values of 3. DiWA and TEP were provided with the same pool of models trained with minor variations in their hyperparameters. We do not consider the negative Rel.A of TEP since (1) TEP achieves very low interventional accuracy, performing barely above random chance, and (2) due to the high variance of its Rel.A. We provide the results on observational data in App. E As mentioned in Sec. 3.1 interventional robustness may be at odds with observational accuracy, as removing spurious information from representations may hurt performance on observational data. Our analysis in shows that the representations learned by RepLIn are less affected by interventional shifts.
Sec. 6.1
Accuracy on interventional data. The relative drop in accuracy shown in parentheses.
Method
0.5
B 0.1
B
0.05
8 0.01
ERM
26.87
(0.18
69.86.3.10
(0.29_0.04)
62.78
00
59.52+1.30
(0.40a001)
60.15
(0.40
ERM-Hes.
13.70
(0.22
(0.24_o.co)
73.62 11
(0.22)
71.03
(0.25
a0.
(0.26_nora)
IRMv1
78.24
(0.16
74.8321.74
(0.20_...
78. 78.61
02
76.2841.87
(0.1840
71.7020.0
(0.2120.0)
Fish
77.23 j2.4
(0.19 (0.0))
77 27:3 1.
(0.19
78.24
(0.1810022
76.42
(0.201oo2)
(0.23 13.92
GRoupDRO
s0.10
(0.02 (0.02:o.oz]
80 1.
04.0..2)
80.35
(0.06:0.02)
77.40=1.11
(0.08:0.o.)
71.86
(0 20.02)
SAGM
18.43
(0.19
19:05
10.17_n..0
76.96.4.59
(0.181ous))
79.86
(0.161002)
12.81
(0.23_mm0)
DIWA
22.61
(0. (0.00)))))
76.71 76.71
(0.19=0.01)
76.00
(0.20
76.834188
(0. (0.20
73.39
(0 22.00)
58.68
(9)
60 60.42:1.00
(0.09.0.0o)
56.07
-0.44
58.52
20.01
50.23g11a
(0.18
RepLin
87.91s1-40
(0.0840.02)
87. 87.76=2..0
(0.10=0.02)
83. 23
(0. (0.1640.0))
73.6842.43
(0.25++0.2))
67.52=2.0
(0.32.0.00)
RepLIn-Res.
88. 88.46 90
88.05 (0.07
20.0)
87.9 87.91
(0.08
86.38 as
(0.10
78.41
(0.18
Table 2: Results on Windmill dataset: We evaluate the variants of RepLIn (highlighted in green) against the baselines on two metrics: interventional accuracy and relative accuracy drop on interventional data compared to obscrvational. Compared the basclines, RepLIn maintains its interventional accuracy as the proportion of interventional data during training (B) decreases. A similar trend is observed in the relative accuracy drop, where RepLIn remains significantly more robust than most baselines. The best and the second-best results are shown in different colors. "Res." stands for "Resampled".
5.2 Facial Attribute Prediction
Smillng
Gender
Smiling
Gender
(a) Observational causal graph and samples
(b) Interventional causal graph and samples
Figure 7: Causal models for CelebA before and after intervention, along with images sampled from them.
We verify the utility of RepLIn for predicting facial attributes on the CelebA dataset [Liu et al. 2015 Among the 40 binary attributes that the images in the CelebA dataset are annotated with, we consider smiling and gender attributes for our prediction task. Since the true underlying relation between smile and gender unknown, we adopt the resampling procedure from (Wang & Boddet 2022) induce a desired causal relation between the attributes (smiling -> gender) and obtain samples. Specifically, to simulato this causal relation, we sample smiling from Bernoulli(O66) first, and then sample gender according to a probability distribution conditioned on the sampled smiling variable. We then sample face image whose attribute labels match the sampled values from the semi-synthetic causal model. The diversity in the images is modeled as the result of unobserved noise variables. Note that, unlike in WINDMILL, these noise variables may be causally related to the altributes that we wish to predict, adding to the challenges in the dataset. The semi-synthetic causal model for this experiment and some sample images are shown in Fig.
We first extract features from the face images using ResNet-50 (He al. 2016) model pre-trained on the TmageNet dataset (Deng et al. 2009). Then, similar to the architecture used for the WINDMILL experiments,
11

---

Published in Transactions on Machine Learning Research (12/2025)
employ shallow MLP to act on these features, followed by linear classifier to predict the attributes. Our loss functions act upon the features of the MLP. We use 30,000 samples for training and 15,000 for testing. We use the relative drop in interventional accuracy as the primary metric for comparing against ERM- ReM--ed We also verify if the correlation between interventional feature dependence and the relative drop in accuracy observed in Sec. 3.3 on WINDMILL. experiments holds a more practical scenario.
ERM- Resample
Ressaln-e
0.20
Figure Facial Attribute Prodiction: (a) RepLIn has lower relative drop in ac- curacy (Rel.) compared to ERM-Resampled. (b) Minimizing interventional feature dependence during training generalizes to testing. (c) Interven- tional feature dependence correlates positively with the relative drop in accuracy (p= 0.86).
0.45
0.4-
0.05
0.2
0.2
(a) Accuracy
(b) Dependence
(c)
Correlation
Fig. 8 reports the experimental results on facial attribute prediction for various amounts of interventional training data. We make the following observations: (1) as the proportion of interventional data increases, the relative drop in accuracy (Rel.A) in all methods decreases, (2) across all proportions of interventional data, RepLIn consistently outperforms the baseline in Rel.A by 4 to percentage points, despite the potential challenges due noise variable being causally related the attributes of interest, (3) relative drop in accuracy and interventional feature dependence show strong positive correlation (p = 0.86), and (4) the interventional feature dependence of RepLIn steadily decreases the amount of interventional data increases.
5.3 Toxicity Prediction in Text
We evaluate RepLIn on a toxicity prediction task on the CivilComments dataset (Borkan et al., 2019), which contains comments from online foruns. We use a subset of this dataset labeled with identity attributes (such as male, white, LGBTQ, etc.) and toxicity scores by humans. The task to classify each comment as toxic
or not. Since prior works have identified gender bias in toxicity classifier models (Dixon et al. 2018 Park ct al. 2018 Nozza al. 2019), will simulate semi-synthetic causal model between the attribute female and toxicity: female-> toxicity. During observation, both attributes assume identical binary values. During intervention, toxicity becomes independent of f female. Input text comments are then sampled according to the attributes obtained from this semi-synthetic causal model. Similar to Sec. 5.2 we first extract features from the comments using BERT (Devlin et al., 2019 and train the baselines on these features. The bascline architecture consists of a linear layer to learn representations and a linear classifier layer to predict toxicity.
ERM-Reeae Resampled
Resompll
0.8
Figure 9: Toxicity Prediction in Text: (a) RepLIn has lower interventional accuracy drop compared to ERM-Resampled; (b) Minimizing Ldep during train- ing gives representations that are independent dur- ing interventions; (c) The strong correlation between accuracy drop and interventional feature dependence further validates our hypothesis Sec. 3.2
0.6
0u6
KCE
3
0.05
0.4
(a) Accuracy
(b) Dependence
(c) Correlation
Fig. 9 compares the performance of RepLIn against ERM-Resample Fig. 9b shows that enforcing inde- pendence between interventional features minimizes the interventional feature dependence during lesting, although its effectiveness drops as 3 approaches 0.01. RepLIn still outperforms the baseline terms of the accuracy drop during interventions (Fig. Dab. We also note that RepLIn becomes more efficient in mini- mizing the interventional feature dependence 3 increases. The correlation between interventional feature dependence and Rel.A in Fig. Dc further validates hypothesis in Sec. 3.2
12

---

Published in Transactions on Machine Learning Research (12/2025)
6 Discussion
6.1 How are representations learned by RepLIn different from those by ERM?
In this section, we qualitatively and quantitatively compare the interventional features learned by RepLIn and baselines to understand how RepLIn improves robust.ness against interventional distribution shift.
Method
FRM
ERM ERM- Resampled
IRMv1
Fish
GroupDRO
RepL.In
Resemappee
When
0.45 0.058
0.423 0.105
+ 0.122
0.34
0.365
0.066
0.03
0.032
When
0.499 0.07
0.456 +: 0.11
0.405 + 0.111
0.37 0.136
0.431 0.048
0.183 1 0.058
0.168 0.047
Average
0.475 + 0.063
0.439 0.105
0.369 + 0.116 0.355 0.113
0.398 0.055 0.166 0.035
0.178 +: 0.036
Table 3: We compare the Jensen-Shannon (JS) divergence between interventional features from the baselines for WINDMILL with 0.5. Since the distribution of F from robust model is invariant to the value assumed by B since A I B during interventions, JS divergence between P(F = 0, 0, a) and P( B = 1, A = a) will be zero for robust model, for E {0, 1J. Among the baselines, RepLIn achieves the lowest values of -hasen-Shannon divergence. The lowest and the second lowest scores are colored.
Windmill dataset: Robust representations of A should change with A, but not. B. We quantify their change with B using the Jensen-Shannon (JS) divergence between the distributions of F for a fixed value of and changing values of B. Tab. 3 shows the JS divergence between P(F B = 0, A A a) and P(F B 1, A a) (obtained through binning) for multiple baselines trained on WINDMILL dataset. JS divergence for robust model will be zero. We observe that Fiat learned by RepL.n achieves the lowest JS divergence, meaning Fint learned by RepLIn varies distributionally the least with B among the baselines.
Top row: When A 0,
Bottom row: When A
0
Density
Demstty
Demity
Denutyy
Inclination
Azimuth
Demity
Density b
6
Denitty
Demsity
Densiry
2
Inclinstion
Acimuth
Inclination
Azimuth
Inclination
Azimauh
(a) ERM
(b) ERM-Resampled
(c) ReppLn-Reeanupled
Figure 10: Visualization interventional features learned by various methods on WINDMILL dataset.
Since the learned representations were normalized to be unit sphere during training, can examine them qualitatively through the histograms of the inclination and azimuth angles they subtend at the origin. We compare these histograms of Fittt learned by RepT.In-Resamplee against the ERM baselines in Fig. 10 Each row shows the histograms for different values of A. Histograms for different values of B are shown in different colors. Remember that the feature distributions for a robust model must change with A, but not B. We observed from the figure that the feature distributions of the baselines are affected by B more than by A due the dependence between Fint and B. However, the feature distributions learned by RepLIn change with A and overlap significantly for different values of B. Thus, representations learned by RepLIn are more similar to those of robust model. Similar visualizations for other baselines are provided in App. G
We inspect the high-dimensional features learned on CelebA through their output atten- tion maps oblained using Grad-CAM (Selvaraju et. al. 2017). Fig. 11 shows the atlention maps from models trained on the CelebA dataset with 0.5 for some samples with smiling = that were misclassified by CRM--Rssampecs but were correctly classified by regions around the lips make predictions about smiling. Observe that CeeLIn-Resamplee tends focus more the region around the lips (associated with smiling), while FRM-RRsampledd attends to other regions
CclebA datasct:
A robust model would attend
13

---

Published in Transactions on Machine Learning Research (12/2025)
of the image, such hair and cap. This supports the trustworthiness of representations learned by RepLIn. GradCAM visualizations on the samples accurately classified by ERM, but not RepLIn, are shown in App. F
Figure 11: Consider these sample face images where the subjects are smiling. The ERM baseline misclassi- fied these samples as not smiling, while RepLIn classi- fied them correctly. We use GradCAM visualizations to identify the regions in the imput images that the models used to make their predictions. The ERM model relied on factors such as hair and the presence a hat that may correlate with gender to predict whether the subjects are smiling, In contrast, RepLin attended to the lip regions to make prodictions.
(a) E ReM--eea
(b) RepLIn-Resaupled
6.2
Scalability with number of nodes
A
A
A
X^ = MLPa(A)
XB MLP
B
C
B
B
B
C
Xc (CP
XD MLP:(D)
D
E
D
E
E
D
X E MLP,(E)
(a) Observational
(b) Intervening on C (c) Intervening on D (d) Intervening on
(e) Generating X
We construct a 5-variable causal graph to demonstrate the scalability of our method with the number of nodes. To collect interventional data, we intervene on C, D, and E separately and measure the performance drop in predicting A and B during these interventions. Intervened nodes in the graph have dashed borders. Note that do not intervene multiple targets constructed as concatenation of individual input signals, each being function of latent variable, i.e., x [X] XB X XJ X Here, MLP, indicates a raudomly initialized MLP with linear layers, each followed by a ReLU, We also add Gaussian noise sampled from A/ 0.01) to the output of the MLP.
Figure 12:
time. The input data signal X is
In practice, the variable for which we wish to learn robust representations may have multiple child nodes. Therefore, is imperative that RepLIn scalable with both the number of intervened nodes and their parents. To verify this scalability, we use the causal graph shown in Fig. [2a with five latent variables. It consists of two binary source nodes, A and B, and three binary derived nodes, C, D, and E. The observed signal X is concatenation of attribute-specifec observable components from each latent variable. The attribute-specific observable component X, for the latent variable p E (4, B, C. D, F} is obtained by passing p through corresponding randomly initialized MLP and adding noise to its output (Fig. 12e The MLPs used for A and B have more layers than those for C, D, and E to entice the model to learn spurious correlations between parent and child nodes. The learning task is to predict the latent variables from X.
Task:
Training Setup: During observations, A and B are sampled from independent Bernoulli(O.5) distributions, and the remaining nodes are obtained through the following logical expressions: C= Av B, D AAB, and collected by intervening nodes C, D, and E one at time, as shown in Figs. 12b to 12d The intervened variable assumes values from Bernoulli(0.5) distribution independent of their parents. Each training batch has either observational or interventional samples, identical to the resampled setting. No batch will contain interventional data with different intervention targets. Thus, our method enforces the independence relations from most one interventional target each batch. The validation and test sets consist of samples collected during interventions on C, D, or E. Our primary metric will be the predictive accuracy for A and B during separate interventions on C, D, and F.
E = -B A C. Interventional training samples
14

---

Published in Transactions on Machine Learning Research (12/2025)
Interventionl|
Predictive accuraty on
Predictive accuracy ea B
Method
sarget
0.5
9 0.3
0.1
0.05
8 0.5
8 3
6=1
3 0.05
ERM-Resampled
79.71 #
76.22 + 0.42
73.97 4: 0.39
73.56 +: 0.36
87.60 0.06
85,45
83.89 + 0.33
83.71 0.40
RepLIn Resampled
95.37 0.97
78.77 0.54
72.15 0.31
73.74 0.36
96.72
0.81
86.16
52.35
0.99
82.43 4 0.65
D
Chop.InReesampe
Ressmppee
96.49 1.01
79.65 0.43
77.76 +: 0.82
75.47 1: 0.61
0.335 71.76
71.20
70.27 1: 0.34
68.80 à: 0.79
97.87 0.31
91.06 =: 0.29
92:21 à: 0.48
90.21 = 0.27
91.40 0.79
90.36 0.58
90.88 0.89
90.53 d: 0.74
E
RepLin Resampled
GRtM-Resampled
96.71 0.49
$6.63 0.33
84.08 0.36
81.90 4: 0.26
76.20 0.8.4
75.01 0.53
73.46 4: 0.37
71:52 a: 0.87
96.89 0.68
81.12 = 0.22
80.88
78.00 = 0.48
1 0.57
74.02 4: 0.38
72.81 1.13
72.97 0.38
71.60 0.59
Results on 5-variable causal graph: We compare the accuracy of RepLIn in predicting the source nodes A and B during interventions on non-source nodes C, D, and E against that of ERM-Resampled. Our approach outperforms the baselines with sufficient interventional data.
Table 4:
The predictive test accuracies in Table 4 show that RepLin significantly improves over the baseline with sufficient interventional training data (3 0.1) for all intervention targets. When B s 0.1, RepL.In is comparable with the baseline. Our results demonstrate that the benefits of enforcing independence between interventional features extend to larger causal graphs with multiple intervention targets.
Observations:
6.3 Limitations
Since RepLIn requires knowledge of the intervened node and its parent variables, RepLIn is affected by causal graph misspecification involving the intervened node. For example, if parent of the intervened node is not known during training, RepL.n will not enforce the associated independence constraint during training and may lead to lower robustness against similar interventions at test time. RepLIn is also sensitive to imperfect interventions, where the intervened variable would be partially dependent on its parents, albeit with lowered dependence strength. Our experiments on imperfect interventions in App. B.4 showed that RepLIn's performance deteriorated when the interventional training data contained imperfect interventions, compared to a vanilla predictor trained with observational and perfect interventional data. However, it still outperformed vanilla. predictors trained on the same dataset, especially at lower values of B. These results indicate that RepLIn works best when perfect interventional samples are available, and is more useful when interventional data is scarce and sample-efficient methods are required to improve robustness. RepLIn also requires luning of the hyperparameters Adep and Agelts whose values lypically increase with the complexily of the data generation process (App. H.1pp The presence of additional hyperparameters also leaves RepLIn less attractive compared to intervention--peciiic predictors those scenarios where inference samples carry information about the underlying intervention, such as with genetic data, where gene-specific interventions are usually known at inference time. Unlike RepLIn, these intervention-speccfie predictors can be trained using only ERM loss, provided sufficient interventional training samples are available. RepLin is more suitable when inference-time intervention markers are unavailable.
7 Conclusion
This paper considered the problem of learning discriminative representations that are robust against known interventional distribution shifts and proposed a training algorithm for this objective that exploits the statis- tical independence induced by interventions in the underlying data generation process. First, we established a strong correlation between the accuracy drop during interventions and the statistical dependence between representations on interventional data. We then showed theoretically that minimizing linear dependence between interventional representations can improve the robustness of linear model against interventional distribution shifts. Building on this result, we proposed RepLIn to learn representations that are robust against interventional distribution shift by explicitly enforcing statistical independence between learned rep- resentations on interventional data. Experimental evaluation of RepLIn across different causal graphs on both synthetic and real datasets on image and text modalities with semi-synthetic causal structures showed that RepLIn can improve predictive accuracy during interventions, even for low proportions of interventional data. RepLIn also scalable to the number of causal attributes and can be used with continuous and dis- crete latent variables. We showed qualitatively and quantitatively that RepLIn is more successful in learning interventional representations that are unaffected by changes in their child nodes during interventions.
15

---

Published in Transactions on Machine Learning Research (12/2025)
Acknowledgments
The authors thank the anonymous reviewers and the action editor for their constructive criticism and general leedback. GS also thanks Ramin Akbari for the useful discussions. This work was supported part by the National Science Foundation (awards #2147116 and #2500983) and the Office of Naval Research (award #N00014-23-1-24177 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of NSF ONR.
References
Kartik Ahuja, Jason Hartford, and Yoshua Bengio. Weakly Supervised Representa- tion Learning with Sparse Perturbations. In Advances in Neural Information Process- ing Systems, 2022a.
URL https://papers. cc/paper htt.:/paer_fffeeessseeeeeeee wtt.f//s//aaeee//////a/aaa html
Kartik Ahuja, Divyat Mahajan, Vasilis Syrgkanis, and Ioannis Mitliagkas. Towards efficient representation identification in supervised learning. In Conference on Causal Learning and Reasoning, 2022b.
Kartik Ahuja, Yixin Wang, Divyat Mabajan, and Yoshua Bengio. Interventional Causal Representation
Learning. In International Conference on Machine Learning. 2023.
Alexander Alemi, Ben Poole, Tan Fischer, Joshua Dillon, Rif A Saurous, and Kevin Murphy. Fixing Broken ELBO. In International Conference Machine Learning, 2018.
Martin Arjovsky, Lóon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprinl Xii::1907.00900 2019. URL https: fictps///ariiv: wttp:////rr/aa/a////////// 028933
Francis R Bach and Michael I Jordan. Kernel Independent Component Analysis. Journal of Machine Learning Research, 3(Jul):1-48) 2002.
David Barber and Felix Agakov. The TM Algorithm: A variational approach Information Maximization. Advances in Neural Information Processing Systems, 2004.
Simon Bing, Urmi Ninad, Jonas Wahl, and Jakob Runge. Identifying Linearly- Mixed Causal Representations from Multi-Node Interventions. In Causal Learning and Reasoning, 2024.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification. In ACM Web Conference, 2019.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 2022.
Philippe Brouilard, Sébastien Lachapelle, Alexandre Lacoste, Simon Lacoste-Julien, and Alexandre Drouin. Differentiable Causal Discovery from Interventional Data. In Advances in Neural Information Processing Systems, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning Linear Causal Representations from Interventions under General Nonlinear Mixing, In NewrIPS, 2023.
Silvia Cateni, Valentina Colla, and Marco Vannucci. A method for resampling imbalanced datascts in binary classification lasks for real-world problems. Neurocompuling, 135:32 41, 2014.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. SMOTE: Synthetic Minority Over-Sampling Technique. Journal of Artificial Intelligence Research, 16:321-355 -357, 2002.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for Learning Kernels based on Centered Alignment. Journal of Machine Learning Research, 13:795- 828,,888,,888 2012.
Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On Kernel-Target Aligmment. In Advances in Newral Information Processing Systems, 2001.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, 1brahim Alabdulmohsin, et al. Scaling Vision Transformers to 22 Billion Paramoters. In International Conference Machine Learning, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Large-Scale Hierarchical Image Database. In IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of Deep Bidiree- tional Transformers for Language Understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Wenhao Ding, Haohong Lin, Bo Li, and Ding Zhao. Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoning. In Advances in Newral Information Processing Systems, 2022a.
Yu Ding, Lei Wang, Bin Liang, Shuming Liang, Yang Wang, and Fang Chen. Domain Generalization by Learning and Removing Domain-specific Features. In Advances in Neural Information Processing Systems, 2022b.
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and Mitigating Unintended Bias in Text Classification. In AAAT/ACM Conference on AI, Ethics, and Society, 2018.
Frederick Eberhardt, Clark Glymour, and Richard Scheines. On the Number of Experiments Sufficient and in the Worst Case Necessary to Identify All Causal Relations Among N Variables. In Conference for Uncerlainly in Arlificial Intelligence, 2005.
Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking Importance Weighting for Deep Learning under Distribution Shift. In Advonces in Neural Information Processing Systems, 2020.
Kenji Fukumizu, Arthur Gretton, Xiaohai Sun, and Bernhard Schölkopf. Kernel Mcasures of Conditional Dependence. In Advances in Neural Information Processing Systems, 2007.
Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. via Targeted Augmentations. In International Conference Machine Learning, 2023.
Robustness
Maxime Gasse, Damien Grasset, Guillaume Gaudron, and Pierre Yves Oudeyer. Causal Reinforcement Learning using Observational and Interventional Data. arXiv preprint arXiv:2106.1443 2021.
Robert. Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelli- gence, 2(11):665-673, 2020.
Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Mcasuring Statistical Dependence with Hilbert-Schmidt Norms. In International Conference on Algorithmic Learning Theory, 2005.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(1):723 773, 2012.
Ishaan Gulrajani and David Lopez-Paz. In Search of Lost Domain Generalization. In International Confer- ence on Learning Representations, 2021. URL https:://penrevvie ott.rr/roo....oooooooooooooaoaar
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CCEF/CVF Conference on Compuler Vision and Pallern Recognition, 2016.
Christina Srinze-Demll and Nicolai Meinshausen. Conditional Variance Penalties and Domain Shift Robust- ness. Machine Learning, 110(2):303 348, 2021.
17

---

Published in Transactions on Machine Learning Research (12/2025)
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised Feature Extraction by TimeeContrrative Learning and Nonlinear ICA. In Aduances in Neural Information Processing Systems, 2016.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ICA Using Auxiliary Variables and Gener- alized Contrastive Learning. In International Conference on Artificial Intelligncce and Statistics, 2019.
Aapo Hyvärinen, llyes Khemakhem, and Ricardo Monti. Identifiability of latent-variable and structural- equation models: from lincar to nonlincar. Annals of the Institute of Statistical Mathematics, 76(1):1-33, 2024.
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Pazz Simple data balancing achieves competitive wonst-group-accuuracyy Tn Conference on Cansal Learning and Reasoning, 2022. URL
https://prooceeding.. mlr. htt...ppreee/v/77/drisssss hmmm1m2222mmm111
Yibo Jiang and Bryon Aragam. Learning nonparametric latent causal graphs with unknown interventions. Advances in Neural Information Processing Systems, 2023.
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Bernhard Schölkopf, Michael C Mozer, Chris Pal, and Yoshua Bengio. Learning Neural Causal Models from Unknown Inter- ventions. Xixiv preprint arXiv:1910.01000 2019.
Nan Rosemary Ke, Silvia Chiappa, Jane Wang, Jorg Bornschein, Theophane Weber, Anirudh Goyal, Matthew Botvinie, Michael Mozer, and Danilo Jimenez Rezende. Learning to Induce Causal Structure. In International Conference on Learning Represenlations, 2022.
Maurice G Kendall. A New Measure of Rank Correlation. Biometrika, 30(1/2):81-93, 1938.
Iless Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. In International Conference on Artificial Intelligence and Statis- tics, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Con- ference on Learning Representations, 2015.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding. In International Conference on Learning Representations, 2021.
Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial Identifiability for Domain Adaptation. In International Conference on Machine Learning, 2022. URL httpps://proceedinggs mlr htt..prrrs//1116/ooog22222 http:///ccceaaaaa.....rrrrss///a htmll
Karl Krauth, Yixin Wang, and Michael I Jordan. Breaking Feedback Loops in Recommender Systems with Causal Inference. ACM Transactions on Recommender Systems, 4(1):1-21 20, 2025.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie E Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacosto-Julien. Disentanglement via Mechanism Sparsity Regularization: A New Principle for Noulinear ICA. In Conference Causal Learning and Reasoning, 2022.
Sébasticn Lachapelle, Pau Rodríguez López, Yash Sharma, Katic Everctt, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Ac- tions, Interventions and Sparse Temporal Dependencies. arXiv preprint F5i::2400044000 2024.
Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient. Neural Causal Discovery without Acyclicity Constraints. In International Conference on Learning Representations, 2022a.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. CITRIS: Cansal Identifiability from Temporal Intervened Sequences. In International Conference on Machine Learning. 2022b.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Phillip Lippe, Sara, Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal Representation Learning for Instantaneous and Temporal Effects in Interactive Systems. In International Conference on Learning Representations, 2023.
Ziwci Liu, Ping Luo, Xiaogang Wang, and Xiaoon Tang. Deep Learning Face Attributes in the Wild. IEEE/CVF International Conference on Computer Vision, 2015.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunmar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging Common Assumptions in the Unsupervised Learning of Disentangled Rep- resentations. In International Conference Machine Learning, 2019.
Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant Causal Rep- resentation Learning for Out-of-Distribation Generalization. In International Conference Learning Hepresentations, 2022.
Huishi Luo, Fuzhen Zhuang, Ruobing Xic, Hengshu Zhu, Deqing Wang, Zhulin An, and Yongjun Xu. A Survey on Causal Inference for Recommendation. The Innovation, 2024.
Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain Generalization using Causal Matching. In International Conference on Machine Learning, 2021.
David McAllester and Karl Stratos. Formal Limitations the Measurement of Mutual Information. In International Conference on Artificial Intelligence and Statistics, 2020.
Charles A Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal Kernels. Journal of Machine Learning Research, 7(12), 2006.
Gemma Elyse Moran, Dhanya Sridhar, Yixin Wang, and David Blei. Tdentifiable Deep Generative Models via Sparse Decoding. Transactions in Machine Learning Research, 2022. URL https://openrevver. net/ forum?1d=xvdonnG2222
Debora Nozza, Claudia Volpetti, and Elisabetta Fersini. Unintended Bias in Misogyny Detection. TEEE/WIC/AC International Conference Web Intelligence, 2019.
Liam Paninski. Estimation of Entropy and Mutual Information. Neural computation, 15()):1191-1253, 2003.
Ji Ho Park, Jamin Shin, and Pascale Fung. Reducing Gender Bias in Abusive Language Detection. In Conference on Empirical Methods in Natural Language Processing, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Newral Information Processing Systems, 2019.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On Variational Bounds of Mutual Information. In International Conference Machine Learning, 2019.
Fengchun Qiao and Xi Peng. Ensemble Pruning for Out-of-distribation Goneralization. In International Conference on Machine Learning, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning, 2021.
Ali Rahimi and Benjamin Recht. Random Features for Largo-Scale Kernel Machines. In Advances Neural Information. Processing Systems, 2007.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Alexandre Rame, Matthieu Kirchmeyer, Thiband Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse Weight Averaging for Out-of-Distriution Generalization. In Advances in Neural Information Processing Systems, 2022.
Alfréd Rényi On Measures of Dependence. Acta Mathematicn Academiae Scientiarum Hungarica, 10: 441 451, 1959.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-Adjusted Regression or: ERM May Al- ready Learn Features Sufficient for Out-of-Distriuuton Generalization. arXie preprint orXiv:2202.0663 2022. URL http::://axx...orr////eee/// wwt.orgg.ab.///aaaa 06856
Sorawit Saengkyongam and Ricardo Silva. Learning Joint Nonlinear Effects from Single-variable Interven- tions in the Presence of Hidden Confounders. In Conference for Uncertainty in Artificial Intelligence, 2020.
Sorawit Saengkyongam, Elan Rosenfeld, Pradeep Ravikumar, Niklas Pfister, and Jonas Peters. Identifying Representations for Intervention Extrapolation. In International Conference on Learning Representations, 2024.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In International Conference on Learning Representations, 2020.
Axel Sauer and Andreas Geiger. Counterfactual Generative Networks. In International Conference on Learning Representations, 2021.
Bernhard Schölkopf and Alexander J Smola. Learning with Kernels: Support Vector Machines, Regulariza- tion, Optimization, and Beyond. MIT Press, 2002.
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward Causal Representation Learning. Proceedings of the IEEE, 109(5): 612 634, 2021.
Antonin Schrab. A Practical Introduction to Kernel Discrepancies: MMD, HSIC & KSD. ar preprint arXiv:2503.0448 2025.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-bart based Localization. In IEEE/CVP International Conference on Computer Vision, 2017.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The Pitfalls of Simplicity Bias in Neural Networks. Advances in Neural Information Processing Systems, 2020.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN Features Off-the- Shelf: An Astounding Baseline for Recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition workshops, 2014.
Zheyan Shen, Peng Cui, Tong Zhang, and Kun Kınag. Stable Learning via Sample Reweighting, In AAAI Conference on Artificial Intelligence, 2020.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannın, Nicolas Usunier, and Gabriel Synnaeve. Gradient Matching for Domain Generalization. In International Conference on Learning Representations, 2022.
Xiangchen Song, Weiran Yao, Yewen Fan, Xinshuai Dong, Guangyi Chen, Juan Carlos Niebles, Eric Xing, and Kun Zhang. Temporally Disentangled Representation Learning under Unknown Nonstationarity. In Aduances in Neural Information Processing Systems, 2023.
Peter Sorrenson, Carsten Rother, and Ullrich Köthe. Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN). In International Conference Learning Representations, 2020.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Charles Spearman. The Proof and Measurement of Association between Two Things. The American Journal of Psychology, 15(1):72-101, 1904.
Chandler Squires, Anna Seigal, Salil Bhate, and Caroline Uhler. Linear Causal Disentanglement via Inter- ventions. In International Conference on Machine Learning, 2023.
Gautam Sreekumar and Vishmu Naresh Boddeti. Spurious Correlations and Where to Find Them. In Spurious Correlations, Invariance and Stability Workshop, International Conference on Machine Learning, 2023. URL
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Gert RG Lanckrict. On the empirical estimation of integral probability metries. Electronic Journal of Stalistics, 111500-1599
2012.
Jithendaraa Subramanian, Yashas Annadani, Ivaxi Sheth, Stefan Bauer, Derek Nowrouzezahrai, and Samira Ebrahimi Kahou. Latent Variable Models for Bayesian Causal Discovery. In International Con- ference on Machine Learning Workshop on Spurious Correlations, Incariance, and Slability, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprinl arXiv:2307009888 2023.
Burak Varicı, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based Causal Representation Learning with Interventions. In Advances in Newral Information Processing Systems Workshop on Causal Represenlation Learning, 2023.
Burak VaricI, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. General Identifiability and Achiev- ability for Causal Representation Learning. In International Conference on Artificial Intelligence and Statistics, 2024a.
Burak Varieı, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. Linear Causal Representation Learn- ing from Unknown Multi-node Interventions. In Advances in Neurol Information Processing Systems, 2024b.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-Supervised Learning with Data Angmentations Provably Isolates Content from Style. In Advances in Neurol Information Processing Systems, 2021.
Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Kekić, Elias Bareinboim, David M Blei, and Bernhard Schölkopf. Nonparametric Identifiability of Causal Representations from Unknown Interventions. In Advances in Neural Information Processing Systems, 2023.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S Yu. Generalizing to Unseon Domains: Survey on Domain Goncralization. IEEE Transac- tions on Knowledge and Data Engineering, (5(8):8052-800 8072, 2022a.
Lan Wang and Vishmu Naresh Boddeti. Do learned representations respect causal relationships? In IEEE/CVF Conference on Computer Vision and Pattern Recognation, 2022.
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-Aware Gradient Matching for Domain Generalization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Yunxia Wang, Fuyuan Cao, Kui Yu, and Jiye Liang. Efficient Causal Structure Learning from Multiple Interventional Datasets with Unknown Targets. In AAAI Conference on Artificial Intelligence, 2022b.
Nathaniel Xu, Feng Liu, and Danica Sutherland. Learning Representations for Independence Testing. arXiv preprint ar Xrx::22400 06890, 2024.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Xiaojiang Yang. Yi Wang, Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, and Junchi Yan. Nonlinear ICA Using Volume-Preservin Transformations. In International Conference on Learning Represenlations. 2022.
Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning Temporally Causal Latent Processcs from General Temporal Data. In International Conference Learning Representations, 2022.
Kui Yu, Lin Liu, and Jiuyong Li. Learning Markov Blankets from Multiple Interventional Data Sets. IEEE Transactions on Neural Networks and Learning Systems, 31(6):2005-200) 2019.
Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and Car- oline Uhler. Identifiability Guarantees for Causal Disentanglement from Soft. Interventions. In Advances in Neural Information Processing Systems, 2023.
Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang, Causal Intervention for Leveraging Popularity Bias in Recommendation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Yujia Zheng, Ignavier Ng, and Kun Zhang. On the Identifiability of Nonlinear ICA: Sparsity and Beyond. In Advances in Newral Information Processing Systems, 2022.
Xiao Zhou, Yong Lin, Renjie Pi, Weizhong Zhang, Renzhe Xu, Peng Cui, and Tong Zhang. Model Agnostic Sample Reweighting for Out-of-Distribution Learning. In International Conference on Machine Learning, 2022.
Roland Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive Learning Inverts the Data Generating Process. In International Conference on Machine Learning, 2021.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix
Implementation details
24
B
Thcoretical Motivation for RepLin
24
B.1 Statistical Risk in Predicting Interventional Latent Samples
25
B.2 Optimal ERM model
26
B.3 Minimizing Linear Dependence
27
B.4 Additional Analysis on How RepLIn Improves Interventional Robustness
31 ................................
C Review of identifiable causal representation learning
35
Differences w.r.t Domain Gencrniiatioon/ntttoDDSitttttttt
Setting
36
E Additional Results from Experiments
37
More GradCAM Visualization
37
Visualization of Feature Distribution Learned on Windmill dataset
38
Balancing Ldep and Lself during training
38
H.1 Why do the hyperparameters change belween experiments?
41
Generating Windmill Dataset
42
23

---

Published in Transactions on Machine Learning Research (12/2025)
A Implementation details
We implement our models using PyTorch (Paszke et al. 2019) and use Adam Kingma & Ba 2015) as our optimizer with its default settings. Training hyperparameterss for each dataset (such as the number data points, training epochs, etc.) are shown in Tab. 5 For training stability, we warm up Atep from O to its set value between s.V and eN epochs where N is the total number of epochs, and s and e are fractions shown in Tab. 5
Table 5: List of hyperparameters used for each dataset.
Dataset
#'Trainin samplee
Epochs
Batchsize
Initial
Scheduler
Asep
Auee
Start (+)
End (e)
WINDSILL
10,000
5000
1000
2e-3
MuluStepL. MullSStepl./uunlessooeee R(wdlesone== [1000] gamms=0.55) 0.5)
1
0.66
0.99
CelebA
30,000
2001
1000
1e-3
MultStepp. (milessoess [1000] gamma=0.1)
2
0.01
0.99
For all methods, we first extract label-specific features from the imputs and pass them through a corresponding classifier to predict the label. The architecture of the feature extractor is the same for all methods on a given dataset, except on the WINDMILL dataset. The classification layer a linear layer mapping from feature dimensions to the number of classes. The specific details for each dataset are provided below.
Windmill datasct: For ERM baselines, activation after each layer (except the last) to extract the features. However, observed that enforcing in- dependence using 1-dimensional features was difficult. Therefore, we used 2-dimensional features for RepLIn, which were then normalized to lie on a sphere.
an MLP with two layers of size 40 and 1, with ReLU
CelebA dataset: We first extract features from the raw image using a ResNet-50 (He et al. 2016 pre- trained on ImageNet (Deng al., 2009). Although these features are not optimal for face attribute prediction, they are useful for face verilication (Sharif Razavian et al. 2014). Additionally, it makes the binary attribute prediction task more challenging. We extract attribute-specifie features from this input using linear layer that maps it to 500-dimensional space.
B Theoretical Motivation for RepLin
In Sec. 3.4 we theoretically motivated RepLIn. This section explains the motivation with detailed proof.
Sketch of proof: First, we estimate the statistical risk predicting the latent variables from interventional data from representations learned by arbitrary linear feature extractors and classifiers. In this statistical risk, we will identify term that is the source of performance drop during interventions. We will then show that the optimal ERM models will suffer from this performance drop when trained on dataset comprising observational and interventional data. Finally, we show that minimizing linear dependence between interventional features can lead to robust linear feature extractors.
Entity
Notation
Examples
Scalar
Regular lowercase characters
a, 7
Random variable
Regular serif uppercase characters
A
Random vector
Bold serif uppercase characters P with subscript
A
Distribution of a random variable A
PA
Table 6: Mathematical notation used in the proof.
We follow the same mathematical notation the main paper, shown in Tab. 6 The input data X is generated as function of two latent variables of interest, A and B. There are noise variables collectively denoted by U that participate in the data generation but are not of learning interest. Our task is to predict A and B from X. and are causally related during observation. For ease of exposition, we will consider simple lincar relation B = w A. This causal relation breaks when we intervene on B. The intervened variable is denoted with an added apostrophe (i.c., B'). The data generation process can be written in the form of structural causal model follows:
Setup:
24

---

Published in Transactions on Machine Learning Research (12/2025)
A~ PA B' Pa.
XA = WAA + UA Xn wr + Un
B WABA A (during observations)
XA
B = B' (during interventions)
X
XB
UA Pu
Training: The distribution from which training data is sampled is denoted by Prrain The training data consists of both observational and interventional samples, which themselves come from distributions Pabs and Pint- We are interested the scenario where (1 - 3) proportion the training data is observational, while the remaining 3 proportion is interventional, where < 1. The training distribution can be represented mixture of observational and interventional distributions as follows:
Pernin X, A, B) = (1 - Pabs X, A, B) 3m (X, A,
Typically, we assume B < 1. We will also assume that A, B, U, and X have zero mean, so that we may use linear models without bias terms extract representations corresponding to the variables of interest and train linear classifiers on these representations. The corresponding classifiers are parameterized by c(4) and clB) The predictions are made by the classifiers from the learned representations Á = c!A)T ((A) X and trained by minimizing the mean squared error the training data,
B- cBB)
(B)) X. The models
[[|A- A 1| + B 8 |1))
CMSE = E
B.1 Statistical Risk in Predicting Interventional Latent Samples
The model predicts À and ß from X during inference. The statistioal squared error in predicting A from interventional samples can be written as,
EA = Eine (A-A) 1) = Ep (A- (A)T x )
The expectation is taken over the interventional distribution over X. [((ATT
A, U denoted by Pint ( can be
written terms of constituent parameter vectors as ( T
The predicted latent Å can hence be
written in terms of these vectors as,
A ((D)T ((d)T X = C(AIT
(X1O(1) =C
+
+ x + +
(A) ((A) + ( 0/211 GIANT
(AA ((4)( x)a ((1 w ()T 0(11)
A+ wwwnBBer
AJT 0 + (
(A)
[ aaw T ((1) A
(1) 0 B² +22
1
+ - wAch (T) 0
(Wge(()T g AB'
+2 w (2)T ( (1)
of/1) 2 (D)
ÜA+ +2
(BB
ÚB'
(8)
A2 Tl 2
0
(UBBCAiTo/()) o(a1)) 2 B + 02
:. EA = Epm.
+ 2
(11 (1- w()(()T 0(44)
(WBel1/T AB
+ Pm
[1 (1- (T) (4) ÜA +
(D) 0 UB|
where Ū= A af ((A) ofa)) ef
UB. U denotes exogenous variables that are indepen- dent of A and B. Due to interventions, we also have A II. B. The expectation of AB' will be zero since
Ug.
25

---

Published in Transactions on Machine Learning Research (12/2025)
they are independent and have zero means marginally. Similarly, the expectation of the products of Ú with A and B will be zero. Therefore,
EA = [1- ( Pa + (c(4)T((11) Pia + (WBe( ) PB. + (c)4)TT(/))
P
(9)
where 03 Epsa [A2], Pi.. En [ Pù, r [2]], and pù. pm U
Statistical risk for robust model: We are interested in robustness against interventional distribution shifts. The predictions of A by robust model are unaffected by interventions on its child variable B. If A must not depend B', then the corresponding representation FA must not depend it either, i.c., 0 0/4) must be a zero vector. Eq. (9 has two terms: E and E. Therefore, a robust model will have Ex 0
since 0(A4) 0. Therefore, showing that an optimal ERM model has non-zero ((4) is sufficient to show that the model is not robust.
B.2 Optimal ERM model
The optimal ERM model can be obtained by minimizing the expected risk in predicting the latent attributes. Since parameters not shared between the prediction of and b, consider their optimization separately. We will consider the optimization of the parameters for predicting a since we are interested in the performance drop in predicting A from interventional data.
(A), c(d)+ argmin E Exrain @(a), ((A)
(TTTTT)) x)
where Pirain the joint distribution of (X, A, B) during training. As mentioned earlier, Perain is a mixture of observational distribution Pobs and interventional distribution Pmn;, with (1 (1-3) 3) and acting as the mixture
weights. Thercfore, the training objective can be rewritten as,
e (A)* c(A)+ = argmin J(O (A) (e(())((A))
((A),
where, ((O(A), c(d)) (1 (1- B)E (A-C()TT(A) (ogGATTx²
x)² + B.ppp [A-CAATT(AATT)]] (10)
Expanding the error term on observational data, we have,
((A)T((A)T X C(AT (X1O11) + X +
=AAA(A)T 0(() ( ((4) + CC
WAAC(A)T (A) Ac (A) ((A) 1 +ClATTOIAITY
(
(A)T (A)T ( (A)T
x
= A a. AAAC (A) ((()) WB AB ((1) - -C(AITO(A)TTT
(1 wA ((A) UBWABC ((4)) A c(A)T
(AC 0 WBWABC (A) A +22
2 (1 (c- (1)T 0(A) w (4)T 0(4) AÚ
where Ù - (A)T () +
from App. B.1 Since the exogenous variable U is independent of A and B, the expectation of their products the observational distribution becomes
B
26

---

Published in Transactions on Machine Learning Research (12/2025)
Therefore,
EPoi (4--(44T0(4)) ((4)T x) x)} = (1-wacld) (A)T (a))) - -BBAABe (4) 2 Epo.. [A²] + Epo [02]]
(xe(A) 0/r11 WBWABC ( 0 p9 + ( pus, + (clatTob") ) P
(11)
Note that, P2 = Ep.. [ = Pr [U3]. and أu = Pa. [UB] similar to App. B.1 since these values are unaffected by interventions. The expansion of the error term on interventional data was derived in Eq. Thus, the overall training objective Eq. 110 can be written as,
J ( (4) c(4)) 38) ((1-wad ((A))
(
(A) 0 Pa Pà+ (cliTo pu: + (c(A)T(/)) pūo) p' (unc(A)T!!A))
+ 3 ( (1-
0 PB + (c pu. + (c(ATT PUo
We set 1 = c(A)T (()) and 2/2 ( Since ERM jointly optimizes the feature extractors and the classifiers, no unique solution minimizes the prediction loss. For example, scaling the feature extractor parameters by an arbitrary constant scalar and the classifier parameters by 1/7 will give the same error. Therefore, we can optimize ((O(4), ((A)) over 1/1 and 1/2 similar to (Arjovsky et al. 2019).
((O(A), ((4)) = (1- (11 -Ath ABWAB Pd + ViPU. + v2/00
+ 3 (1 (1- wAt)) 2 PA W + Pu + v22002)
(12)
The optimal values of 1 and tha are the stationary points of (() (Al) (denoted by J for brevity). Thus the optimal parameter values can be solved for by taking the first-order derivatives of J w.rtt. un and 2/2 and setting them to zero.
0J = 2(1 - 3) (- (1-1-u0
al--waY WArt - (BWABU>) WAPA + vIP(,) + 2B (- (1- WAVI) WAP + ))
0+1
0J
2(1 - 3) (1-wAhh (1 WB + v 0xh2
2B (wiweri + p22pūn))
Setting 04h
aJ 0, we have,
002
(wAP3 + pū,)) 4
+(1 - Ay + (Bw + (1- w + pú,) v22 -(1 - ()WBWABP =
AA =0
(1 8 MyAAB/ABPA*
The equations are of the form unth + v1 + w = and 42 + 02th2 + W2 = O. We can solve for de as ab - Since we are only interested in probing the robustness of ERM models, we will chock if
2 is zero instead of fully solving the system of linear equations. E in Eq. is zero if t/2 = 0, i.e. if
u2ul- W1 =1 0.
W2lt with -(1 -B)BBWA (w2r3s pū.) p 4(1 3)
-(1 - AB
Unless the training data entirely composed of interventional data (i.e., 3= 1), w2 woug ug is not zero. Thus, the optimal ERM model is not robust against interventional distribution shifts.
B.3
Minimizing Linear Dependence
In Sec. 3.3 we showed that dependence between interventional features correlated positively with the drop
in accuracy interventional data. We will now verify if minimizing dependence between interventional
27

---

Published in Transactions on Machine Learning Research (12/2025)
features minimize the drop in accuracy. For case of exposition, will minimize the linear dependence between interventional features instead of enforcing statistical independence. The interventional features are given by FA- = ( X and FB ( X.
( FA ( = A
0 B
XA XB
x10 XB0(4)
FB O X
0 A
B B
XA XB
*x0
(B) XB0B
(B)
To define linear independence between interventional features, we use the following definition of cross- covariance from (Gretton ct al., 2005):
Definition 1. The cross-covariance operator associated with the joint probability pxy is a linear operator Cxy + F defoned as
Cxv Exy [(x(X) - pxx) (+(Y) py)l
where G and FF are reproducing kernel Hilbert spaces (RKHSs) defined by feature maps & and up respectively, and & is the tensor product defined as follows
(S 60 g)h = fgg M)g for all h E e
where (5.) is the inner product defined over g.
In case, instead of RKHS, we have finite-dimensional feature space Rd. Therefore, we have the cross- covariance matrix as follows,
CXY = [x(X) @ (()) Exy [o(X)o(Y)]
given that the feature maps have zero mean. Following the definition of HSIC (Gretton al. 2005 linear dependence in the finite-dimest dimensiima case between and Y is defined as the Frobenius norm of the cross- covariance matrix. Therefore, define the linear dependence loss between the interventional features as,
Laep = Dep (F FB) = Pm FAF
(13)
Leveraging the independence relations during interventions, we can expand Eq.
13 as,
Ep... Pm
FAFT
= Epin
(xa0k²) + +X (x + +XBO
Epan X3
( (B)T +XIX8O B)T + +XAX A (
BIT
0 (B))
+
(whph + p 1 (1) B)T
BYT + (Wipt + p 0 ( g B)T
(wāp3 + p (B)
B)T + (wit + , 00 BBT
Laep
In the last step, all cross-covariance terms zero due the independence of the corresponding random variables in the causal graph. The dependence loss is the Frobenius norm of sum of rank-one matrices o(4) (BBT and o(4))(B) Consider the following general form: Z ==at ab + ed Then Zig a,b, + c,dj.
2 F Z
Σ (arb; + czd,)²
|Z| 2 is sum of squares and thus zero a,b, + + = O, Vi,j. Therefore, Ldep is minimized when
F
0/ (4) ((B)
((A)
Bs o( 0, Vi,j. The potential solutions that minimize Laep are (1) 0 o( 0%/4) 0
28

---

Published in Transactions on Machine Learning Research (12/2025)
(B) = 0, (2) 0/10) = and 70 = for some 7 A O, and (3) 0 0 0 0, and 0/²²) or 0/ 0. The former two solutions result in trivial features and will increase the classification error. The latter solution contains four possible solutions, out of which two solutions result in trivial fealurres. Solutions resulting trivial features are unlikely to occur during optimization due to a large classification error. Therefore, we need to consider only the remaining two solutions.
The possible solutions are: (1) 0 = 0, 0 O, and (2) 0// = 0, 1 = 0. Intuitively, in the former solution, A and B will be predicted using XB and Xa respectively, and the latter solution corresponds to a robust feature extractor that minimizes the reducible error in Eq. 19). We will compare the predictive error achieved by these solutions compare their likelihood during training.
Recall the expression for training error in predicting A from Eq.
12
Ja(O(A) (4) ((()))) = (1 B) (1 - WAR UBWABWAR) Pd - EAIPU, + VAAPUU))
+ 3 - (A-WA))))* PA
V WAP + SAPU + +22Pdnn)
(1 - () (1 (1- (A-WARAI -W ph.)
+ 3 (11 - =iA-A)))) PA
" A2 + WAIPU + A
We use A and 42 instead of 1 and t/e respectively to denote the parameters for predicting A. A similar expression can be written for the error in predicting B with dBl and d'B2 denoting the parameters for predicting B.
(n( ( = (1 -11-8] 3) ((1 WAYBAI - (BWABDD2) )² "a vird + v220)n)
+ 3 11113 + (1 wBthB2 )2 PBB + W BII + tbba)
(1 - 3) (1 WAWBBI 7DWABUBE) p3.
3 ( + wDdm2) /3 + BIPU + V
When 0/a11 = 0, 0 0: In this case, A = 0 and th = 0. Therefore, the predictive error during training for each latent variable can be written as,
Case 1:
JA = (1 - 3) (W - 1)² p3 3r3 + Bub +
JB (1 3) (WALBI WAB)² pa 3w U Bpb. + ViIPi
The optimal values of A2 and dDι can be obtained by equating the gradients of RA and Rg to zero.
OJA 2(1 B)WBWAB (WBWABWAR 1) Pa + 28 2
(1- (1-3)WBWABP
J. 1.122
(1 B)w u W
3w
BPB.
PUa
(1- 3)² (Bw 211 rt2) + 3p3
Ja = (1 - B)w A
3w P PU
0Jg = 2(1 3 (WARM WAR) p3 + 28w mm 220mI =
0ym
(1 (-
:. 1/
w3p2 + Pē,
(1 () (B pda)
JB
+ 3p3
w 3AP +
29

---

Published in Transactions on Machine Learning Research (12/2025)
The combined training error for this solution is,
= + JB
(1- (1-8)p² (SwBar3 + ttu))
+ Bp'
= (1 - B) w + Bw PUB
+ B)w AB (Bwir + Pú,))
+ BPB
(14)
wxph + PU,
Case 2: When 0591 = 0, ( = 0: Here, 12 = and eb = 0. The prodictive error during training for each latent variable can be writlen as,
JA = (WARA) - 1) AAPUA
JB = (1-3) + Bpp.)) (W - 1)² + B
We follow the former procedure to estimate the optimal values of thai and d82
AJA
A (WAR p2 + 2 A1PL -0
WAP
AP's +
PaAPU
JA
2 + PUAa
0JB
28B ((1 B) 3 BB3 + Bptt)) (WBB)22 1) + 2/
0182
- B wi
AB A BWBPB.
:. 0i
(1 - B) ABP + Sw + PUU
((1 3) 3) + 30 P
JB (1 - 3)w A + fu B + PU.
The combined training error for this solution is,
J J + J
PAPU
((1 (w- 3p3x)) p
+
(15)
w AA PU.
(1 B) A Bu PUn
Comparing Ji and J2.
J - J3 (1- 3) 2 PT (1- 8)-3 - (1 B)w² P.P - Bp
(1 3)03 w/ 2B + 3.0 + P
(1 - B) w 2BP2 + +11 - 3B) - -aaPe +3(03 + p%x)
22p2 Pt.
Simplifying the above expression, we get the condition that J J > 0f satisfics the following conditions:
(1) 10121- : (2) 8 2 min tp w "inn The conditions imply that enforcing linear indepen-
dence results in robust feature extractors when enough interventional data is available during training.
However, this is only a sufficient condition that strictly ensures J J2 > 0. practice, B could be much lower, especially when the total loss is of the form Liotal AMSELMSE Adep where AMSE and Xdep
30

---

Published in Transactions on Machine Learning Research (12/2025)
are positive hyperparameters. We verify this empirically by randomly setting the parameters of the data generation process and plotting the predictive errors Ji and Jz for different values of 8. We calculate Ji and J for 5000 runs (shown using thin curves) and plot the average error (shown using thick curves) in Fig. 13 We observe that the average value J is always higher than that of J2 for all values of B. But,
when 3 + 0, their average values gel closer to each other.
J
J
Training prediction error
8
6
0.0
0.2
0.4
0.6
0.8
1.0
Proportion of interventional data (5)
Figure 13: Comparing Jf (Eq. (14) and J1 (Eq. 15 as functions of 3 for 5000 runs with randomly sampled data generation parameters. We show individual runs using thin curves and the average orror values using thick curves. We only show the errors from a few randomly sampled runs for visual clarity. We observe that the average value of J (shown using thick red curve) always higher than that of J (shown using thick blue curve), indicating that enforcing linear independence between interventional features is more likely to obtain robust feature extractors than degenerate solutions.
B.4 Additional Analysis on How RepLIn Improves Interventional Robustness
In the previous section, we demonstrated, using a linear 2-variable causal model, that enforcing independence can provably improve statistical test-time risk over the interventional distribution for a sufficient proportion of interventional data in the training distribution. In this section, we will further show how enforcing dependence improves test-time risk over interventional distribution. We will limit this analysis to linear models, but extend it to include multiple latent variables (including exogenous noise) with possibly nonlinear causal relations between them and imperfect interventions.
Similar to our former setups, the observable data X is a function of the latent variables of interest, Z, and exogenous noise variables U, = gx U). Here, can be considered a concatenation of individual latent variables, such as A and B that we used in our previous analysis. 9x could be nonlinear function. Although linear model to predict Z from X may be insufficient for nonlinear 9x, the following analysis is still valid. Consider the task of predicting one of the latent variable clements in Z, Zz, using lincar weights wi as 2, = = X.
To learn this predictor, we have access to a training distribution Purin which is mixture of the observational distribution Pobs and the interventional distribution Pint as follows:
Pirain [X (,Z) = 33) (x Z) BB Z)
During observations, some latent variables are causally related to each other. For this analysis, let Z1 be a parent of multiple child variables during observation. Here also, the causal relation from Zi to its child nodes may be nonlinear. These child variables are also affected by external noise variables. Similar to our previous setups, during interventions, we intervene on one or more of these child nodes, rendering them independent of Zz. To understand how enforcing independence between the predictors for Z₁ and its child nodes over interventional distribution improves robustness, we will compare the weights wy obtained through vanilla risk
31

---

Published in Transactions on Machine Learning Research (12/2025)
minimization (minimizing only the prediction error), denoted by 1 and the weights obtained through the proposed approach, denoted by Idop,,1 against the weights of robust linear predictor, denoted by Wroh,1
Robust linear predictor: The weights for a linear predictor that is robust against interventional distribu- tion shifts can be obtained by minimizing the prediction error over a hypothetical training distribution that consists of only interventional data. Here, Pint the interventional distribution where all child nodes of Z are intervened on.
[21 rob, x
x)]
w rob,
argmin Ein.
Myobb.1
The closed-form solution to the above equation, under assumptions of zero-mean latent variables and a mean-preserving mixing function 9x;
w Pob,1
Cx1
Xin Cz X
(16)
where Cx... En XXT is the covariance matrix of the input data X over the interventional distribution, and CinXin Epm.. [1] is the cross-covariance vector between X and Z1 during intervention. Note that 1ob,1 are the weights of a robust linear predictor, irrespective of whether the mixing function gx is linear or not.
Optimal linear predictor under vanilla risk minimization: The optimal linear predictor for Z1 under vanilla risk minimization distribution. The weights of this optimal predictor will appear similar to Eq. [6 except the involving terms will be computed over the training distribution. The optimal weights are
be obtained by minimizing the prediction error for Z1 under the training
Wrm.1 X.ram CZ2 Xariin
(17)
Cx Punii X and cz, Xteoi P [X 21,, similar to the corresponding quantities defined earlier for interventional distribution. Due the discrepancy between training and intervention distributions, the optimal linear predictor under the training distribution will have excess risk compared to the robust linear predictor in Eq. 16 This excess risk can be quantified as
Cexcess (m)m))) = Ep (witm,1X
T x)) 1ob.,1
En
(wrrmu) wibb.)) XXT (wirm,1 witbh.))
(w wfob,1) Oxm. (Wor,,1 wrob,1) = || - wfob, 1 ll
(18)
How does enforcing independence over interventional distribution help? For this analysis, we will use the simplified version of RepLIn that we used for our analysis in the previous section, which consists of only the dependence loss. Specifically, we will minimize the squared covariance between the predictors for Z and its parent nodes over the interventional distribution when we intervenc on Z,. Recollect that the latent variables had zero mean, and the mixing function was mean-preserving. Combining this dependence loss with the prediction loss from vanilla. risk minimization, our training objective becomes the following.
(uri,..., wdz) Epacin
Σ w x)²
Σ
E2
[w X . un X
jECh(1)
Σ
2 cz + u Cx wi 2un cz, Xteain
(wn) Cxim wyy²²
jEPa(1)
prediction loss
dependence
where Pa(1) is the of Zi's parent nodes' indices. The dependence loss in the above equation is equivalent to HSIC without any additional nonlinear feature extractors over the predictors. Computing the gradient of
32

---

Published in Transactions on Machine Learning Research (12/2025)
J (W1,..., Wdz) w.r.t. wi and equating it to the zero vector gives the following:
0J x. w1 + jECh(L)
Σ
wi Cxi.
wy
Cx... wi 2cz, Xunin = 0
Own
X Xanina
wi
Σ
wi Cx Wy Cx Wi = CZ₁y X train
jECh(I)
wi
+
w Cxm: Wj- Xitaia Cx.: Wj
Cx
Avain CZ₁
X
(19)
)ECh(I)
w vrm,1
Weorr.1 correction vector for M1
Note that the RHS of Eq. 19 is w v from Eq. 17 Thus, we obtain wy w Womm1 Weorr, This means that enforcing independence between Zi and its child nodes over the interventional distribution essentially adds a correction vector to the optimal predictor under vanilla risk minimization.
Can we obtain an analytical solution for waep., ?: We may write similar equations as Eq. 19 for the predictors of other latent variables in Z. In a 2-variable case, these equations would be those of a hyperboloid, implying infinitely many solutions. A unique solution may be arrived at, although not analytically, through additional commonplace regularization such as La regularization or SGD's implicit regularization, resulting in minimum norm solution that satisfies Eq. (19h. Even with heuristics such selecting minimum- norm solution, it is not casy to obtain an analytical solution for Wdep. since the correction vectors are interdependent on the weights of the predictorr for other latent variables.
Let 1dep,1 be a solution to Eq. 19 The excess risk for Wdep,1 can be written similar to Eq.
18 as
Cexcess (1 = ||addep,1 - Woob,1 Cx...
(20)
We can conclude that enforcing independence between predictors over interventional distribution improves robustness if the excess risk for 1 is lower than that for 1 i.e., if Cexcess (Waap.1) ) < &excess (we In the next part, we will empirically observe the effects of imperfect intervention and the amount of inter- ventional data in the training distribution on the utility of RepI.In.
Empirical Analysis of 1 The empirical analysis in this section will focus on two aspects: (1) the effect of imperfect interventions, and (2) how the correcting vector acts on Waep.1. To answer both questions,
we construct a l-letent variable toy dataset. The latent variables are Z₁ *N and Za Ø2 21 + e,
7
where σI, 02 ~ ()) and e ~ A'0 4). The observable data X then constructed as X W
Ze
U
where ~ (O, 10-2) is the exogenous noise variable and W E Rdxx3 x3 is a randomly generated orthogonal matrix that acts as the linear mixing function. dx is also chosen randomly from ( 203.
We model the imperfect intervention using imperfectness hyperparameter 17 by essentially replacing each interventional sample in the training distribution with an observational sample with probability. During training, we minimize dependence between predictors over this imperfect intervention. As n increases, the proportion of causally related latent variables masquerading as independent variables increases, and enforcing independence between predictors over this imperfect intervention can then hurt the predictive performance. Our intuition stems from the boundary case of 1) + 1 (all interventional samples replacod with observational samples), where only random predictors can satisfy the independence condition that we aim to achieve.
Fig. 14 shows the results of RepLIn imperfect interventions. In Fig, 14a we compare the test error for RepLIn for various values of π) against the test error of a vanilla model (shown in black) trained on a dataset with observational and perfect-interventionn are either lower or equal (near the boundary values of B) compared to the vanilla predictor when 17 + 0. The errors match as respectively. As the interventions become more imperfect, the test errors for RepLIn increase. In Fig. 14b] we view the difference between the excess risks of vanilla predictors and RepLIn predictors for various values
data (17 = 0). We can see that the test errors of RepLIn
and 3 - 1 due to the unavailability and abundance of interventional data,
33

---

Published in Transactions on Machine Learning Research (12/2025)
0.8
0.75
Rxx (oapp)) Re Wea.ll
0.50
0.66
Inpedfoctnes 1
Test Eroo
0.25
Imperfoctmess
0.4
0.00
0.25
-0.50
0.0
-0.7
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
B
(a) Test error RepLin for various values com- (b) Difference between excess risks for a vanilla predictor pared to a vanilla predictor trained on training distri- and RepLIn predictor for various values of bution with 77 0.
Figure 14: We examine the effect imperfect intervention on RepLIn by (a) comparing the crror of RepLIn predictors trained distributions with various imperfectness probability 7 against vanilla predictor trained distribution with observational and perfect interventional data (1 0), and (b) comparing the errors for RepLIn and vanilla predictors trained on distributions with various values of 17. The plots indicate that imperfect intervention can hurt RepLIn's performance, especially for higher values of B.
of 1.. Here, note that RepLIn consistently outperforms the corresponding vanilla predictor for most values of 17 at lower values of B. As y) increases, RepLIn begins to perform worse than vanilla predictors for higher values of B, and RepLIn eventually consistently underperforms vanilla predictors for - 1.
Since RepLIn relies on enforcing independence between samples where the underlying variables are truly independent, is naturally prone to imperfect interventions, particularly for large values 3. However, we envisioned RepLIn for scenarios where interventional data is scarce (3 < 1), and where targeted approaches improve robustness are desirable. In this regime of 3 < 1, our results indicate RepLIn performs better than vanilla predictors when intervention noise v) is considerable.
Conclusion:
5-3.22
0-4.44
18
10
3-00
8 0,67
3- 0.38
8- 0,59
8- 1.00
The weights of a linear RepLIn predictor the sum of the weights of vanilla predictor and correction vector (Eq. 19h). In this plot, black, red, and green arrows show the weights of robust, vanilla, and RepLIn predictors. The negative correction vector is shown in magenta. We see that the correction vector points nearly orthogonal to the robust. predictor (black), and its magnilude decreases as 3 increases, when the vanilla predictor (red) approaches the robust predictor.
Figure 15:
We repeat this experiment with dx so that we can plot the resulting predictor weights, as well as the correction weights. In Fig. 15 we plot the weights of robust (black), vanilla (red), and RepLIn (green) predictors for various values of B over the density plot of X samples. The observed samples from X are
34

---

Published in Transactions on Machine Learning Research (12/2025)
also shown underneath the contours of density. The samples are colored based on their corresponding value of Z1. Among the subplots, B = 0 corresponds to the case without any interventional data, where red and green arrows overlap. In the remaining scenarios, the RepLIn predictor (green) lies between robust (black) and vanilla (red) predictors. From Eq.
19 we know that d is the sum of the vanilla predictor weights 1 and the negative of the correction vector woorr. Therelore, we also plot the negative of the correction vector using magenta in Fig. 15 We can see that 1eorr.11 is always nearly orthogonal to the robust prodictor weights, and indeed acts as a correction vector that "pushes" wirm1 towards Woob. As B increases, the vanilla predictor improves its performance, as saw in Fig. [4 eventually matching the robust predictor. The magnitude of the correction vector also subsequently reduces as B increases.
C Review of identifiable causal representation learning
The primary objective of identifiable causal representation learning (ICRL) is to learn a representation such that it is possible to identify the latent factors (up to permutation and elementwise transformation) from the representation. These methods are commonly built upon autoencoder-based approaches and learn generative ropresentations. The advantage of learning a causal representation is that the decoder then implicitly acts as the true underlying causal model, facilitating counterfactual evaluation and interpretable representations.
Locatello et al. (2019); Khemakhem et al. 2020 showed that disentangled representation learning impossible without additional assumptions on both the model and the data. Some of the inductive biases that have been proposed since to learn disentangled representations include auxiliary labels (Hyvarinen &
Morioka 2016 Hyvarinen et al. 2019 Sorrenson et al. 2020 Khemakhem et al.
2020 Lu et al., 2022 Ahuja
ct al.
2022b Kong ct al.) 2022), temporal data. (Klindt et al. 2021 Yao et al.
2022 Song et al.
2023 and
assum cosumptionsoo on the mixing function (Sorrenson et al.] 2020 Yang et al. 2022
Lachapelle al.
2022 Zheng
et al.
2022 Moran et al.
2022
Use of interventional data: Some works also use interventional data as weak supervision for identifiable
representation learning (Lippe et al. 2022b Brehmer et al.) 2022 Ahuja et al.) 2022a 2023 Varieı et al.
2023 von Kügelgen et al. 2023). Lippe et al. (2022b) learns identifiable representations from temporal sequences with possible interventions at any time step. Similar to our setting, they assume the knowledge of the intervention target. They also assume that the intervention on latent variable at a time step does not affect other latent variables in the same time step. Lippe et al. (2023) relaxes the latter assumption as long perfect interventions with known targets are available. |Von Kügelgen et al. (2021) Zimmermann et al. (2021) b showed that self-supervised learning with data augmentations allowed for identifiable representation learning. Brehmnr et al. (0022) use pairs of data samples before and after some unknown intervention to learn latent causal models. Ahuja et al. (202aa) learns identifiable representations from sparse perturbations, with identifiability guarantces depending on the sparsity of these perturbations. Sparse perturbations can be treated as a parent class of interventions where the latent is intervened through an external action such as in reinforcement learning. [Ahuja et al. (2022b) use interventional data for causal learning for polynomial mixing functions, under some assumptions on the nature of support for non-intervened variables. Varic et al. (2024a) relaxes the polynomial assumption on the mixing function and proves identifiability when two uncoupled hard interventions per node are available along with observational data. Varie et al. (2023) learn identifiable representations from data observed under different interventional distributions with the help of the score function during interventions. vvo Kügglgen et al.] (2023) uses interventional data to learn identifiable representations up to nonlinear scaling. In addition to the above uses of interventional data, a few works (Sacngkyongam & Silva, 2020 [Sannkyyongam et al., 2024 Zhang et al. 2023) have also attempted to predict the effect of unseen joint interventions with the help of observational and atomic interventions under various assumptions on the underlying causal model.
Difference from our sctting: The general objective in ICRL is to "Icarn both the true joint distribution over both observed and latent variables" " (Khemakhem et al. 2020). In contrast, the objective of our work is to learn representations corresponding to latent variables that are robust against interventional distributional shifts by leveraging known interventional independence relations. We pursue this objective in the hope that, as large models (Radford et al., 22021] Brown et al.] 2020) Touvron et al., 2023 Dehghani
et al. 2023) become more ubiquitous, efficient methods to improve these models with minimal amounts of
35

---

Published in Transactions on Machine Learning Research (12/2025)
experimentally collected data will be of interest. Stated more formally, full identifiability of the underlying causal model is not in our interest, robustness to interventional distribution shift can be achieved without full identifiability. For instance, consider the following setup: Let A = [A1,A2] cause B during observation. Here, A1 is a binary variable (also, the class we are interested in prodicting) and A2 is a continuous variable from a Gaussian mixture with modes. The mode is decided by A1, and therefore informs the class. The observed data is X = [ XA, X)] where XA, depends only on A1, XA, only on A2. and XB only on B. Suppose the relations from the latent variables it is possible to learn A1 and B from X, but not A2 (say, due to noise or information loss in the mixing function). The discriminative task here is to predict which class A belongs to. Here, RepL.n can learn robust representations that can fully predict the class of A (through A1), but is not fully identifiable since it does not have information about A2.
the corresponding observed variables are such that
Morcover, the representations learned by ICRL methods usually have permutation ambiguity. That is, the representations are disentangled but not mapped to the semantic attribute that we wish to predict in a rewm task. Our approach overcomes this ambiguity by explicitly learning altribute-specifie representations.
D Differences w.r.t Domain Generalization/OutttoooDDDiiiiii Setting
The setting of RepLIn differs from domain generalization (DG) or out-of-distributio (OOD) tasks due to the assumptions in our setting, which RepLIn exploits to get more robustness benefits. Expressed in terms the random variables A, B, and X in our problem setting from Sec. 3 the task in DG to predict some variable of interest A from the observed data X such that the learned model can generalize to unseen domains (Wang) et al. 2022a Ding et al., 2022b To ensure robust prediction, multiple sets of training data that vary in the
their distributions of some variable B are provided. DG does not typically have any requirements learned predictor, apart from its transferability to different domains. As a result, the learned predictor may store information about the environment to improve their predictions (Rosenfeld et al. 2022). In contrast, RepLIn removes information from intervened variables to improve their robustness of predictors that use RepL.In representations. In summary, RepLIn operates on a setting similar to DG DG/OOD, but under more assumptions, with the goal of obtaining more trustworthy representations. Tab. 7 shows the differences between the DG framework and ours. The first two rows show the differences in scttings, while the last two rows show the differences in learned representations.
Differences
DG/OOD
RepLIn
Relation from A to X between domains
May or may not change
Does not change
Is A independent of B in one more domains?
May may not be. It is also possible that A is always inde- pendent of B.
A - B in observational data. and A independent of B in in- terventional data.
Can accommodate more than
No. B is not of interest.
Yes. Learns representations for B as well.
one B?
Is the representation learned for free of information from B?
Not necessarily. Some DG meth- ods are designed to remove in- formation from B, while others are not (e.g, DARE (Rosenfeld ct al. )
Yes, the dependence loss ensures that the features for A are free of information from B in the in- terventional data.
Table 7: Differences between the problem settings of domain generalization and RepLIn.
In addition to domain-level differences, training with group-inbalanced data also leads to models that suffer from group-bias during inference. In such cases, resampling the data according to the inverse sample fre- quency can improve generalization and robustness. Studics such as (Gulrajani & Lopez-Paz) 2021 Idrissi
et al. 2022) have shown that ERM with resampling is effective against spurious correlations and strong baseline for domain generalization. Recent works such as dynamic importance reweighting (Fang et al. 2020
36

---

Published in Transactions on Machine Learning Research (12/2025)
SRDO (Shen et al. 2020], and MAPLE (Zhou et al. 2022 learn to resample using separate validation set that acts as proxy for the test set. However, learning such a resampling requires large dataset of both observational and interventional data, which is often not practically feasible. In contrast, we will exploit known independence relations during interventions to improve robustness to interventional distributional shifts.
E
Additional Results from Experiments
As mentioned the main paper, our objective is to improve the robustness of representations against interventional distribution shifts. However, this robustness might come the cost of observational accuracy since it removes spurious information that gives better performance obscrvational data. In this section, report the results of the baselines and our methods on WINDMILL, CelebA, and CivilComments datasets. The accuracies of various methods in WINDMILL, CelebA, and CivilComments datasets are given in Tabs. to 10 respectively.
Machod
- 0.5
033
1 0.1
- 0,05
0.01
ERM
05.85 1:84
08.06
20
90.70
90.92 0.02
90.05 :: 0.01
70.87
50.52 +
00.35
ERM-RRs
ERMM-Hos Mes-
78.70
3.19
71.19
8.25
76
1.54
71.01 4: 2.55
70.20 4: 8.738
I[MvI
IRMv
78.24
0.79
74.83
1.74
78.61
2.24
76.25 + 88
71.75
2.005
Fish
05.84
0.42
05.37
0.36
96.42
0.30
96.83 0.51
04.22
2.53
GRo
62.02
3.00
84.40
2.72
85.35
2.35
84.25 0.91
00.25 1.11
GroupD8O
SAGAI
91.77
0.02
95.17 17
0.71
94.13 1.68
96.61 0:00
94.01 1150
SAGM
78.43 2.37
79.05
2.25
70.96
4.30
79.96
72.51
131.0
0.95
0.30
DIWA
76.61 2.10
76.71
1..0
76.10
70.88
1.83
75.30
1.13
Repldn
0.55 05.07
07.83
90.10
BopLin
Repilla-Ros.
95.77 0: 0.68
90.38 50 1.08
96.00 0.35
06.51 1.71
Reppllo-Res.
88.00 0.96
88.00
1.04
87.00
1.30
86.38
0.85
T8.41
(a) Observational
(b) Interventional
Table 8: Accuracy of various methods used in Sec. 5.1
Methorl
0.
Methol BRM-Re Res
8-
8- 04
ERSU Res.
0008
81 0:00
00:59 0007
0m 87.07
018
80.6 0.1
50.0 0.77
7661
714 78.36 0.00
RepLlin Res.
2015
0:331
8038
ReplAtoRes
81.97 011
31020177
51.5-015 018
75.77 000
(a) Observational
(b) Interventional
Table 9: Accuracy of various methods used in Sec. 5.2
Meshod
5-
8
8-01
8
Method
8
8
8
ERM Res.
81.20 = 0.12
81.77 0.14
79.78 1:088
79.97
0.12
79.13 0.99
ERM Res.
74.11
78.29 1: 0.22
7200 1 0.18
71.78 1.122
69.80 0.45
Repl.n Res.
79.27 0.09
80.16 0.12
77.85 1.00
77.84 0.12
7851 1 0.16
DepLin Res.
75,30 1 0.57
75,81 L. 0.51
7200 1.25
71.70 0:110
60.99 10.99.10 0.50
Observational
(b) Interventional
Table 10: Accuracy of various methods used in Sec. 5.3
F More GradCAM Visualization
Figure 16: GradCAM visualizations for the samples that were correctly classified by ERM, but not by RepLIn. The top shows the visualizations for ERM, while the bottom row shows that for RepLin.
37

---

Published in Transactions on Machine Learning Research (12/2025)
We show more GradCAM visualizations to illustrate the differences between the representations between FRM and RepLIn. In Sec. 6.1 we compared the GradCAM visualizations for those samples that were correctly classified by RepLin, but incorrectly by ERM. Here, we visualize the GradCAM from samples that were correctly classified by ERM, but incorrectly by RepLIn. In Fig. 16 the top row shows the GradCAM visualizations of ERM, while the boltom row shows the visualizations for RepLIn. These samples are chosen randomly. Although these samples were incorrectly classified by RepLIn, the attention maps RepLIn for most of the shown samples focus on the mouth region. On the other hand, attention maps of ERM do not focus the mouth region often.
G Visualization of Feature Distribution Learned on Windmill dataset
In this section, we compare the feature distributions learned by RepLIn on WINDMILL dataset against all the baselines from Sec.
5.1 The feature distributions are shown in Fig.
Top row: When 0,
Bottom nww When A
Deasity
Density
Densiyy
Donaity
Dow
Inclinstion
Inclination
Inclination
Donity
Density b
6
Density
Demity
Donityy
Inclination
Inclination
Inclination
Azimatt
(a) ERM
(b) CRRM-Resampp
(c) IRMv1
Densty 5
Denalt
Density
Inclination
Azimath
Inclination
Azimuth
Inclination
Azimath
Donsty
Donsty 1 6
Density
Inclination
Azimath
Inclination
Azimuth
Inclination
Azimath
(d) Fish
(e) GroupDRO
(I) RepLIn
Dansty
Density
Inclination
Azimanh
Density
Demity
Inclination
Azimuth
(g)
Figure 17: Visualization of interventional features learned by various methods WINDMILL dataset.
H Balancing Ldep and Eeer during training
The goal of RepLIn learn robust discriminative representations corresponding to variables of predictive interest such that each representation contains only the information from the latent variable it models, especially when these latent. variables causally related. This goal must be evaluated two fronts the
38

---

Published in Transactions on Machine Learning Research (12/2025)
absolute utility of the representations for downstream tasks and performance equity between observational and interventional distributions. We quantified these evaluat ions in our experiments through the performance on interventional data and the relative accuracy drop between observational and interventional distributions. Our proposed loss functions also reffected these objectives: (1) self-dependence loss (Ceelt) maximizes the information that representation learns about its corresponding latent variable, and (2) dependence loss (Ceep) minimizes the information shared by the representations of causally related latent variables during interventions, to obtain distributionally robust representations.
However, feelr and Ldep have somewhat conflicting objectives. Minimizing Ceelf maximizes the statistical information shared between latent variables and their corresponding representations. It docs not discrim- inate the nature of this information and, thus, could include information about the child variables in the representation when minimized on observational data. Minimizing Edep ensures that the interventional rep- resentations corresponding to independent variables do not share any information, regardless of whether these representations contain any discriminative information useful for predicting their corresponding latent variable. Thus, fundamenially, Lastt enriches the information in the represenlations, while Ldep removes the information from the representations. If these loss functions are not balanced during training using their respective hyperparameters Aaelf and Adep the learned representations may not be robust and discriminative.
We experimentally demonstrate the above statements with the help of a synthetic dataset with linear relations between variables, similar to the one used for theoretical analysis in Sec. 3.4
Experiment setup: Our dataset consists of the high-dimensional observed signal X E R.0 from which we must predict two latent variables of interest, A, B E TR During observation, A B in the underlying causal graph with the following lincar causal relation between them.
A ~ N(0,, To)
(Tp is p p identity matrix)
6 ((0, To))
(Noise in observational relation)
= V V0.1e
To collect interventional data, we intervene B and set it to independently sampled B ~ (O, Li)). During intervention, A IL B. A and B, along with exogenous random variable U ~ (0, create the observed signal X from which we are tasked with learning representations corresponding to A and B. Formally,
n ~ (/0,0.25/
(Noise in the mixing function)
Å = +n
x= = [ B u]
(21)
X = W + Z,
where W 100 and Z E R 1000 are the lincar coefficients of the mixing function whose entrics are independently sampled from (o, 1). During its sampling, we verify that W is full-rank matrix to ensure that a linear model can predict A and B from X. Note that the noise n added to A has higher variance than the noise e in the observational causal relation. This would prompt the model to learn shortcut (Geirhos et al. 2020) and rely on the information from B to predict A. Since we know the variance of the noise added to A, we can also compute the statistical error of a robust linear model.
Our model consists of linear layer each learn the representations corresponding to A and B, and linear layer each to make the final predictions Å and B from their respective representations. The model does not have any non-linear activation function. The models are trained by minimizing the mean squared error between their predictions and the ground truth, in addition to Laep and Laeft weighted by their corresponding hyperparamelers Adep and Awelf respectively. Each batch comprises the entire training dataset. For each run, we first generate a different random seed $yun that affects the sampled values for W, Z, A, and B. Random values for Syun are generated using a meta random seed Smeta obtained from the system timestamp during the experiment run. We also use smeta to randomly sample Adep and Agelt from their uniform distributions in their log space. In total, 27,748 random hyperparameter settings were sampled.
39

---

Published in Transactions on Machine Learning Research (12/2025)
39
0.92
14.
10
0.71
10
1.08
10"
128
10
87
10-1
if
0.53
0833
1.50
1.56
10P
500
5 10-1
05
0392
0.312
10-2
0.88
10
10
03
10
0.50
0.32
10
0.72
0.56
1.6
0.1
02
0.0
(a) Observational error in (b) Interventional error in (c) Relative increase in error (d) Dependence predicting A
between
predicting A
representations
Figure 18: Results of RepLIn models trained with different values for the hyperparamcters Adep and Awelf- The healmaps show the variations of interventional accuracy (left) and relative drop accuracy wetw observational and interventional distributions (right) with the hyperparameters.
The results of our experiments are shown in Fig. 18 In the results, we plot and analyze the prediction accuracy on A since we intervened B. To obtain contimuuus- plots, we interpolate between the sampled pairs of Adep and Aselt through triangulation. We make the following observations from the results:
(1) Small values for Adep and A RepLIn behaves similarly to vanilla ERM method as Adeps Aselt + 0. In Fig. 18 this setting corresponds to the lower-left quadrant of each plot. Due to the designed difficulty in predicting A from X, the model uses information from to predict A, resulting in a low error in obscrvational data (Fig. 18ab and a high error in interventional data (Fig. [8b]. Statistical dependence between representations during interventions measured using NHSIC is also high (Fig. 18d as expected.
(2) Increasing Adep alone: When Adep is increased without changing Aselt dependence between repre- sentations of interventional data decreases, as expected. However, incrcasing Adep sometimes provides only limited reductions in interventional error, as seen in Fig. 18c For instance, increasing Adep from 10-3 to 1, while keeping a constant Awelf = 10-3 slightly decreased the error on interventional data from 1.89 to 1.76, while nominally increasing the error on observational data from 0.127 to 0.129. This shows that while minimizing interventional dependence helps learn robust representations against interventions, the benefits in performance may be marginal.
(3) Increasing Asolf alone: Interestingly, increasing only Amslf leads to drop in interventional dependence and reduces the error disparity between observational and interventional data (Fig, [8c), even when Adep is nearly zero. However, this decrease in performance disparity comes at the cost of higher observational error (left to right in Fig. 18a).
(4) Lowest interventional error: Fig. 18b we can observe valley of relatively lower interventional error. The hyperparameter combination corresponding to the lowest interventional error occurs within this valley, marked with a yellow diamond. The same position is marked on other plots for ease of viewing. The lowest interventional error obtained exper- imentally was 0.4, considerably higher than the theoretical interventional error 0.25 that a robust model would have attained. This indicates that the best hyperparameter combination did not result fully robust model. However, this is not surprising since our theoretical results in Sec. 3.4 [.44 suggested that linear model cannot learn fully robust model if the training dataset contains any observational data. Additionally, note that this hyperparameter combination did not result in the lowest perfor- mance disparity between the distributions and, instead, it appeared near a phase change in the loss values. To observe this phase change more clearly, we plot the loss values along the white dashed line Fig. [8b] where we vary Agelf and fix Adep to the value takes in the best hyperparameter combination (yellow diamond).
Labeled valne
Obs Error
1.0-
Iot Error
Abs Inc
0.5
Dop
0.0
10
10-1 A
10'
Figure 19: Change in observa- tional and interventional error values for a fixed Xdep corre- sponding to the yellow diamond in Fig. [8b and varying Aselt-
40

---

Published in Transactions on Machine Learning Research (12/2025)
In Fig. 19 we observe that Aeel increases, interventional error drops rapidly, achieving its minimum Adep corresponding to the yellow diamond (denoted by the dashed black line in Fig. 19 and then increases steadily to eventually saturate. Similarly, observational error gradually increases with increasing Asel initially and then displays a more rapid increase, eventually matching the interventional error at higher values of Azelt- Throughout these changes, the statistical dependence between the representations of interventional data remains nearly zero.
Our results indicate that, while both Ldep and Letf needed to learn discriminative representations that are robust to interventional distribution shifts without losing their utility in downstream applications, hyperparamcter tuning is still necessary to balance the effects of these loss functions.
H.1 Why do the hyperparameters change between experiments?
In our main experiments, we chose different hyperparameters for different experiments. In this section, we explore the factors that affect the choice of hyperparameters between experiments. In particular, we focus Adep: dep is the primary loss function responsible for enforcing statistical independence between the interventional reprosentations. We start by noting that robust representations are obtained by (at least partially) inverting the data-generating function from the latent, variable to the observed signal. Therefore, we hypothesize that as the complexity of this data-gener function increases, Adep and Aself generally increase. Here, we use the term "complexity" to roughly mcan the minimum degree of a polynomial required to model the data-generating function. Informally, the more complex the data-generating function, the more hesitant the model to learn robust representations (Geirhos et al., 2020
(emal))
Figure 20: (Top) Sinusoidal transformation a 1D Gaussian random variable with added noise n. N(0,, 0.01), and (bottom) variation of interventional error for various values of Adop and Asotf-
We now formally verify our hypothesis by adding a non-lincar function in Eq. 21 the simple dataset that we used to investigate the effect of Adep and Aself. We modify Eq. (21 as follows:
x=[sin(6 = (
A)
B B.J.
(22)
where s controls the amount of non-linearity. A very low value for will result in nearly linear function, as sin function is approximately linear near the origin. As 8 increases, the non-linearity also increases. For higher values of s, multiple values of Á will be mapped to the same value. For the remainder of this section, we will refer to s as the "non-lincarity factor." See Fig. 20 (top) on how the value of s affects the simusoidal transformation of a Gaussian random variable with added noise.
In addition modifying the data generation process by using non-linear relation from the latent variable to the observed signal, we also use non-linear models to learn the representations for each variable. Specifically, use MLPs with 2 hidden layers and the ReLU activation function. We are interested in the variation in Adep that gives us the minimum value of interventional crror as the non-lincarity factor changes. If our hypothosis is correct, then Adep must increase as the non-linearily factor s in Eq. (22) increases. Following the previous setup, we will sample several values for Adep and Aself, and train models for each combination of Adep and Asell- To save compute, we restrict the range of > to [10-0.5 10 while sampling Adep and A as the minimum interventional error usually lies in that range. Additionally, we utilize Bayesian optimization, employing the probability of improvement, to guide hyperparameter selection, thereby sampling more hyperparameters
41

---

Published in Transactions on Machine Learning Research (12/2025)
from promising regions where the interventional error is typically low. The interpolated heatmap showing
the interventional errors for various chosen hyperparameters is shown in Fig. 20 (bottom).
0.6
Conppeeding 0.5
2.0
Concyendeg
Maxiuum
Nco-linzarty
Non-linarty
Figure 21: Variation in the minimum interventional error, and the corresponding Adep and Xself when the non-linearity factor 8 is increased. The results are for the bottom-5% percentile lowest interventional error values obtained over 4500 runs for each value s. The median values are shown using dark lines, and the region between the first and third quartiles shaded.
Fig. 21 shows three plots: bottom-5% percentile of interventional error, and corresponding Adep and Aselt values. For each of these values, we plot the median value using dark blue curves, and the region between the first and third quartiles is shaded in light blue. As expected, the minimum interventional error increases with the non-linearity factor A similar trend can be observed for Adep and Acef- Particularly, for Adep: the shaded region expands as increases, indicating that higher values of Adep can now obtain very low values of interventional crror.
Generating Windmill Dataset
We provide the exact mathematical formulation of WINDMILL dataset described in Sec. 3.1 following constants:
We define the
Constants
Description
Default value
Parms
Number of "arms" in WINDMILL dataset
4
Tuax
Radius of the circular region spanned by the obscrved data
2
Owid
Angular width of each arm Offset wavelength. Determines the complexity of the dataset.
0.9r
0.7068
Aoff
6
0max-off
Maximum offset for the angle
T/6
Table 11: Constants used for generating WINDMILL dataset, their meaning, and their values.
RB 2.1,2.5)
(Sample radius)
R
Tmax
(BRB (1 B)(2 RB))
(Modify sampled radius based on B)
2
A ~ c (f22
i
1t)
Narms + 0, ..... Paruss
(Choose an arm)
R Tholl Fmax
O. mmaxcooft sin
(Calculate radial offset for the angle)
U 1(0
(To choose a random angle)
e Owid (U -0.5) + O
+ (1 - A)O Pot
Narms
(Angle decided by A and the radial offset)
X1
X, = Reos θ X, = Rsin O, x
(Convert to Cartesian coordinates)
X₂
42

---

Published in Transactions on Machine Learning Research (12/2025)
Py'Torch code to generate WINDMILL dataset is provided in Listing
Listing 1: Code for WINDMILL dataset
ingort
import torch
Constants
nun arna # nunber of bladea in the windmill
max nax_th_ oh 0.5236 offset
added angle shearing (- pi/6)
r_aax 2 length of the blade
mun_p
generated adjusta the complexity of the blade
points
offaet_ sarelathav
Sample latent variables according the causal graph.
torch. bernoulli one if obeervational B
0.6)
torch. bernoulli (torch, ones (nin_pointa) 0.5)
th_A0 torch. Linepace 2ematà.pi. pi, num_arm=*1) [==1=)) [:-1]
th,A1 linspace(O, 2emath.pi, aun_arme+1) [:=1] math. math.pi/buuz.rr
Choose randon arm for λнD from possible arns. Likevise for Am1. th AD A0ftarch.randimm th_A1 [h._Al[torch.aadie (audint(uuu_erre (nun_p,))]
arma, (mun_p,))]
distribution alpha=1, beta=3
beta_dist torch. distributions wew.bbutioon.bbt..eeeee.... Beta(,, 2.5)
Sample according to B. If B=0, sample snall alse sampla large ranges from r_eax
BC_r dis__dde..ssmpleec.. Size [own_p))) r_uex/2.
Bi_r r_nax beta_dist. sanple Size Size((num_p)) ([um__))) r_naz/2. Bi_r
Sample according
Chooge the thata arn according and than sample from this are using a uniforn distribation.
Firet will have cartsheel style.
theta torch, (mnd_))))))bb_pee tb_A0=(1-A) th th_xid/2.
Add an offset to theta according to th, ofdottfeet_dd torch.sin((r/r. saffsttt.arcllggghemettt
th_offaat nax th, th_offaeteth offaat__eo nod thets offeffett
xl r*torch.cos con (ttbteaa))(
torch.stack([xi x2], dinsi)
labels sorch.atack([A, B], din=1) (il=:t).tppeettttteee (torchh.
43


Reviews:
[{"review_id": "twio47Luoi", "reviewer": "Reviewer_tJsN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of learning robust representations that perform well under interventional distribution shifts. The authors identify a critical oversight in existing methods: they often fail to leverage independence arising from interventions, treating interventional data merely as samples from a different domain. The core contribution of the paper is to establish a strong correlation between this performance degradation on interventional data and the statistical dependence of learned features that ought to be independent. To address this, the authors propose a new training algorithm RepLIn, which explicitly regularizes the model to enforce these known statistical independencies in the feature space during training. The paper provides theoretical justification for this approach in a linear setting, deriving sufficient conditions under which enforcing independence provably reduces test-time error. The effectiveness of RepLIn is demonstrated empirically on a synthetic dataset as well as on image and text datasets, showing improved robustness compared to ERM and several domain generalization baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provides both a theoretical foundation (although limited in scope) and extensive experimental results to demonstrate its central claim: incorporating interventional independence improves robustness against interventional distribution shift.", "audience_yn": "Yes", "audience_explanation": "This paper reveals that incorporating interventional independence improves robustness against interventional distribution shift, which may interest some researches in the community of trustworthy machine learning.", "requested_changes": "There are several areas where the analysis could be expanded and clarified to further strengthen the contribution.\n\n1. The theoretical analysis in Section 3.4, while providing valuable intuition, rests on a simplified causal model that may limit its generality. \n- The causal mechanism is modeled as a deterministic assignment ($B = \\omega_{AB}A$). In most literature, such relationships include an exogenous noise term (e.g., $B = \\omega_{AB}A + \\epsilon_B$) to account for unobserved factors. It would be beneficial to discuss or show whether the paper's conclusions still hold in this more common and realistic stochastic setting.\n- The analysis considers \"hard interventions\" where a variable is set to a fixed value. I am curious if the analysis can be extended to the more general case of stochastic interventions, where the target variable is made to follow a new distribution independent of its causal parents. Broadening the theoretical scope to these settings would significantly strengthen the paper's claims.\n\n2. The theoretical results, while insightful, could be developed further.\n- The analysis is confined to the simplest case of a two-variable linear system. While I understand the difficulty of non-linear analysis, extending the theory to the multi-variable linear case would be a valuable addition. This would better align the theory with the scalability experiments presented in Section 6.2 and provide a more complete theoretical foundation for the method.\n- Proposition 1 claims that if $\\beta$ is larger than a specific threshold, the model will be more robust. But the threshold depends on the unobservable causal strength parameter $\\omega_{AB}$. Moreover, It remains unclear what happens when $\\beta$ is smaller than the threshold.\n\n3. The paper proposes minimizing MHSIC to enforce independence. While the authors correctly note its relation to MMD, the paper would benefit from a more explicit justification for this choice over other common independence measures. For instance, a discussion on the pros and cons of NHSIC versus minimizing Mutual Information (MI)—a very common technique in representation learning—in terms of computational complexity, estimator variance, and suitability for gradient-based optimization would be highly valuable.\n\n\n4. There are a few minor points that could improve the clarity and accuracy of the presentation.\n- First, the paper refers to the experiments on CelebA and CivilComments as being on \"real image and text datasets\". This is slightly imprecise, as the causal graphs and the nature of the interventions are synthetically imposed onto the data. I would suggest a more accurate phrasing, such as \"experiments on real-world datasets with semi-synthetic causal structures.\"\n- Second, there are several places where citations are unnecessarily enclosed in parentheses.", "broader_impact_concerns": "I have no concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5TcISVzbUr", "reviewer": "Reviewer_6NZU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes RepLIn, a method for improving robustness to interventional distribution shifts by enforcing statistical independence between representations of intervened nodes and their non-descendants. The motivation is from the observation that hard interventions induce independence in the causal graph. The method adds NHSIC-based penalties and self-dependence regularization, with theoretical support (in linear settings) and empirical results on synthetic and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the paper are supported with:\n- It provides a clear theoretical explanation of why ERM fails under intervention and under what conditions enforcing independence improves robustness.\n- Experimental results on both synthetic and real datasets support the effectiveness of the proposed approach.\n\n\nHowever, several aspects weaken the overall clarity and novelty of the contribution:\n- While standard domain generalization (DG) baselines (IRM, Fish, GroupDRO) are included, the conceptual connection between interventional robustness and domain generalization is not fully developed. The paper does not clearly explain how RepLIn relates in assumptions, objectives, or methodology from existing DG/OOD approaches.\n- The discussion of identifiable representation learning is limited to the appendix, without elaborating on how RepLIn achieves robustness without full identifiability, or how it compares to related works such as CITRIS[1] and DMSVAE[2], which also utilize interventions in similar settings.\n- The paper would benefit from discussing design choices—e.g., why the authors chose NHSIC-based independence constraints instead of full identifiability losses (like ELBO + MMD), and why this might be a lighter-weight, more scalable alternative.\n\nOverall, while the core idea is valid and supported by evidence, the contribution feels incomplete without stronger comparative analysis and positioning.\n\n[1] CITRIS: Causal Identifiability from Temporal Intervened Sequences, 2022\n\n[2] Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependencies, 2024", "audience_yn": "Yes", "audience_explanation": "This paper is on some interesting and relevant areas — causality, representation learning, and robustness — so I think it would definitely catch the attention of people working in those areas. The idea of using interventions to encourage independence in learned representations is a nice one, and the method is lightweight and practical.\n\nHowever, the links to prior work in domain generalization, OOD robustness, and causal disentanglement aren't clearly discussed. As a result, it's a bit hard to tell how novel the method really is or where it fits in. It would be better to include a clearer discussion and some stronger comparisons.", "requested_changes": "Please check the review above.", "broader_impact_concerns": "No negative social impacts", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tuQhgGBN2f", "reviewer": "Reviewer_MhWe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper deals with the problem of learning representations that are robust to distribution shift arising from causal interventions. Throughout the paper, it assumes that the underlying causal structure is known, i.e. it is known which variables are intervened upon and on which other variables they depend. The paper contributes new insights into the interplay between adherence to the underlying causal structure and the predictive performance on both observational and interventional data. That is, representations that violate the given causal structure by, for instance, not enforcing independence between interventional and observational variables correlate with representations that exhibit a large gap in performance on observational and interventional data. The authors use this initial empirical insight to then provide certain sufficient conditions for a linear causal model to provably adhere to the causal structure and later to propose a novel method called RepLIn that aims to improve causal adherence through the addition of two connected loss functions. RepLIn is evaluates on a synthetic and two more realistic datasets with an induced causal structure and compared to a range of state-of-the-art baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides ample empirical evidence that supports its claims of lowering the gap in predictive performance between observational and interventional data.", "audience_yn": "Yes", "audience_explanation": "This paper proposes a method to improve adherence of learned representations to a given causal model. Given the ubiquity of machine learning and representation learning, enforcing causality, when known, is an interesting research direction.", "requested_changes": "While I am generally positive about this submission, I do have a couple of questions and remarks.\n1. Table 1: Why are these results not also provided with standard deviation/error like all other results later on? The results also seem to be different from those reported in Table 2 (ERM-resampled reaches a observational accuracy of around 96 while it reaches around 93 in Table 1). The difference, at least from the provided variability metrics, seems larger than what run-to-run variance would suggest possible. What could be the cause of this?\n2. One bigger apparent weakness would seem to me that the representations you train do not have any *inherent* ability to distinguish observational cases from interventional ones. While the distributions for $X$ and $X'$ are different, the same model is used for predicting both variables, leaving the model to effectively try to model a mixture of these two distributions since the inputs for $X$ and $X'$ are, of course, still the same. Intuitively, I would expect that the model could be explicitly conditioned on an intervention taking place to facilitate full distribution shift invariance. I do understand that the information that the model is predicting interventional data is not always available, but I feel a discussion on this topic would make sense in the paper. If nothing else, for future work? This topic is important, because it also appears in your derivation of the linear model, where the non-degenerate solution S2 is effectively enforcing fully independent representations for $A$ and $B$ because the linear model is unaware it is being intervened upon, while it is expected to give reasonable performance still.\n3. Top of Section 3.3: \"We also randomly set the number of training epochs to use early-stopping\", I do not fully understand what you mean here. Do you set the patience of early stopping with a random number of epochs?\n4. Section 3.4: \"Assuming zero mean for all latent variables\", is this realistic? Do you have any empirical evidence that this at least *tends* to hold for linear models?\n5. Proposition 1 does seem on the weaker side because of the conditions on $\\beta$. The proof of the statement seems sound and I understand why these conditions arise, but from you empirical validation it does appear that the more practical setting of a small $\\beta$ mostly does not satisfy those conditions. Hence, I feel this result is not saying much of practical importance, especially since the discussion does seem to favour solution S2 because of its more intuitive interpretation of \"no signal from $A$ is used for $B$\". A more extended discussion on the importance of this result could help place it better in the overall story of the paper.\n6. Can you give an intuitive reason why the optimal regularisation parameters do not generalise between the two datasets? I can understand why in the non-resampled case, but for resampled data I would expect them to be similar.\n7. Why would the dependence go down with increasing $\\beta$ for resampled data (Figures 8 and 9)? Given you resample, I would expect both interventional and observational components to have a similar impact in the loss and so have a more uniform final dependence.\n8. I do not think the qualitative analysis on CelebA contributes much of substance to the paper. Not only do methods like GradCam have their own problems, it is hard to grasp how representative the 5 handpicked examples are. In particular, what happens for images where ERM-resampled was correct, but RepLIn-resampled was not?", "broader_impact_concerns": "No concerns.", "additional_comments": "#### Strengths\n1. The paper is generally very well written. It is easy to follow from the very start as the authors make it very clear what the aim of their research is by using a clear example. The remainder of the paper is equally easy to read and follow. Some more technical parts do need to be read twice to fully understand, but I fully attribute that to being more technical. \n2. It seems the general idea of the paper to assume that the causal structure is known and that you want to enforce that knowledge as much as possible is quite novel. Most other approaches do not assume so and need to jointly learn the causal structure with their representations. While the latter is more flexible, I believe it might also lead to learning causal structures that help the representations to solve their task instead of being truly correct.\n3. The experimental section provides sufficient evidence that RepLIn does both increase predictive performance as well as lower the predictive gap between observational and interventional data. Apart from that, I also appreciate how the synthetic experiment is used to clarify the initial intuitions.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)