Invariant Structure Learning for Better Generalization and Causal Explainability
Authors: Yunhao Ge
Sercan O Arik
Jinsung Yoon
Ao Xu
Laurent Itti
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: Learning the causal structure behind data is invaluable for improving generalization and ob- taining high-quality explanations. Towards this end, we propose a novel framework, Invariant Structure Learning (ISL), that is designed to improve causal structure discovery by utilizing generalization as an indication in the process. ISL splits the data into different environments, and learns a structure that is invariant to the target across different environments by imposing a consistency constraint. The proposed aggregation mechanism then selects the classifier based on a graph structure that reflects the causal mechanisms in the data more accurately compared to the structures learnt from individual environments. Furthermore, we extend ISL to a self-supervised learning setting, where accurate causal structure discovery does not rely on any labels. Self-supervised ISL utilizes proposals for invariant causality, by iteratively setting different nodes as targets. On synthetic and real-world datasets, we demonstrate that ISL accurately discovers the causal structure, outperforms alternative methods, and yields superior generalization for datasets with significant distribution shifts.

Published in Transactions on Machine Learning Research (10/2023)
Invariant Structure Learning for Better Generalization and Causal Explainability
Sercan Ö. Arik*, Jinsung Yoon', Ao Xu*, Laurent Itti*, and Tomas Pfister fyunhange, comu, itti] @rusc.edw. {soarik, finsungyoon, tpfister) @google.coom
Yunhao Ge
Google Cloud AI, Sunnyeale, CA, USA
University of Southern Colifornia, Los Angeles, CA, USA
Reviewed on OpenReview: https: // optp//veepereeew...mmee/./ net/ ftrmm? id= A9t. 7KTusk
Abstract
Learning the causal structure behind data is invaluable for improving generalization and ob- taining high-quality explanations. Towards this end, we propose a novel framework, Invariant Structure Learning (TSL), that is designed to improve causal structure discovery by utilizing generalization as an indication in the process. ISL splits the data into different enviromments, and learns a structure that is invariant to the target across different environments by imposing a consistency constraint. The proposed aggregation mechanism then selects the classifier based on a graph structure that reflects the causal mechanisms in the data more accurately compared to the structures learnt from individual enviromments. Furthermore, we extend ISL to a self-supervised learning setting, where accurate causal structure discovery does not rely any labels. Self-sueervieed ISL utilizes proposals for invariant causality, by iteratively setting different nodes as targets. On synthetic and real-world datasets, we demonstrate that IST. accurately discovers the causal structure, outperforms alternative methods, and yields superior generalization for datasets with significant distribution shifts. We open-source our code at https:://gith// com/ AaaroooorrIKIL........ git
1 Introduction
High capacity machine learning models such as deep neural networks (DNNs) have fueled transformational progress in numerous domains where the i.i.d. assumption is mostly valid (Zhang et al. 2021b) as they can be very effective in fitting to available training data. However, as a severe blind spot in conventional machine learning, the performance such models can be much worse on the unt-of-distribution (OOD) test data. This 'overfitting' phenomena can be attributed to over- peerpparaneeeriee models such as DNNs absorbing spurious correlations as shown in Fig. from the training data and resulting in biases unrclated to the causal relationships that. truly drive the input-output mapping for both training and test samples (Zhang al.,
2021a Schott et al. 2021 Roelofs et al. 2019 D'Amour et al. 2020 Bartlett et al. 2021).
In most cases, the machine learning problems are underspecified, i.c. there are multiple distinct solutions that solve the problem by achieving equivalent held-out performance on i.i.d. data. Underspecification in practice can be an obstacle to reliable real-world deployment of high capacity machine learning models, as such models can exhibit unexpected behavior when the test data deviate from the training data (D A'mour
et al. 2020 Arjovsky et al. 2019).
Various methods have been proposed towards reducing mitigating underspecification and overfitting: reg- ularization approaches Arpit et al., 2017 Ng. 2004 Tibshirani 1996 Hoerl & Kennard
1970 constrain
Work done while Google
The implementation is available in nttps://github. http:///tttttccc//aaaaa/////////

---

Published in Transactions on Machine Learning Research (10/2023)
Environment
Ground-truth
X1
X2
X3
X1 Shape
X2 Texture
Back- ground
~ PlAs 11 X1 ~ P(X- OY 0.9
Environment 2
Environment
NOTEAR-MLP (Baseline)
ISL (Ours)
Label
X1
X2
X3
X1
X2
X3
X, N(O, + E1
X2 ~ NO, oz) +
Y
Y ~ PY 1[x
PEX, SgmPPK (V 00 0.101
s(X, 2X²)
Xy P(N op 0)
MSE: 0.12
MSE: 0.02
(a) Structure causal model
(b) Different environments
(c) Discovered DAGs with MSE.
Figure 1: A motivational example. (a) For the image label Y (1 means the label is "cow" and otherwise), X1 and X2 represent the causal parents about the image details (here, shape and texture), and X3 (background type where 1 indicates the presence of grass and 0 otherwise) represents factor that isn't causal to Y. S(.) is sigmoid function. In this example, texture (X2) is twice as causal to Y than shape (X1). (b) The relationship between Y and X3 vary across environments; since the conditional dependence not consistent across envirouments X3 may not be treated as a major causal factor for Y. (c) We utilize the Mean Squared Error (MSE) as metric to assess the prediction error for 'Y: This carried out by using the projected causal parent of 'Y' as features inputted into two-layer neural network. Smaller MSE value implics that the causal parents variables used for prediction are more precise. Our proposed method ISL yields more accurate discovery of the underlying causal relation - here, it correctly identifies X1 and X2 but not X3 the causal factors of Y, improving the explanation quality and prediction accuracy.
the flexibility of models; data augmentation methods (Yaeger al. 1996 Krizhevsky et al.) 2012 generate artifciilly-traasfforeedd samples invariant to labels; judicious DNN designs (Arik & Pfister 2019 Oord et al.
2016 Jaderberg et al. 2015 Lim et al. 2019 introduce appropriate inductive biases for the data types of interest. Notably, the CASTLE (Kyono el al. 2020) approach introduces novel regularization method that incorporates causal relationships into the regularization process.
These approaches have shown some progress in improving generalization, and in some cases they yield siguificant improvements in test accuracy, however, underlying systematic framework beneath them is missing they do not tackle the fundamental challenge of discovering causal relationships that are consistent across the training and test data and basing the decision making on them. Thus, their improvements remain restricted to specific scenarios consistently showing significant OOD generalization improvements require discovery of casual relationships. Accurate discovery of causal relationships would not only improve accuracy and reliability, but also enable explainable decision making, which is crucial for high-stakes applications such as healthcare or finance Shin 2021
Learning the true causal relationships very challenging, fundamentally (Schölkopf 2022 It is infeasible to consider all combinations for factors of variation (such as shape, size and color of an image), as it would be exponential in size M combinations with M categorical features where each can take N different values). Effective methods should reduce the prohibitively-hige search cost and data inefficiency while accurately discovering the underlying mechanisms. Causal discovery has been studied using various approaches. There are methods based on interventional experiments by randomized controlled trials (Pearl 2009), but they are often prohibitively costly. A more realistic setting is learning from observational data. Constraint- based algorithns(Spirtee et al.) 2000 2013) directly conduct independence tosts to detoct causal structure. Score- based algorithms (Chickering) 2002; Huang et al. 2018)) adopt score functions consistent with the conditional independence statistics, however, these only find the Markov-equivalence class (Guo et al.
2

---

Published in Transactions on Machine Learning Research (10/2023)
2020 Functional causal models (Shimizu et al. 2006; Peters et al. 2014 aim to identify the causal structure from the equivalence class, but the heuristic directed acyelic graph (DAG) search methods suffer from high computational cost and local optimality, as the number of nodes increases. To address this problem. NOTEARS (Zheng et al., 2018) proposes a differentiable optimization framework. NOTEARS-MLP (Zheng et al. 2020 and Sran-DAG (Lachapelle et al., 2019) extend NOTEARS to non-linear modeling with DNNs.
NoFear (Wei et al. 2020) reevaluates NOTEARS continuous optimization framework, and subsequently,
innovates a local search algorithm that enhances its performance. GOLEM (Ng et al. 2020) propose a likelihodd-based structure learning method that applies soft sparcity and DAG constraints. DARING (He
et al. 2021) uses constraints on independent residuals to facilitate DAG learning. One common shortcoming of these approaches is relying on empirical risk minimization. Recent work has shown on the other hand that invariant risk minimization (Arjovsky et al. 2019) can be very powerful in preventing the absorption of spurious correlations during DAG learning.
Our goal in this paper to push the stato-of-tho-art in accurate discovery of structural causal models (SCM), and as a consequence, improve the accuracy and reliability of models especially in the presence of severe distribution shifts. Motivated by the limitations of existing work mentioned above, we propose Invariant Structure Learning (ISL), a framework that yields causal explainability, based on tying generalization and SCM learning. Intuitively, better generalization should lead to more accurate SCM learning, and an accurate causal structure should yield improved robustness and generalization. ISL encourages reinforcement between these two goals. Specifically, ISL uses generalization accuracy as a constraint to learn the invariant SCM (as DAG) that represents the causal relationship among variables. Take Fig. | as an example, where we simplify the object recognition task by using variables to represent the key factors: X1: object shape, X2: object texture (including color), X3: image background (as context), with the output label Y. Fig. 1 (a) shows the ground truth (GT) Structural Causal Model (SCM). During training, the data (Fig. [ (b)) consist of samples from different environments. Baseline methods such as SSTEARS-MLE and CASTLE directly estimate the underlying causal structure, which leads to spurious correlations being absorbed, which in turn results in sub-optimal test accuracy. Our method ISL, on the other hand, learns the invariant structure that correctly identifies the SCM and yields better test accuracy. Overall, our contributions are highlighted as:
We propose Invariant Structure Learning (ISL), a novel learning framework that yields accurate causal explanations by mining the invariant causal structure underlying the training data, and generalizes well to unknown out-of-distribution test data.
We generalize ISL to self-supervised causal structure learning, which first treats the discovered invariant correlations as potential causal edges, and then uses a DAG constraint to finalize the causal structure.
We demonstrate the effectiveness of ISL on various synthetic and real-world datasets. ISL yields state-of- the-art SCM discovery (clearly outperforming alternatives on real-world data) with a particularly prominent improvemnnt for complex graphs structures. In addition, ISL improves the test prediction accuracy throughout, with especially large improvements in cases with significant data drifts (up to 80% MSE reduction compared to alternatives).
2 Related Works
Improving machine learning generalization. Many different approaches have been studied to improve generalization (i.e. bringing the test performance closer to training). Regularization methods (Arpit et al.,
2017 Ng 2004 Tibshirani 1996: Hoerl & Kennard 1970: Hinton et al. 2012 Wager et al. 2013). early
stopping Goodfellow et al. 2016), gradient clipping Pascanu et al. 2013), batch normalization (loffe &
Szegedy 2015), data augmentation Yaeger et al.. 1996 Krizhevsky et al. 2012) are among the most popular
ones. These aren't based on discovering the true relationship between the features. Towards generalization improvements with input feature discovery direction, supervised auto-cncoders (Le et al. 2018) add a reconstruction loss for the input features as a regularizer. Recently, some works (Janzing 2019 Ba
et al. 2017) combine causal discovery with model regularization for better generalization. CASTLE (Kyono ct al. 2020) implicitly uses underlying Structural Equation Model (SEM) reconstruction as the regularization to improve model generalization. However, it can't explicitly yield a DAG for the causal structure and it can't completely prevent learning spurious correlations. ISL. addresses these two challenges by learning the
3

---

Published in Transactions on Machine Learning Research (10/2023)
invariant structure across environments and outputting a DAG to describe the causal data structure, and eventually showing better generalization.
Causal structure discovery. Constraint- bastd causal discovery algorithms (Spirtes et al.
2000 2013). and
some woore-based (Chikers( 2002; Huang et al. 2018: Shimizu et al. 2006 Peters et al. 2014)
conduct exhaustive and heuristic search for the DAG structure, yielding combinatorial explosion issue when they scale up to a larger number of the nodes. NOTEARS (Zheng et al.] 2018) proposes directly applying a standard numerical solver for constrained optimization to achieve global approximate solution overcoming
the scalability bottleneck. They formulate the structure learning problem maximum likelihood estimation over observational data with the additional constraint that the weight matrix has to represent DAG with
the acyclicity and sparsity properties. NOTEARS-MLP (Zheng et al. 2020) and Gran-DAG Lachapelle et al. 2019) extend NOTEARS to non-linear functions by using DNNs. RL-BIC (Zhu et al. 2019) uses Reinforcement Learning to search for the DAG with the best scoring. GOLEM (Ng et al., 2020) applies a
likelihood-basel objective with soft sparsity and DAG constraints. These methods don't consider using the generalization quantification as an indication or constrain during DAG learning, which makes the learned DAG sometimes absorb biases and spurious correlations from data.
Invariant Learning. The field of invariant learning is increasingly gaining traction, largely due to its implications for causality. EIIL, (Creager et al. 2021 focuses on the inference of environments to facilitate invariant learning, while "ZIN' (Lin et al.] 2022) delves into the conditional feasibility and methods for environment inference. Another noteworthy contribution GALA (Chen et al. 2023), which advocates for invariant learning without the need for explicit environment partitioning. Despite these advances, our work stands apart in its capability for causal discovery. Unlike the aforementioned studies, our methodology uniquely allows for simultaneous learning of causal structures and predictions.
3 Methodology
In Section 3.1, we first present the problem definition and the motivation behind our work by discussing how spurious correlation can affect the model generalization and how its influence can be alleviated. Section 3.2 describes the proposed ISL framework for supervised learning setting. In Section 3.3, we extend our discussion to the generalization of our approach in a self-supervised setting.
3.1 Motivations
Problem definition. Standard supervised learning is defined for a dataset with given input variables x= (Y, X1, X2, ... .)), including X= (X,) E X and Y E y, the goal is to learn a predictive model x Px, denotes the joint distribution of the features and target, Dtrain denotes the training data with N samples, and Dreet denotes the testing data. Ideally, we expect both Ptrain and Drest to be i.id.., sampled from the same distribution Px, However, it is hard to satisfy this condition for real-world data. It becomes more severe when the model overfits to the training set or the training set does not reveal the underlying distribution Px,Y
Spurious correlations and causality. One perspective to explain poor generalization due to overfitting is models learning spurious correlations. Broadly, a correlation can be considered as spurious when the relationship does not hold across all samples the same manner (Arjovsky et al., 2019). For example, for the image recognition task in Fig. [[a,, the model may use the green color of the grass to recognize cows, instead of complete profile of its shape. The correlation between green-colored grass and the cow label would be spurious and not consistent. In contrast, with causal learning, our goal is to learn stable and invariant relationships, that generalize well. Let's consider that there is a SCM defining how the random variables 8 =Y, X1, X2,.. Xa) define each other. The larget variable Y is generated by a function (ra(a(Y), uy), where PY(Y) denotes the cansal parents of Y in SCM. parametri SEM (Pearl et al. 2000) proves that if we use the causal parents of Y as inputs to predict Y, the learned model would be optimal and generalize well on the unknown test set. Identifying causal parents of Y from spurious correlations is the key to obtain better generalization and causal explainability.
4

---

Published in Transactions on Machine Learning Research (10/2023)
Invariant structure across environments: An environment is used to distinguish different properties of data (such the generative source characteristies). and can help reveal reasons for spurious correlations. Examples of environments can be different devices for capturing the images, or the hospitals at which the patient data are collected. Broadly, it can be considered as a set conditions, interpreted as 'context' of the data (Moneda 2021). As an important indication of distinguishing causality from spurious correlations, the causal structure of should be invariant across all possible environments. Our goal is learn such invariant. structure across environments, which should yield better generalization.
3.2 Learning framework
Share
ISL modile
Predictor
Training data
Predictions
Env-2
Environment building (e.g.. K-means)
ISL module
Bi-lavel optimiza
Env-K
Aggre- gation
Discovered
DAG
ISL module
Liperse
LoaG
ex
O
X
0
X
The proposed Invariant Structure Learning (ISL) framework. Given raw data, we build different environments using unsupervised clustering, unless the data source information is provided. For different environments, each ISL. module outputs a summarized DAG to represent the learned invariant structure. An aggregation mechanism then selects the optimal predictor based on graph structure that reflects the causal mochanisms in the data more accurately. During training, the constraint on the Y prediction across environments helps learning an invariant structure. Consequently, the learned DAG leads to a superior predictor. Bottom: Details of the ISL, module. 0f is the invariant structure of Pa(Y) shared across all modules.
Figure Top:
Fig. 2 shows the overall Invariant Structure Learning (ISL) framework. Given raw training data, we first build different environments. If we had the information data generation and collection process, it would be simple to build different environments based on them. Without such information, environments in an unsupervised way using unsupervised clustering, K-means (Lloyd 1982), which clusters the raw data into different clusters that each representing an environment. The clustering can be done for the raw data or learned representations, which would require an encoder to map raw data. To determine the number of clusters, we use Flbow (Thorndike) 1953) and Silhouette (Rousseeuw) 1987) methods. To balance the data size across environments, we employ upsampling. We augment the data size to reach to n, the largest number of data samples in an environment. Fig. [ ((top) depicts the environment building process, where each color represents different source or generation method for the sample. In general, at least two diverse environments are sufficient to learn the invariant structure (Arjovsky et al. 2019). We show that ISL is robust to the number of environments and typically an intermediate value is optimal.
propose to build
After assigning data samples to different environments, our goal is to learn the invariant structure that results superior generalization for prodicting Y. In each environment, using that environment's data, an ISL module (see Fig. [boottom) independently learns a DAG which defines the variable relationship for that specific environment. To learn the invariant structure for predicting Y, add constraint among IST.
5

---

Published in Transactions on Machine Learning Research (10/2023)
modules that the parameters to reconstruct Y should be identical across environments. The desiderata for invariance is expressed as loss function over all environments in the training data. Tdeally, the generalization goal would be minimization of an OOD risk ROOD (f) = maxegtau Rt (f(()) over all possible environments, not only the oncs in the training data. We aim to approximate this with a tractable objective. We propose where I denotes the loss function.
the risk under a certain environment E as (f) = Ex xx= yr (I(](Xx, We decompose the objective function f() into two components. The first is to find the representation of causal parents of Y from given X, which can be considered as the invariant structure for Y across environments. The second is to optimize the classifier with the learned Pa(Y) as the imput. I is multi-layer perceptron used to approximate f(. It consists of: (i) g(-) with parameters oY to learn a representation of Pa(Y), and (ii) h(-) that inputs the representation and yields the prediction for Y. To learn the representation of Pa(Y), x()) should follow the causal structure of Y. Overall, the proposed objective to learn the invariant causal structure (as a DAG) is summarized as below:
min
Σ
(h 0f (X)),
eEcall of .h
(1)
s.t. 00 0x arg min 8
RD,G((
This objective is in bi-level optimization form. The outer loop is for the final goal of obtaining the predictive model for Y to generalize well on all environments, requiring that or extract the representation of Pa(Y). The inner loop adds the constraint that of should be the invariant across all the environment during learning their DAGs, As shown in Fig. ? we use a DNN to learn the SCM (represented as DAG) of the dataset. The parameter of 0 consists of three parts: first layer to reconstruct variable Y, oY, rest layers to reconstruct variable Y, 0%, , and layers to reconstruct other variables X, 6x We use the following objective function represented as the DAG loss RDAA(X,6)
R R Croel (x, 0) 2
+ sCspars (0),
(2)
where (0) = 2N 1 - 8(8)11 and h(W) = Tr(eWow) d, with || || being the Frobenius norm, LDAG = () +a)(W) denotes the constrain of DAG. N denotes the number of samples, Tr() is the trace operator, W is a (d+1) x (d+1) adjacency matrix (W E R(d+1)x(d+1)) which represent the connection strength between variables and the final DAG is summarized from W. (Zheng et al. 2018) proves that W is a DAG if and only if h(W)= = O. W is summarized from the first layer O2 of 0. Specifically, [W],, is the L2 norm of the k-th row of the parameter matrix 0.. 0 is the DNN parameters used to reconstruct variable j. that we decompose as the first layer of and the remaining layers of (Fig. 2 Laparse (8) = 81 111 + 82| 1/ where ||- Ila and || ll2 denote I and la regularization, respectively, and 3, are hyperparameters that can be optimized on validation set. Fq. 2 is a solution using Augmented Lagrangian (Fortin & Glowinski 2000) approach, where a > 0 and p > O are gradually increased to find solutions that minimize h(W). To learn the invariant structure across different environments, in all modules, we use a shared layer BY. It learns a representation of Pa(Y) given input feature X, (0) (X) Ri Pa(X)), which is a constraint to learn the invariant. structure of Y prediction among environments. We simplify the training of Eq. 1 and the overall training objective of the proposed ISL is defined as:
||a + -//110* ||2,
min Σn
RR Eeerum ( (x)) 07 07 000 0x)).
(3)
0
where y the tradc-off parameter and Ro (8, 18,0 ,xx) the invariant structure constraint. We propose solving this problem with second order Newlon method, L-BFGS-B (Zhu et al. 1997).
Algorithm D summarizes the training procedure. After DAG learning converges at all environments, we obtain the invariant structure of Y prediction by first computing Y-related columns in adjacency matrix W from shared of and then using threshold (please refer to Appendix B for details on how the threshold was selected) select the learned Pa(Y) (Y-related DAG). For the final target, we fix all parameters in Eq, 3 except h(.), and finc-tune h(.). To obtain the overall DAG for the entire datasct, we aggregate the DAG across different environments by keeping only the overlapping edges across all environments. Please note that h(-) always takes the output of g(.) as input and yields the prediction for Y. In practice, the parameters of h(.)
6

---

Published in Transactions on Machine Learning Research (10/2023)
(a) Step 1: Iteratively set each variable as target and propose invariant structure for each variable.
X1 X2
X
x2 XI
Constrained Leurning
Aggrepated
(b) Step 2:
Aggregate all proposed causal parents for each variable into a single graph (may not be DAG), then adjust the activation ability of DNN by de-activating all parameters that are related to the non non-proposed edges, and lastly optimize Eq. 2 to obtain final DAG.
Figure 3: ISL in self-supervised setting.
are updated twice: initially, s() and h(-) are combined to discover the causal parents of the target Y through the learning of 9(-), and ft-) merely assisting the learning of g(-) through the DAG constraint and ERM. h(-) maybe suboptimal for Y prediction at this stage. Then, after determining the causal parent the target variable Y by applying a threshold in the weight matrix W and reconstructing the DAG, we updated g(-) and fix it. Then, we fine-tune h(.) using only the causal parent variables discovered by gf)xee)) to predict the target variable Y.
Algorithm 1: Supervised Invariant Structure Learning
Input: Dataset D
Output: DAG, Y predictor f(X) = ho 0f
Build environments falt, for each e E call, pe = 1, a = O, (WW = co.
Termination conditions h(W)tol = 10 Pmax = 10 max iteration as NMAX with =
while i < NMAX = 100 and maxeer.n (h (W)) > h(W) and mineerau (p*) < Pmax do
++=1
4 for = to k do
Calculate (1 of (X)) and Ro (8, 0Y 00,00 x in Eg Update h, BY, 0% 0r 8x with L-BEGS-B (huu et al. Calculate W from of; Update (WWW;; Update pe and a
1997]
Summarize DAG from 0, Fix all trainable parameters in Eq.
except h, fine-tune h and obtain final f(X) = h o
3.3 Generalizing to self-supervised setting
In many scenarios, the target labels aren't available, rendering self-supervised causal graph discovery as an paramount problem. Conventional functional causal models, such as NOTEARS, aim to find a trade-off between three objectives, which are optimal to SEM: XW, whilst W should both resemble a DAG and be sparse (sce Eq. (3)). As all variables aren't distinguishable with equal importance, there no prior knowledge about which nodes should be the source target nodes. Due to the large variance among node distribution caused by variable semantic meaning, reconstruction accuracy driven learning is unstable, and sensitive to the variable distribution - some variables can be described using simple distribution, while others may be hard to estimate due to the differences in data source. It can lead to a local minima, causing the learned DAG deviate from the real causal st.ructure [Kisser & Sipos 2021 We propose a two-step DAG learning approach, as shown in Fig.
Step 1. Invariant causality proposal: We first build multiple environments. Then, we iteratively set each node Y (Fig.) and IST. to propose the invariant structure for Y as candidate causal parents.

---

Published in Transactions on Machine Learning Research (10/2023)
ISL keeps the invariant variables that are important to prediction of Y under the overall DAG constrain. As such, the learned invariant structure corresponds to either true causal parents of Y or the variables which have strong correlation with Y, thus treated as candidate of causal parents of Y).
Constrained graph discovery: We aggregate the candidate causal parents of each variable in Step 1 and form an aggregated graph (Fig. B) where there are bi-direction edges, which isn't allowed in a DAG. We can build a (d+1)x(d+1) binary adjacency matrix W to represent the graph where d+1 represent the number of nodes. j-th column of W represent the potential causal parents of node j. As described in DNN @ to reconstruct X₃ given other variables. There is a corresponding mapping between the j-th column of W and the first layer 0f of DNN 0i; the k-th row the parameter matrix of oj encode the contribution of node & to node j. which associate with the value of [Wa,l. To narrow down the search space and improve DAG Icarning, if [Way] is 0 (node h aren't potential causal parent of node summarized from Step 1), we deactivate the corresponding parameter by fixing the value of k-th row in Of as 0. if [Way] is we don't 't add weight constraint to the first layer of in the DAG. We use the parameter modification of DAG learning (Eq. 2 to obtain the final DAG.
Step 2.
Sec. 3.2 during DAG learning, for each variable X,,
constraint on DAG learning and run constrained version
Algorithm 2: Self-Supervised Invariant Structure Learning
Input:
Dataset
Output: DAG
# Step 1 Invariant causality proposal
Build environments, tall
for X; = X1 to Xa do
Set Y = X, and run algorithm (main manuscript), select only the Pa(X,)
# Step 2 Constrained graph discovery
Aggregate Pa(X,) for each variable to form the initial graph G (may not be DAG).
Summarize initial adjacency matrix W
Add weight constraint on A based on W
Run DAG mining (Eq.2 in main manuscript) to optimize 0 and yield DAG to describe the approximated
causal structure.
4 Experiments
In this section, we evaluate the proposed ISL framework for causal explainability and better generalization. We conduct exlensive experiments two settings based on the availability of larget labels: supervised learning tasks in Sec, 4.1 and self-supervised learning tasks in Sec. 4.2 Details and more results are provided in the Appendix D.
On causal explainability, we choose NOTEARS-MIPP Zhhng et al. 2020 GOLEM (g et al. 2020), and NoFcar (Wei et al. 2020) as the basclines for learning the SCM which represented as a DAG. On target prediction, we choose a standard MLP and CASTLE (Kyono et al. 2020 as the baseline met
Baselines:
We evaluate the estimated Y-related DAG and whole DAG structure using Structural Hamming Distance (SHD): the number of missing, falscly detected or reversed edges, lower the better. We evaluate the (arget (Y) prediction accuracy in Mean Squared Error (MSE). We compute SHD and the errors for multiple times and report the mean value.
Metrics:
4.1
Supervised learning tasks
4.1.1
Synthetic data
We first examine the performance ISL in accuratoly discovering the casual structure, as well as the target prediction performance using synthetic tabular datasets with known casual structure information as well the target labels. We aim to mimic challenging scenarios encountered for data generation and collection
8

---

Published in Transactions on Machine Learning Research (10/2023)
Table 1: Synthetic tabular data experiments in supervised learning setting. Note that black-box MLP and
CASTLE can't provide DAGs. ISL yields lower MSE for ID and OOD, and lower SHD.
Number of nodes
Metrics
MLP
NOTEARS-
CASTLE
GOLEM
NoFear
ISL (Ours)
ID MSE 0.008
0.090
0.012
0.171
0.422
0.005
(c=2, s=1)
OOD MSE 0.016
0.191
0.020
0.394
0.451
0.010
Average SHD
2
3
2
0
ID MSE 0.006
0.082
0.019
0.250
0.441
0.006
(c=2, s=2)
OOD MSE
0.014
0.152
0.032
0.250
0.411
0.009
Average SHD
2
ID MSE 0.004
0.093
0.020
0.250
0.427
0.004
(c=3, s=2)
OOD MSE
0.004
0.060 3
0.016
0.250
0.419
0.004
Average SHD
5
3
ID MSE 0.006
0.061
0.025
0.250
0.416
0.005
(c=4, s=5)
OOD MSF 0.031
0.174 4
0.160
0.250
0.407
0.005
Average SHD
10
4
0
ID MSE 0.008
0.051
0.137
0.250
0.403
0.008
20(c=10, s0
OOD MSE 0.018
0.251 9
0.252
0.250
0.431 9
0.007
Average SHD
19
processes in real world, that the data may consist of samples from different environment sources, while the target-related causal structure is consistent in the entire dataset (e.g., Fig. D We construct the synthetic
datasets in the following way (more details are provided in Appendix A):
Step 1: We randomly sample an initial DAG G' following Erdos- Renyi or Scale-Free schema with different edge densities. We randomly select one node (which isn't the source node) the target. Y. We calculate the number of causal parent nodes C of Y, If Cmins we randomly add Cmin number of nodes into
C' as the causal parents of Y.
Step 2: To simulate the spurious correlations, we create 8 E [1,... new nodes S, and these nodes act as cansal descendants of Y. After defining the cansal parents and descendants of Y, now we obtain the GT DAG G. For all nodes except Y and S, we define an ANM X = F(X) + e to generate data on top of G, where F' is a two-layer MLP whose parameters are uniformly sampled from (-2, -0.5) U (0.5, 2). is the external noise which is randomly sampled from Gaussian, Exponential and Uniform. Y is generated from its causal parents C, Y ~ P(Y [(i))) + +)). G can be either linear (uniformly random weight matrix) or non-linear (same initialization method as F).
Step 3: We randomly select the number of environments from the uniform distribution of [2, 5). For each environment, all nodes (except for s added spurious correlation nodes S ) in the GT DAG G follows the ANM (defined Step 2) but with different random seed and noise term. For S, their correlation to Y isn't invariant among environments and controlled by a continuous variable E [0, 1]. Specifically, for each node S; in S, S P(S = 1|Y = 1) = = P(S = olY = 0).
Step 4: We generate two different kinds of test set: In-distribution (ID) and Out-of-distribution (OOD). Both have the same number of environments with the training set. ID) test set uses the same value of training set, while OOD test set uses uniformly random sampled which represents the unknown test environments. S P(S P(S==g(Y))) ((Y) = r, P(S ~ random variable) = 1-r.
We generate different sizes of graphs with 5 and s combinations (sce Table. 1 (10 datascts with 1000 samples for each environment). Table shows that ISL significantly outperforms others for both Y prediction and Y-related DAG learning. Particularly in OOD scenarios, the outperformance is more prominent, up 83% decrease in MSE compared to black-box MLP and 96% decrease compared to CASTLE. This is attributed more accurate casual graph discovery (evident from lower SHD), allowing to capture dynamics that are consistent. between ID and QOD data, and hence the model generalizes better.
9

---

Published in Transactions on Machine Learning Research (10/2023)
Counterfactual simulations: Besides the accurate predictions for test data, causal structure discovery is also notable for its capability of accurate modeling of counterfactual outcomes, i.e. predicting how the output would change with certain input changes. To demonstrate this, we design experiments by modifying the dataset used above. We randomly select a node X; from causal parents set C or spurious correlation set S and change the value of X; while keeping the other nodes unmodified. Then, we test. the prediction accuracy on this dataset. Table. P shows that ISL yields more accurate counterfactual outcomes, compared alternatives. Particularly when the counterfactual source is spurious correlation variables, baseline methods are much worse at outcome predictions.
Table Synthetic labular data counterfactual simulation experiments. MSE is shown for various counterfac- tual outcomes, obtained by modifying the 'counterfactual source' variables.
Counterfactual source
ML.P
NOTEARS-MLP
CASTLE
IST. (Ours)
Causal parent X1
0.021
0,280
0.034
0.016
Causal parent X2
0.043
0.301
0.012 0.064
Spurions correlation S1
0.184
30.962
0.471
0.012
4.1.2 Real-world data
We perform supervised learning experiments on real-world datasets with GT causal structure: Boston Housing [Bndder et al. 1997 bos) and Insurance (Binder et a.l. 1997 ins) datasets. For each, we randomly split the train/ noan/validation/ stestttn//tttttteeeee test with the proportion 0...0.1/0.1. We conduct three experiments and show the average performance. We consider the accuracy for Y prediction and target-related DAG (causal parents of Y) learning. Specifically, Boston Housing contains information collected by the U.S Census Service concerning housing in Boston. There are attributes including 1 binary variable and 506 samples in which the median value of homes (MED)
be predicted. For ISL, we first calculate the Within- Within-Chuster-Sume of Squared (WSS) errors for different values of k, and choose the & for which the WSS starts to diminish. Based on this, we build k environments. We obtain the Y-related GT DAG of Boston Housing from (Wei & Feng 2021 Zhang et al. 2012). The Insurance dataset is based on network for insurance risk estimation. The network has 27 nodes and 52 edges with 20000 samples. The Insurance dataset provides the GT causal structure as a DAG. Three of the observable nodes (PrropCost', 'LiabilityCost' and "MeCCost') are designated as 'outputs'. Besides the designated output, we add other variables 'CarValue' (based on the importance for the task) as the larget as well. For ISL, similarly, we use K-means clustering to build & = 3 different. environments. Table. D summarizes the results for Y prediction and Y-related causal structure learning. We observe that ISL significantly outperforms black-box MLP in all cases (up to 74% MSE reduction), well NOTEARS-MLP and CASTLE.
Table 3: Supervised learning experiments on real-world data. Note that MLP and CASTLE cannot provide DAGs (and thus don't have SHD).
Dataset.
Target
Metries
MLP
NOTEARS-MLP
CASTLE
GOLEM
NoFear
ISL. (Ours)
MSE (L)
0.16
0.12
0.10
0.53
0.53
0.05
Boston Housing
MED
SHD (4)
2
3
3
MSE (L)
0.40
0.99
0.36
0.68
1.03
0.34
PPropCost'
SHD (4)
2
I
1
MSE (L)
0.69
L.03
0.55
0.86
0.99
0.52
'MelCost'
Insurance
SHD
2
4
4
MSE (L)
0.94
0.39
0.38
0.50
1.03
0.25
Coatillttyostt
SHD (4)
2
3
MSE
0.23
0.60
0.23
0.97
0.97
0.23
'Car Value
SHD (1)
2
6
4
10

---

Published in Transactions on Machine Learning Research (10/2023)
Boston Housing
GoodStudent
LST
DIS
Age
OtherCar
AGE
Con
SeniorTrain
RiskAversion
MED
HomeBase
Insurance
Theft
DrivingSkill
MakeModel
VehicleYear
Mileage
Cast
Car Cost
Accid
Cashioni
DrivWist
DrivQuality
AirBag
RuggeddAto
CarValus
Cont
Cost
Cushioning
Accident
AcoM OFF
Make
Wetccle
MedCost
LiabilitCCost
OtherCarCost
ThisCarDam
LINGI
Car
CarrCarCost
PropCost
(a) Y-related DAG (Supervised)
(b) Discovered DAG Insurance dataset (Self-uupervised)
Figure 4: Visualization of discovered causal structure in (a) supervised and (b) self-supervised settings. Blu solid arrows are overlapped edges between our results and GT, red solid arrow denotes the edges that can identify but with wrong direction, green solid arrow denotes our proposed edge that is not contained for, yellow dash arrows denotes our missing edges that GT contains.
Impact of the number of environments: The default number of environments we used in all experiments is 3. We investigate the impact of the number of environments on Boston Housing and synthetic data. Table 4 shows the clear value of having multiple environments with only one, the invariant constraint is not effective, yielding worse results. Increasing the number of environments has diminishing returns. More ablation studies are provided in the Appendix C.
Table 4: The inpact of number of enviromments for ISL in supervised learning setting.
Dataset
Target
Metrics
ISL (e=1)
ISL (e=2)
ISL (e=7)
MSE
0.067
0.051
0.051
Boston Housing
MED
SHD
MSE
0.110
0.022
0.020
Syuthetic data (Fig.
Y
SHD
4.2 Self-supervised learning
For self-supervised learning tasks, there no target variable, and the goal learn accurate SCM, represented as DAG, that represent the underlying causal structure of given dataset. We conduct experiments on two real-world datasets: Sachs (Sachs et al., 2005 sac) and Insurance (Binder et al. 1997 ins) datasets. The Sachs dataset is for the discovery of protein signaling network on expression levels of different proteins and phospholipids in human cells (Sachs et al. 2005). and is popular benchmark for causal graph discovery, containing both observational and interventional data. The true causal graph from (Sachs et al. 2005) contains 11 nodes and 17 edges. We conduct our two-stage DAG learning based on ISL by building 3 environments and compare the DAG results with different basclines. Table. shows that ISL outperforms all other methods in correct discovery of the GT DAG on both Sachs and Insurance. On the challenging Insurance data, the number of corrected edges is 72% higher for IST,, compared to NOTEARS-MI.P.
11

---

Published in Transactions on Machine Learning Research (10/2023)
Table Self-supervised causal graph discovery on the Sachs and Insurance datasets.
Total Edges
Correct Edges (1)
Dataset
Method
SHD ())
RL-BIC (Zhu al. 2019
10
7
11
GraN-DAG (Lachapelle et al. 2019
10
5
13
NOTEARS-MIN MLP Zheng et al.
2020
11
6
11
DAG-GNN Yuu et al. 2019
15
6
16
GOLEM Ng et al. 2020
11
6
14
Sachs
NOTEARS (Zhng et al.
2018
20
6
ICA-LINGAM (Shimizu al.
2006
8
4
CAM Glymmur al. 2019)
10
6
12
DARING He et al. 2021
15
7
11
ISL (Ours)
12
8
8
NOTEARS MLP Zheng al.
2020
35
18
39
Insurance
NOTEARS Zheng et al. 2018
24
10
46
GOLEM Ng et al. 2020
36
28
61
NoFear Wei et al. ISL (Ours)
2020
15
10
49
46
31
27
5
Conclusions
We propose novel method, ISL, for accurate causal structure discovery. The ISL framework is based on splitting the training data into different environments and learning the structure that is invariant to the selected target. We demonstrate the effectiveness of ISL in both supervised and self-supervised learning settings. On synthetic and real-world datasets, we show that IST. yields more accurate causal structure discovery compared to alternatives, which also results in superior generalization, especially against severe distribution shifts.
6 Limitations and Future Work
Our approach has proven effective in uncovering the causal structure by leveraging the assumption of its invariance across multiple environments. However, this method relies heavily on partitioning the dataset into distinct environments via clustering, a process that may present challenges in cases where data is characterized by high dimensionality or fluctuating densities.
Therefore, potential limitation of our current method is its dependency on successful clustering, and shortcoming may occur when clustering algorithms struggle due data complexity.
Moving forward, we aim to focus on developing enhanced algorithms with an emphasis more evenly distributing the dataset. By improving the partitioning of the data, we hope to increase the robustness and applicability of our approach in diverse data scenarios. In doing so, we anticipate further optimizing our method's capacity to accurately discern causal structures, thereby advancing the field's understanding and modeling of complex systems.
7 Acknowledgement
This work was in part supported by the National Science Foundation (award 2318101), C-BRIC (one of six centers in JUMP, Semiconductor Rescarch Corporation (SRC) program sponsored by DARPA) and the Army Research Office (W911NF2020053) The authors affirm that the views expressed herein are solely their own, and do not represent the views of the United States government any agency thereof.
12

---

Published in Transactions on Machine Learning Research (10/2023)
References
http://lib.stat. http://tb..saa..mm.cdii/ wtt.:aatw..cc.ttetaaaaaaaattttaa
http:///hhhaaiiiiee..crrarraaaa2 1105809
Sercan O. Arik and Tomas Pfister. Tabnet: Atlentive interpretable tabular learning, 2019.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv 2019.
preprinl 1rXXi:1907.000
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua. Bengio, and Simon Lacoste-Julien. closer look at. memorization deep networks, 2017.
Mohammad Taha Bahadori, Krzysztof Chalupka, Edward Choi, Robert Chen, Walter F Stewart, and Jimeng Sun. Causal regularization. arXiv preprint Xrv: Nee:1702.02644 2017.
Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. Acta Numerica, 30:87-201, 2021. doi: 101107/5000224222000777
John Binder, Daphne Koller, Stuart Russell, and Keiji Kanazawa. Adaptive probabilistic networks with hidden variables. Machine Learning, 29(2):213-244, 1997.
Yonggiang Chen, Yatao Bian, Kaiwen Zhou, Binghui Xie, Bo Han, and James Cheng, Rethinking invariant graph representation learning without environment partitions. 2023.
David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-554 2002.
Elliot Creager, -öen-k Jacobsen, and Richard Zemel. Environment inference for invariant learning. International Conference on Machine Learning, pp. 2189 2200. PMLR, 2021.
Alexander D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucie, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequcira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellic Webster, Steve Yadlowsky, Taedong Yun, Xiaohua Zhai, and D. Sculley. Underspecification presents challenges for credibility in modern machine learning, 2020.
Michel Fortin and Roland Glowinski. Augmented Lograngian methods: applications to the numerical solution of houndery-value problems. Elsevier, 2000.
Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Ruocheng Guo, Lu Cheng, Jundong Li, P Richard Hahn, and Huan Liu. A survey of learning causality with data: Problems and methods. ACM Computing Surveys (CSUR), 53(4):1 37, 2020.
Yue He, Peng Cui, Zheyan Shen, Renzhe Xu, Furui Liu, and Yong Jiang. Daring: Differentiable causal discovery with residual independence. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Dala Mining, pp. 596-69666 605, 2021.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdino,, Improving neural networks by preventing co-adaptation of feature detectors. arXio preprint arxi::1207 10070058, 2012.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-66) 67, 1970.
Biwei Huang, Kun Zhang, Yizhu Lin, Bernhard Schölkopf, and Clark Glymour. Generalized score functions for causal discovery. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1551 1560, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks, 2015. URL https://arxiv.org/ 1506.0006000000 02025
Dominik Janzing. Causal regularization. Advances in Neural Information Processing Systems, 32, 2019.
Marcus Kaiser and Maksim Sipos. Unsuitability of notears for causal graph discovery. arXiv preprint arXiv:2104.0544 2021.
Alex Krizhevsky, 1yy Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in newral information processing systems, 25, 2012.
Trent Kyono, Yao Zhang, and Mihacla van der Schaar. Castle: Regularization via auxiliary causal graph discovery. Advances Neural Information Processing Systems, 18:1501-15113 2020.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient-basse neural dag learning. XrXiv preprint arXiv:1906.0220 02226, 2019.
Lei Le, Andrew Patterson, and Martha White, Supervised autoencoders: Improving generalization performance with unsupervisod rogularizers. Advances in neural information processing systems, 31, 2018.
Bryan Lim, Sercan O. Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting, 2019. URL htttp:://aarri..org//bb////////s 19323
Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. Zin: When and how to learn invariance without environment partition? Advances Neural Information Processing Systems, 2444222,,,,, 2022.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2):129-13) 1982.
Luis Moneda. Exploring invariance in machine learning 2: Invariant risk minimization. Blogpost at igmoneda.github.iio io, 2021.
Andrew Y Ng. Feature selection, 1 vs. 1 2 regularization, and rotational invariance. In Proceedings of the hwesty-first international conference Machine learning, pp. 78, 2004.
Ignavier Ng, Emad Ghassami, and Kun Zhang. On the role sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems, 33:17943-17955 2020.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Scnior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio, 2016.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318. PMLR, 2013.
Judea Pearl. Causal inference in statistics: An overview. Statistics surveys, 3:96-148 2009.
Judea Pearl el al. Models, reasoning and inference. Combridge, UK: SngbbriddeUnivversttyyrrrrr 19:2, 2000.
Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. 2014.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Rebecca Roclofs, Vaishaal Shankar, Benjamin Recht, Sara Friilvich-Keil, Moritz Hardt, John Miller, and Ludwig Schmidt. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems, 32, 2019.
Peter Rousseeuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-68 65, 1987.
Karen Sachs, Omar Perez, Dana Pe'er, Douglas A Laufienburger, and Garry P Nolan. Causal protein-signalngg networks derived from multiparameter single-cell data. Science, 308(5721) 529, 2005.
Bernhard Schölkopf. Causality for Machine Learning, pp. 765-806 Association for Computing Machinery, New York, NY, USA, edition, 2022. ISBN 9781450395861. URL http:://00...rr/001111//11111111
3501755
Lukas Schoot, Julius von Kügglggen, Frederik Träuble, Peter Gehler, Chris Russell, Matthias Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain, 2021. URL https://arxxw..r///bbb.t 08221
Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyelic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
Donghee Shin. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable ai. International Journal of Human- ComuunnC Studies, 146: 142:51,,,,, 2021. ISSN 1071- 5819. doi: https://doi.org/ 11..0////./..//////1 atticlleepp///3177555119000333
http://..11/./.........01.. .jjlcs.2000.0255.. URL htttp://www.scienncee. cww.s/iencccocc//icccceccccccoee
Peter Spirles, Clark Glymour, Richard Scheines, and David Heckerman. Causalion, prediction, and search.
MIT press, 2000.
Peter L Spirtes, Christopher Meek, and Thomas S Richardson. Causal inference in the presence of latent. variables and selection bias. arXiv preprint arXi::1302.4493: 2013.
Robert L Thorndike. Who belongs in the family. In Psychometrika. Citeseer, 1953.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58 58(1):267 -28, 1996.
Stefan Wager, Sida Wang, and Percy Liang. Dropout training as adaptive regularization. Advonces in neural information processing systems, 26, 2013.
Dennis Wei, Tian Gao, and Yue Yu. Dags with no lears: closer look al continuous optimization for learning bayesian networks. Advances in Neural Information Processing Systems, 33:3895 3906, 2020.
Wenjuan Wei and Lu Feng. Nonlinear causal structure learning for mixed data. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 709 718. IEEE, 2021.
Larry Yaeger, Richard Lyon, and Brandyn Webb. Effective training of a neural network character classifier for word recognition. Aduances in neural information processing systems, 9, 1996.
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. Tn International Conference on Machine Learning, pp. 7154-7163. PMLR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107 115, feb 2021a.
Daniel Zhang, Saurabh Mishra, Erik Brynjolfsson, John Etchemendy, Deep Ganguli, Barbara Grosz, Terah Lyons, James Manyika, Juan Carlos Niebles, Michael Sellitto, Yoav Shoham, Jack Clark, and Raymond Perrault. The ai index 2021 annual report, 2021b.
Kun Zhang, Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Kernel- Keseel-based conditional independence test and application in causal discovery. XrXiv preprint ar Xiv: 1 1222.3775 2012.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advonces in Newral Informalion. Processing Systems, 31, 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametric dags. In International Conference Artificial Intelligence and Statistics, pp. 3414-3425. PMLR, 2020.
Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: T-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization. ACM Transactions on mathematical software (TOMS), 23 (4):550-560, 1997.
Shengyu Zhu, Ignavier Ng, and Zhitang Chen. Causal discovery with reinforcement learning. arXiv preprint 1S0::9900044777 2019.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Appendix
A Selection of ISL hyperparameters
Thresholds: As described in Sec. 3.2 and Algorithm 1, after Eq.3 converges at all environments, we employ a threshold & to convert the adjacency matrix W to a DAG. To find a proper threshold, we use the following stratogy. We set a minimum edges number Emin and a maximum odges number Emas based on the dataset information. Usually, Emin is half of the number of nodes |E//2 and Emar is 5/ We also set a range of threshold t E [imin: [maa] and step size t, base on the value range of W. Usually we use Emin min and Imaz max(W). Then, we employ grid search over the range [Imin: mmaz] with step size t,, and keep the thresholds and corresponding DAG that satisfied the following requirements: (1) The graph after the filtering with threshold should be DAG (no cyclicity). (2) The number of graph edges Emin < F < Fmaz]. For the selected threshold values and DAGs, we remove the duplicated group as different threshold may obtain the same DAG, which further refines the interval. Then, for cach threshold, we use the selected Pa(Y) as input to train a one-layer MLP to predict Y and select the threshold & that has smallest Y reconstruction error in the validation set.
For training of ISL, we different loss terms. The hyperparameter y controls the trade off between Y reconstruction and DAG constrain among environments. As we decrease the value of y, the training would focus more on the target Y reconstruction. We also have 4 regularization hyperparameters: (0) = 31110 3:|10[I| IlI + 32 |0 ll2 + fa|100 + 84] 10 where ||. |11 and || 1/2 denote 4 and l2 regularization. 31 controls the importance of the I regularization on the By, increasing 31 makes the selection of Pa(Y) more conservative (most of the values of the first column in W would be zero). f8 helps avoid overfitting of h(.). 83 and B4 controls the regularization on gx_ We choose the value of y and Br that achieves the smallest larget Y reconstruction on the validation set. We find the parameters: y 1; 81 = 0.001; 32 0.01; 33 0.01; $4 0.01 reasonable choices across many different settings, although they are not extensively optimized.
Regularization coefficients:
Table. 6 shows the results on Boston Housing for the prediction larget of median value of homes (MED) ISL with different regression parameters. We demonstrate that the results are not too sensitive to the change of regularization. That is bocause the regularization coofficients mainly influence the DAG Icarning process, and we apply fine-tuning for h(-) after convergence of DAG learning, which provides a mechanism to mitigate the differences at the first training stage.
Table 6: Boston Housing median value of homes (MED) target prediction results by ISL with different regression parameters.
B1
82
3s
84
MSE (1)
0.001
0.01
0.01
0.01
0.052
5
0.001
0.01
0.01
0.01
0.054
0.001
0.001
0.01
0.01
0.057
B Building environments
To show the elficacy of the proposed unsupervised environment building method based on k-means clustering, present comparisons to the setting with the environment building based on known data source information, i.c. the data comes with the indication on how the environments are split based on data collection or generation process. Table. shows that their difference is quite small (much smaller than the outperformance of ISL. compared to the ot her methods) and the proposed ISL. is highly effective and robust with unsupervised environment building by clustering.
We employed standard clustering evaluation techniques such as the Elbow and Silhouette Method. These methods enabled us to assess various cluster numbers systematically, identifying the point at which adding
17

---

Published in Transactions on Machine Learning Research (10/2023)
more clusters led to diminishing returns (the "elbow") and evaluating how well each object lies within its cluster (the silhouette score). More specifically, for the Elbow method, we plot the sum of squared distances from each point to its assigned center (known as inertia or within-cluster sum of squares) for a range of values of K (o.g., from 1 to 10) and automatically detect the "elbow' point on the plot, where the reduction in inertia starts to slow down. For the silhouetie method, we calculate its silhouette score, which measures how similar point to cluster compared to other clusters for each data point, and compute the average silhouette score for different values of K and plot them. An algorithm is then applied to look for an optimal value K that maximizes the average silhouette score across all data points. This process allowed to arrive at an optimal K value, and our extensive experimentation and analysis demonstrated that our approach remains robust across different numbers of environments. Our empirical evaluations provide further evidence of the effectiveness and stability of our selection method.
Table 7: Unsupervised vs. supervised environment building for ISL on synthetic data. We observe very small difference between them, showing the efficacy of the proposed unsupervised enviroument construction mechanism.
Number of nodes
Metries (4)
Supervised
Unsupervised
ID MSE
0.005 +0.0001
0.005
+.000011
3 (c=2, s=1)
OOD MSF
0.010
+0.0002
0.010
+000002
Average SHD
0
0+:0
ID MSE
0.006
+0.0002
0.006
+:000002
(c=2, s=2)
OOD MSE
0.009
#0.0001
0.009
#0.0001
Average SHD
0+:0
0+:0
ID MSE
0.004
10.0001
0.004
10.0001
(c=3, 8=2)
OOD MSE
0.004
+0.0001
0.004
+0.0001
Average SHD
0++0
0++0
ID MSE
0.004
+000006
0.004
+0.0006
9 (c=4, 5=5)
OOD MSF
+000001 0.005
0.005
+0000011
Average SHD
01:0
0
ID MSE
0.007
#000005
0.009
+.0.0005
20 (c=10, ==10)
OOD MSE
0.007
+0.0001
0.061 +0.009
Average SHD
1+:0
2+1
C Error statistics
In this section, we present the standard deviations for the errors to highlight the statistical significance of ISL improvements (Table. 8 and Table. D. Overall, the improvements ISL. are much larger than the standard deviation values. In addition, the variance of performance is observed to be lower for ISL compared to the other methods, indicating its superiority in robustness.
Table 8: Supervised learning experiment results on real-world data along with their standard deviations. Note that MLP and CASTLE cannol provide DAGs (and thus don't have SHD values).
MSE (L)
SHD (L)
Dataset
Target
MLP
NOTEARS-MLP
CASTLE
ISL (Ous)
NOTEARS-MLP
ISL (Ous)
Boston Housing
MED
0.16 ++0.22
0.12 +0.03
0.10 +0.01
0.05 +:0008
2 +0
1+00
PropCostt
0.40 +0.02
0.99 +0.02
0.36 40.001
0.34 +:0.004
||
0+0
Insurance
'MedCost'
0.69 10.09
1.03 +0.01
0.55 10.03
0.52 +00002
||
2:1
0+:0
"dabilityCostt
0.94 +0.08
0.39 +-0.11
0.38 4-0066
0.25 +0.0004
1:+:0
0+0
'CarValue'
0.23 :10011
0.60 4:005
0.23 +0003
0.23 200.0004
||
2 ::0
1::
18

---

Published in Transactions on Machine Learning Research (10/2023)
Table 9: Synthetic tabular data experiments in supervised learning setting. Note that black-box MLP and CASTLE can't provide DAGs. ISL yields lower MSE for ID and OOD, and lower SHD.
Number nodes
Metries (4)
MLP
NOTEARS-MLP
CASTLE
ISL (Ours)
ID MSE
0.008
10.002
0.101 10.010
0.016 10.007
0.005 10.0001
3 (c=2, s=1)
OOD MSE
0.016
10.002
0.195 10.005
0.017 10.004
0.010
10.0002
Average SHD
2 10
0110
ID MSE
0.006
10.009
0.087 +0.005
0.017 10.002
0.006
10.0002
4 (c=2, 2=2)
OOD MSE
0.040
=0.022
0.174 10.024
0.036 10.010
0.009
10.0001
Average SHD
2 +0
0+00
ID MSE
0.004
#0.002
0.110 #0.018
0.025 #0.006
0.004
#0.0001
5 [c=3, 2-2)
OOD MSE
0.004 #0.002
0.078 #0.020
0.019 #0.004
0.004
#0.0001
Average SHD
10
010
ID MSE
0.012
#0.006
0.070 #0.010
0.034
#0.010
0.004
#0.0006
9 (c=4, 5=5)
OOD MSE
0.052
10.024
0.201 10.028
0.152 10.022
0.005
10.0001
Average SHD
10
0100
ID MSE
0.009
00.008
0.061 +0.011
0.121 #0.021
0.007
+0.0005
20 (c= 10, s=10)
OOD MSE
0.094 #0.061
0.303 10.550
0.272 10.046
0.007 10.0001
Average SHD
=0
1::
Time Complexity and Scalability Comparison
With respect to runtime and scalability, our runtime to 5 times greater than that of NOTEAR ? due the additional clustering step. The computational complexity of our framework aligns closely with that of NOTEARS. Specifically, the complexity for NOTEARS-MI is given by O(nd2m + d dd) FLOPS per iteration of L-BFGS-B, where is the number of data samples, d is the number of nodes, and m is the number of edges.
In our framework, as we partition the dataset into different environments, each requiring its own convergence, the effective runtime becomes Olk(nd²m + dmm + d²)). However, since & (the number of environments) is generally a small constant typically ranging from to 5 the overall time complexity remains on the same order as O(ddmm + dm + dd).
The following table summarizes some of the quantitative results we have recorded. The time measurements were obtained on Apple M1 Pro chip with 16GB of memory.
Experiment
NOTEAR time (s)
ISL (ours)
Self-supervised ISL (ours)
x2s1
32.6
100.5
280.5
x2s2
48.1
139.7
400.2
x3s2
62.9
188.5
520.8
x4s5
139.0
420.1
1220.3
x10s10
414.5
1300.4
3600.4
Table 10: Time benchmarks for NOTEAR and our proposed methods.
E
Discussion
While we have indeed demonstrated our idca through an image classification task, this example serves more to elucidate the intuition of our algorithm rather than showcase a specific larget application. Our work primarily focuses on tabular data because most of the real world domains like economics, biology, and social social that we are interested applying the causal discovery algorithm gather data in structured, tabular form and most causal discovery algorithms such as conditional independence tests and structural learning algorithms are designed to work with structured data. We have evaluated our approach on two well-known real-world
19

---

Published in Transactions on Machine Learning Research (10/2023)
datasets: the Sachs dataset [9] and the Insurance dataset. Both of these datasets are widely recognized and extensively used as benchmarks in the field of causal discovery (Zheng et al. 2018 2020
He et al.
2021
Wei
et al. 2020 Yu et al. 2019 We have clarified this in the revised manuscript.
F
Limitations and the societal impact
In this paper, we propose novel method for causal structure discovery, which can improve the explainability and generalization of key machine learning use cases. Lack of their explainability remains to be a bottleneck for widespread adoption of DNNs for many high-stakes applications, such as from Healthcare, Finance, Public Soctor, Insurance, Legal etc. There are other forms of explainability methods used in practice, but since they cannot explicitly distinguish the causality from the correlations, there are many cases that they cannot satisfy the high bar for explainability in such applications. We believe that method constitutes contribution towards this, as it can be directly adopted to applications where obtaining accurate causal explanations is crucial. In some cases, causal explanations can uncover the undesired biases in the data such as when the dominant factor for the output label comes from one of the features that corresponds to sensitive attribute such as gender. these cases, the causal explanations can be further validated with additional analyses (as our model still far away from achieving the perfect SHD of 0 on complex real-world data with many features), and if they seem to be convincing, further data manipulation or model debiasing actions can be performed. In addition causal explainability, the improved generalization aspect. is expected to play major positive role, the distribution differences between training and testing settings sometimes hinder the reliability of machine learning models. In some applications where the data collection is limited to certain locations or times or subsets, our method can be utilized to enhance the performance of the trained modols when they are deployed to operate for different locations or times or subsets.
important.
Overall, we believe there significant. room for improvement in causal structure discovery. Especially for complex real-world data with many features, the obtained SHD values are not very low in the literature. Further rescarch in unsupervised environment building with better representation learning, end-to-end approaches in combining graph discovery and supervised learning, and adding more nonlinearity to the model to make it higher capacity in a systematic way, can be promising towards this direction. We demonstrate the robustness of our model in various settings, but further exploration of theoretical convergence guarantee can be useful as well. Lastly, methods to improve hyperparameter tuning and model selection with small validation data, without relying on ground truth causal graph structure, would be of high value.
20


Reviews:
[{"review_id": "3691ClXlI4", "reviewer": "Reviewer_kEJ3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe idea of using invariance across environments to learn causal structure is intuitive and reasonable. Environments provide a natural way to identify spurious correlations.\n2.\tExtending the method to a self-supervised setting is useful, since labeled data is not always available. The two-stage approach seems effective.\n3.\tThis paper is very well written and easy to follow.\n\nWeaknesses:\n1.\tThe evaluation is limited to a few well-known datasets. Testing on a wider variety of datasets would better demonstrate generalizability.\n2.\tThe environment construction process relies on clustering, which may not work well for high-dimensional or complex data.\n3.\tThe clustering for environment construction could be inefficient for large datasets. Analysis of scalability limitations would help understand applicability.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel framework called Invariant Structure Learning (ISL) to improve generalization performance and enable causal explainability by discovering the underlying causal graph structure. The key idea is to split the data into multiple environments and learn a model structure that is invariant across environments through an aggregation mechanism. This aims to capture stable causal relationships and avoid spurious correlations. The method is also extended to self-supervised learning by iteratively treating each variable as the target and proposing candidate causal parents. The proposed method is validated in both synthetic and real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.\tPlease provide more details on the environment creation process - how are optimal number of clusters and hyperparameters selected? \n2.\tPlease elaborate more on the time complexity and scalability compared to alternatives like NOTEARS.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QWaYQpjLEK", "reviewer": "Reviewer_uRxt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \nThis paper takes advantage of invariance in multiple environments in causal discovery, which is ignored by most research in causal discovery. As shown in several prior works, in machine learning, invariance is a significant indicator of causality. So it is an important research direction on how to utilize invariance in improving the accuracy of causal discovery. From this perspective, this paper could be of interest to TMLR's audience.\n\nWeakness:\nThe paper has some weaknesses, especially in the methodology component.\n1. It seems that the performance of the method heavily depends on the accuracy of the unsupervised clustering. However, the clustering is a stand-alone component and will not be updated during the training. Thus, it raises a concern about the sensitivity of the method on clustering, especially when the clustering is obtained using a relatively simple method K-means. There are no experiments evaluating this sensitivity.\n\n2. The proposed framework is not clearly described. \n\n- What are meanings of $\\theta, \\theta_r^Y, \\theta^X$? What is the difference between $X$ and $\\hat{X}$? What is the form of the function $\\theta(X)$? What is the relationship between $W$ and $\\theta$? The definitions of these notations need to be clearly stated.\n\n- 2.2 Funtion $g()$ with parameters $\\theta_1^Y$ is used to learn a representation of $Pa(Y )$. Is the surprised signal provided during the training? If so, this is not reflected in the loss function. If not, how does the model ensure that the representation learned by $g()$ represents $Pa(Y )$?\n\n- 2.3 In Eq. (1), how is the composition between $h$ and $g$ been done? In the paragraph above Eq. (1), the authors said that $h()$ inputs the representation and yields the prediction for $Y$. However, in the paragraph above Alg 1, the authors said the learned parents are selected using a threshold and then $h()$ is fine-tuned. So it is confusing.\n\n3. Why Newton method rather than gradient-based optimization algorithms like SGD or Adam is used to solve the optimization problem? \n4. Step 1 of self-supervised setting seems time-consuming as it needs to run ISL for every variable. What is the running time in the experiments?\n5. The baselines in the experiments are not designed to handle multiple environments. Why related methods like Invariant Risk Minimization [arXiv:1907.02893] are not included in the experiments?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a causal structure discovery approach that is based on invariance in multiple environments. The paper proposes a framework that combines prediction and structure learning so that both objectives will be improved during learning. Specifically, the paper first uses K-means to group the raw data into different clusters. Then, for each cluster, there is a module for learning the causal structure with a layer shared among all clusters to represent the parents of Y. In addition, a predictive model is trained to predict Y based on the learned parents of Y. Experiments using synthetic and real data are conducted to compare the proposed method with several causal discovery methods as baselines. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Improve the clarity of the methodology section.\n- Include multi-environment-aware causal discovery approaches in the experiments if possible.\n", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sehpJkiB4f", "reviewer": "Reviewer_nUZd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\ni) The problem setting is new.\n\nii) The idea of tying generalization and causal structure learning seems to be interesting.\n\n\n**Weakness**\n\ni) The motivation of this work is questionable. It’s unclear why OOD generalization and causal structure learning could contribute to each other.\n\nii) The discussion of multiple related work is missing.\n\niii) The clarity could be highly improved.\n\niv) No theory is provided and it’s unclear why ISL outperforms the baselines. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies causal discovery under distribution shifts. The authors argue that enforce the OOD generalization of the discovered causal structures could improve both generalization and SCM learning, and yield causal explainability. To implement the idea, the authors propose Invariant Structure Learning (ISL) that first clusters the data into different environments, and learn the an invariant SCM shared across different environments. The network is also required to learn the representation of causal parents of the target variable for OOD generalization. ISL is also generalized to the SSL setting where each variable will be tested for the target variable. Experiments on both synthetic and real-world tabular data verifies the effectiveness of ISL.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Although I believe the idea of invariant structure learning has some potential and could be impactful, the current version of the work fail to provide sufficient justification for why ISL is promising, and could be improved by addressing the following issues:\n\ni) First, the motivation of this work is questionable. It’s unclear why OOD generalization and causal structure learning could contribute to each other. It seems to have some logical issues, as either one of them could not bring *additional information* to each other. More specifically, if the model could succeed in OOD generalization, then the model already learns the representation of the direct causal parents of the target variable, which makes the causal structure learning nearly *done*, and vice versa. \n\nMoreover, although the authors demonstrate the idea in a real-world image classification task, the whole experiment focuses on tabular data. It’s unclear how ISL generalizes to real-world data.\n\n\nii) The discussion of multiple related works is missing, especially for the invariant learning without environment partitions, e.g., existing solutions [1], and feasibility when with auxiliary information [2] and without auxiliary information [3]. \n\n\n\niii) The clarity could be highly improved.\n- First, it’s unclear what the exact network is even with careful reading of the notation. In other words, what are $\\theta$s and how are they implemented?\n- Besides, there are too many typos in the notations that even adds up the difficulty to understand the work:\n    - In sec. 3.1, the line after ``Problem definition’’, it seems there is some space issues for the $\\hat{X}$.\n    - $\\theta$s in Eq.1 are not clearly defined.\n    - How are $W$s trained via objective (2)?\n    - What is $\\theta_f^Y$ in Eq. 3?\n    - The $n$ is used multiple times in line 1 Algorithm 1.\n- Important experimental details are lacking:\n    - What is the number of nodes in $G’$ and what exactly are the different densities?\n    - What are the number of environments specified in experiments?\n    - How the regularization weights are tuned in experiments?\n    - What is the exact neural architecture used in experiments?\n    - What are the variances in the results?\n\niv) No theory is provided and it’s unclear why ISL outperforms the baselines and generates accurate causal explanations.\n\n**References**\n\n[1] Environment Inference for Invariant Learning, ICML 2021\n\n[2] ZIN: When and How to Learn Invariance by Environment Inference? NeurIPS 2022.\n\n[3] Rethinking Invariant Graph Representation Learning without Environment Partitions, ICLR DG Workshop.\n\n", "broader_impact_concerns": "Yes", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for discovering causal structures indicating causal relationship by exploiting invariance across environments. This is done by a fairly ingenious pipeline: first use clustering to build sufficiently different environments from a common dataset. Then run causal structure learning on each, producing structures that can be aggregated. In addition, these structures are tied by requiring consistency across predictions when using them as part of model training. 

The authors provide convincing results on a set of synthetic and real data experiments, comparing with a variety of causal structure learning methods. 

All of the reviewers appreciated the work and saw evidence of the claims. They asked for a number of clarifications and improvements to the writing. The authors provided these and there is consensus for acceptance.
Decision: Accept (poster)