Title: Implicit Causal Representation Learning via Switchable Mechanisms
Venue: NeurIPS 2024
Accepted: no
Keywords: Causal Representation Learning

Implicit Model

Soft Intervention

Variational Inference
Abstract: Learning causal representations from observational and interventional data in the absence of known ground-truth graph structures necessitates implicit latent causal representation learning. Implicit learning of causal mechanisms typically involves two categories of interventional data: hard and soft interventions. In real-world scenarios, soft interventions are often more realistic than hard interventions, as the latter require fully controlled environments. Unlike hard interventions, which directly force changes in a causal variable, soft interventions exert influence indirectly by affecting the causal mechanism. However, the subtlety of soft interventions impose several challenges for learning causal models. One challenge is that soft intervention's effects are ambiguous, since parental relations remain intact. In this paper, we tackle the challenges of learning causal models using soft interventions while retaining implicit modeling. Our approach models the effects of soft interventions by employing a *causal mechanism switch variable* designed to toggle between different causal mechanisms. In our experiments, we consistently observe improved learning of identifiable, causal representations, compared to baseline approaches.

Abstract

Learning causal representations from observational and interventional data in the absence of known ground-truth graph structures necessitates implicit latent causal representation learning. Implicit learning of causal mechanisms typically involves two categories of interventional data: hard and soft interventions. In real-world scenarios, soft interventions are often more realistic than hard interventions, as the latter require fully controlled environments. Unlike hard interventions, which directly force changes in a causal variable, soft interventions exert influence indirectly by affecting the causal mechanism. However, the subtlety of soft interventions impose several challenges for learning causal models. One challenge is that soft intervention's effects are ambiguous, since parental relations remain intact. In this paper, we tackle the challenges of learning causal models using soft interventions while retaining implicit modeling. Our approach models the effects of soft interventions by employing a causal mechanism switch variable designed to toggle between different causal mechanisms. In our experiments, we consistently observe improved learning of identifiable, causal representations, compared to baseline approaches.

1 Introduction

One of the long-standing challenges in causal representation learning is how to recover the ground-truth causal graph of a system solely from observations. Termed the identifiability of causal models problem, this endeavor is crucial. Without achieving identifiability, we risk erroneously attributing causal relationships to learned representations. Furthermore, statistical models can masquerade as Directed Acyclic Graphs (DAGs) where edges lack causal significance, further complicating our pursuit.

When considering the challenge of identifying 29 causal models, it is known that the Markov con30 dition in graphs is insufficient for this task [26]. 31 Thus, without additional assumptions or data, 32 we find ourselves limited to learning only a 33 Markov Equivalence Class (MEC) of the causal 34 model. Existing works have made different 35 assumptions about availability of ground-truth 36 causal variables labels [34], model parameters 37

Soft

Intervention

Hard

Intervention

Observations

Object's class

Object's color

Fish

Bird

Bee

Cat

Spider

Bug

Classes

[1], availability of paired interventional data [3, 31], and availability of intervention targets [17] to 38

39

ensure identifiability of causal models.

Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.

Interventional data are usually obtained through soft or hard interventions. Hard interventions 40 usually involve controlled experiments and they severe the connection of an intervened variable 41 with its parents [24]. In terms of Structural Causal Models (SCM), hard interventions set the causal 42 mechanism relating a causal variable to its parents, to a constant. Due to ethical or safety reasons, it 43 may not be possible to perform hard interventions in many real-world applications. On the other hand, 44 the effects of soft interventions are more subtle since parent variables can still affect their children. 45 These effects can be modeled by a change in the set of parents, the causal mechanisms, and the 46 exogenous variables [7]. Consequently, hard interventions can also be seen as a special case of soft 47 interventions where the causal mechanism is set to a constant. Illustrated in Figure 1, a prominent 48 challenge in causal representation learning lies in dealing with the ambiguity surrounding the effects 49 of soft interventions. The observed alterations in object colors fail to distinctly elucidate whether 50 they stem from parental influences or the applied interventions. 51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

Additionally, a lack of comprehension regarding causal graphs can pose significant challenges in causal representation learning. In certain applications, the causal graph can be constructed using domain knowledge, allowing us to subsequently learn the causal variables [2, 18, 20]. However, this is not universally applicable, necessitating the direct learning of the causal graph itself. In a Variational AutoEncoder (VAE) framework, there are generally two approaches for causal representation learning: Explicit Latent Causal Models (ELCMs) [34, 1, 35, 37, 17, 15] and Implicit Latent Causal Models (ILCMs) [3]. In ELCMs, the latents are the causal variables and the adjacency matrix of the causal graph is parameterized and integrated into the prior of the latents such that the prior of latents is factorized according to the Causal Markov Condition [27]. This approach to causal representation learning is highly susceptible to becoming stuck in local minima as it is hard to learn representations without knowing the graph, and it is hard to learn the graph without knowing the representations. ILCMs [3] were introduced to circumvent this 'chicken-and-egg' problem by using solution functions , which can implicitly model edges in the causal graph rather than explicitly modeling the entire adjacency matrix of the causal model. In ILCMs the latents are the exogenous variables and the there is no explicit parameterization for the graph.

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

In implicit causal representation learning, the task involves recovering the exogenous variables E from observed variables X and learning solution functions. In [3], interventions are assumed to be hard, but this is often unrealistic and does not align with real-world problems. In this paper, we propose a novel approach for Implicit Causal Representation Learning via Switchable Mechanisms (ICRL-SM). We will introduce the causal mechanism switch variable as a way of modeling the effect of soft interventions and identifying the causal variables. Our experiments on both synthetic and large real-world datasets, highlight the efficacy of proposed method in identifying causal variables and promising future directions in implicit causal representation learning. Our key contributions can be summarized as follows:

I. A novel approach for implicit causal representation learning with soft interventions. II. Employing causal mechanisms switch variable to model the effect of soft interventions. III. Theory for identifiability up to reparameterization from soft interventions.

2 Related Work

Causal representation learning has recently garnered significant attention [27, 14]. The primary challenge in this problem lies in achieving identifiability beyond the Markov equivalence class [26]. Solely relying on observational data necessitates additional assumptions regarding causal mechanisms, decoders, latent structure, and the availability of interventional data [22, 28, 36, 25, 15, 1, 40, 13, 34]. Recent works have focused on identifying causal models from collected interventional data instead of making strong assumptions about functions of the causal model. Interventional data facilitates identifiability based on relatively weak assumptions [1, 6, 3, 39, 33]. This type of data can be further categorized based on whether it involves soft or hard interventions, and whether the manipulated variables are observed and specified or latent. Our focus in this paper is on examining soft interventions, encompassing both observed and unobserved variables.

2.1 Explicit models vs. Implicit models

Table 1 presents a comparison of the assumptions and identifiability results between our proposed theory and other related works on causal representation learning with interventions. In causal representation learning with interventions, one approach assumes a given causal graph and concentrates on identifying causal mechanisms and mixing functions. For instance, Causal Component Analysis (CauCA) [33] explores soft interventions with a known graph. Alternatively, when the graph is

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

not provided, explicit models seek to reconstruct it from interventional data [6, 17], potentially resulting in a chicken-and-egg problem in causal representation learning [3]. Current methods face the challenge of simultaneously learning the causal graph and other network parameters, especially in the absence of information about causal variables or the graph. Addressing these challenges, [3] recently introduced ILCM, which performs implicit causal representation learning exclusively using hard intervention data. In contrast, our approach introduces a novel method for learning an implicit model from soft interventions. [3] describes methods for extracting a causal graph from a learned implicit model, which could be applied to our method as well. In our experiments, we will compare our method with ILCM and dVAE [21], given their implicit nature and similar experimental settings and assumptions. Additionally, to showcase the superiority of our method over explicit models, we will employ explicit causal model discovery methods like ENCO [16] and DDS [5], in conjunction with various variants of β -VAE.

2.2 Hard interventions vs Soft interventions

The identification of explicit causal models from hard interventions has been extensively explored. [29] investigate causal disentanglement in linear causal models with linear mixing functions under hard interventions. Similarly, [4] focus on identifying causal models with linear causal mechanisms and nonlinear mixing functions, also utilizing hard interventions. In a more general setting with non-parametric causal mechanisms and mixing functions, [32] examine the identifiability of causal models, utilizing multi-environment data from unknown interventions. Similarly, [2] explore identifiability of causal models using multi-environment data from unknown interventions. [30] investigate the identifiability of causal models with nonlinear causal mechanisms and linear mixing functions, considering both hard and soft interventions.

Recent work has expanded the concept of explicit hard interventions to include soft interventions. In their study, [38] address the identification of causal models from soft interventions, leveraging the sparsity of the adjacency matrix as an inductive bias. However, when dealing with implicit models, soft interventions introduce new complexities. Identifiability becomes more challenging, as the causal effect of variables on observed variables is less apparent. This ambiguity arises from the dual possibility of effects originating from interventions or influences from parent variables on the causal variables. Moreover, in scenarios where implicit modeling is retained, the absence of knowledge about parent variables further complicates identifiability. While [3] theoretically establishes identifiability for hard interventions, practical experiments involving complex causal models with over 10 variables reveal increased ambiguity and confounding factors. Consequently, model identification becomes less straightforward.

3 Methodology

3.1 Data Generating Process

A structural causal model (Definition A1.1) is used to understand and describe the relationships between different variables and how they influence each other through causal mechanisms. A decoder function , g ( z ) = x , maps a vector of causal values z to observed values x . The causal variables Z are unobserved and the goal is to infer them from interventional data. For each causal variable, a diffeomorphic solution function , s i : E i → Z i , deterministically maps a value for exogenous variable E i to a value for causal variable Z i . In implicit modeling, we learn the solution functions s i directly, rather than defining them through local mechanisms f i . We write S for the set of all solution functions s i ∈ S , so S : E → Z .

Identifying causal models from data can be complex and is often studied within classes of models such as those identifiable up to affine transformations. For example, in the context of nonlinear Independent Component Analysis (ICA) , the generative process also involves a mixture function g of latent causal variables Z ∈ R n , resulting in observations X ∈ R n [15, 41]. However, a significant distinction between causal representation learning and nonlinear-ICA is that in the former, the causal

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

variables Z may have complex dependencies. Our objective in this paper is to recover E from X and eventually map E to Z using solution functions.

Identifying a causal model from observational data is not trivial and requires assumptions on the parameters of the model [1]. Adding information about interventions in addition to observations, helps to identify causal variables by exhibiting the effect of changing a causal variable on the observed variables. An interventional data point ( x, ˜ x, i ) includes the pre-intervention observation x , the postintervention observation ˜ x , and intervention target i ∈ I where I is the set of intervention targets selected from the causal variables. The post-intervention data ˜ x is generated by a soft intervention that targets one of the causal variables in Z . To achieve identifiability up to reparametrization, we rely on a series of assumptions within the data generation process, outlined as follows:

Assumption 3.1. (Data generating assumptions)

Atomic Interventions: For every sample ( x, ˜ x, i ) , only one causal variable is targeted by an intervention.

Known Targets: Targets of soft interventions are known.

̸

Post-intervention Exogenous Variables: The exogenous variables' values change only for the corresponding intervened causal variable, while the others maintain their pre-intervention values, thus e i = ˜ e i if i ∈ I ,and e i = ˜ e i otherwise.

Sufficient Variability: Soft interventions alter causal mechanisms to introduce sufficient variability [15]. These interventions should modify causal mechanisms to ensure non-overlapping conditional distributions of causal variables (refer to Figure A1).

Diffeomorphic decoder and causal mechanisms: Diffeomorphism guarantees no information loss and avoids abrupt changes in the function's image.

The known targets assumption can be relaxed in applications where such data is not available and the same procedure in [3] can be used to infer the intervention targets. In fact, in our realworld experiments, intervention targets are not available and based on the nature of the datasets, we hypothesize our causal variables to be object attributes and actions to be intervention targets.

3.2 Causal Mechanisms Switch Variable

The major difference of soft intervention with hard intervention is that post-intervention causal variable ˜ Z i is no longer disconnected from its parents and its causal mechanism ˜ s i is affected by the intervention. This is why identifying the causal mechanisms is more difficult for soft interventions. Soft intervention data yield fewer constraints on the causal graph structure than hard intervention data. For more details refer to string diagrams of soft and hard interventions depicted in Figure A5. Figure 2b shows our main generative model. It includes a data augmentation step that adds the intervention displacement ˜ x -x as an observed feature that directly represents the effect of a soft intervention in observation space.

Augmented implicit causal model To model the effect of soft interventions, we introduce the causal mechanism switch variable V [26]. By leveraging V , we can effectively switch to the preintervention causal mechanisms within post-intervention data. This facilitates the model's ability to solely focus on discerning alterations in the intrinsic characteristics of each causal variable. These changes are encapsulated within their respective exogenous variables, aiding the model in learning the causal relationships more accurately. We propose to use a modulated form of V to model the soft intervention effects on each causal variable as an additive effect with a nonlinear function h i such that ∀ i, ˜ Z i = ˜ s i ( ˜ E i ; ˜ E /i ) = s i ( ˜ E i ; E /i , h i ( V )) . As the parental set for each causal variable is not known, we have to use a modulated form of V in every causal variable's solution function and the inclusion of h i ( V ) enables the model to encompass variations in the parental sets of all causal variables in V . Therefore, there is a switch variable V i for each causal variable Z i . Adding switch variables to solution functions leads to the concept of an augmented implicit causal model .

Definition 3.2. (Augmented Implicit Causal Models) An Augmented Implicit Causal Models (AICMs) is defined as A = ( S , Z , E , V ) where V ∈ R n is the causal mechanism switch variable which models the effect of soft interventions on solution functions S :

where ˜ s i is the new solution function resulting from the soft intervention, ˜ E /i is the altered set of all exogenous variables except i , including the ancestral exogenous variables, due to intervention, and

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

The usage of V in soft interventions is analogous to augmented networks in [23] which were mainly designed for hard interventions. Pearl [23] even foresaw this possibility by saying: "One advantage of the augmented network representation is that it is applicable to any change in the functional relationship f i and not merely to the replacement of f i by a constant."

By using Taylor's expansion, we can expand the solution functions as follows:

where we'll use R i as a short-hand for Equation 2. We define the separable dependence property for solution functions as ∃ h i ( v 0 ) : s i ( ˜ E i ; E /i , h i ( v 0 )) = s i ( ˜ E i ; E /i ) . An example of such a scenario could be in location-scale noise models such as, s i ( ˜ e i ; e /i , h i ( v )) = ˜ e i + loc ( e /i ) + h i ( v ) = ˜ e i + loc ( e /i ) + v 2 + v where v 0 would be zero . By assuming the separable dependence property, we can write the solution function in Equation 2 as:

As a result, we can switch to pre-intervention solution functions. Subsequently, by modeling soft intervention effects using h i ( V ) , we can recover pre-intervention solution functions. During inference, we simply disregard the h i ( V ) term in the solution functions. Nonetheless, it is possible to train the prior p ( V ) to ensure that the separable dependence property is maintained for pre-intervention data.

Observability of switch variable The intuition behind using V is to separate the effect of soft intervention on ˜ Z i into two: (1) The effect on causal mechanisms and parents, and (2) The effect on exogenous variable E i . For example, we can say that causal variables in images of objects are the objects' attributes such as shape, color, and size, and performing actions like "Fold" change these attributes. Furthermore, it can be asserted that the camera angle within a given image may influence the shape of the object. If the images were generated from a hard intervention, the camera angle remains fixed between pre and post intervention. However, the camera angle changes along with the performed actions indicating that the interventions are soft. In this case, if we had a knowledge of how the camera angle affects the attributes of objects, then we could separate the effect of soft intervention. In other words, if V is observed, then we can extract the effect of the intervention that we are interested in (i.e., the effect on the causal variable itself). For more details, refer to Figure A4.

Lacking an understanding of how soft intervention influences the causal model, a more complex model becomes necessary. Consequently, the term R i in Equation 2 would involve a higher order of h i ( V ) . Therefore, we assume the observability of V :

Assumption 3.3. (Observability of V ) Given an intervention sample ( x, ˜ x, i ) and linear decoders, we can approximate the soft intervention effects h i ( V ) as follows:

where R = [ R 0 , R 1 , ..., R n ] and n is the number of causal variables. R and ∆ e i are the vectors indicating the soft intervention effects and change in effect of the exogenous variable of the intervened causal variable, respectively. Note that elements of R will be all zero except for the intervened causal variable. Consequently, with linear mixing functions and some pre-processing on observed samples (here subtraction), we can observe R i .

Our synthetic data is generated using a linear decoder, however, the decoder for the real-world datasets is not necessarily linear. Therefore, we do not observe V from ˜ x -x in the real-world dataset. Nevertheless, our findings suggest that incorporating soft interventions through V leads to superior performance compared to other implicit modeling approaches. Clearly, understanding the impact of soft interventions on the generative system of the dataset would result in improved outcomes.

3.3 Identifiability Theorem for Implicit SCMs with Soft Interventions

In this paper, our focus lies in identifying the causal variables up to reparameterization through soft interventions. We first define identifiability up to reparameterization (Definition 3.4) and subsequently introduce the identifiability theorem 3.5. The proof of theorem is extensive and is available in full in Appendix A1.

We establish identifiability up to reparameterization, allowing for the mapping of causal variables Z and Z ′ between two Latent Causal Models ( M and M ′ ) through component-wise transformations

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

M

V

V

=

"

!

⋮

"

"

Encoder

𝑿

Pre

Intervention

Post

Intervention

FC

FC

Causal

Mechanism

Switch

Location

Scale

Solution Function (S)

Encoder

𝑿 " -𝑿

𝑋 " 	 - 𝑋

M

V

! =

!

!

⋮

!

"

M

V

!̃ =

!̃

!

⋮

!̃

"

Decoder

Decoder

𝒁 "

𝑋

𝑋 "

𝑿 "

𝑒

𝑥 #

𝑥 " -𝑥

𝑥

𝑒̃

𝑣

𝑧

𝑧̃

̸

(Definition A1.2). Given our implicit modeling approach, lacking knowledge of the causal graph, we include all exogenous variables in the solution functions, as depicted in Equation 1. Notably, the causal graph remains unaltered during learning . To illustrate, we contrast hard interventions, which neglect parent influences, with soft interventions that acknowledge parental effects in a simple example. Consider a basic causal model Z 1 → Z 2 alongside a location-scale noise model [12] for the solution function, given by ˜ z 2 = ˜ e 2 -˜ loc ( e 1 ) ˜ scale ( e 1 ) . The distribution p ( ˜ Z 2 ) mean is 1 ˜ scale ( e 1 ) × mean ( ˜ E 2 ) -˜ loc ( e 1 ) ˜ scale ( e 1 ) In the context of hard interventions, we can assume p ( ˜ Z 2 |Z 1 ) = p ( ˜ Z 2 ) = N (0 , 1) as there are no parental effects. Consequently, the location and scale networks within the solution function tend to dampen parental effects, given the absence of parental influence in the ground-truth data. Contrarily, soft interventions exhibit parental influence in the ground-truth data, thus p ( ˜ Z 2 |Z 1 ) = N (0 , 1) . Due to the lack of parental knowledge in implicit modeling, we model p ( ˜ Z 2 |Z 1 ) = p ( ˜ Z 2 |E 2 ) , as E 2 is a known parent of ˜ Z 2 . Consequently, parental effects are propagated to E i (the corresponding exogenous variable of each causal variable), violating identifiability up to reparameterization. By leveraging V , we allow parental effects to propagate to V instead of E i .

Definition 3.4. (Equivalence up to component-wise reparameterization) Let M = ( A , X , g, I ) and M ′ = ( A ′ , X , g ′ , I ) be two Latent Causal Models (LCM) based on AICMs A , A ′ with shared observation space X , shared intervention targets I , and respective decoders g and g ′ . We say that M and M ′ are equivalent up to component-wise reparameterization M∼ r M ′ if there exists a component-wise transformation (Definition A1.2) ϕ Z from the causal variables Z to the causal variables Z ′ and a component-wise transformation ϕ E between E and E ′ such that:

1. Indices are preserved (i.e., ϕ i ( z i ) = z ′ i and ϕ i ( e i ) = e ′ i ). Corresponding edges are preserved (i.e., Z i →Z j holds in G iff Z ′ i →Z ′ j holds in G ′ . Edges E i →Z i should be preserved as well.)

The exogenous transformation preserves the probability measure on exogenous variables p E ′ = ( ϕ E ) ∗ p E (Definition A1.4).

3. The causal transformation preserves the probability measure on causal variables p Z ′ = ( ϕ Z ) ∗ p Z (Definition A1.4).

.

Theorem 3.5. (Identifiability of latent causal models.) Let M = ( A , X , g, I ) and M ′ = ( A ′ , X , g ′ , I ) be two LCMs with shared observation space X and shared intervention targets I Suppose the following conditions are satisfied:

Data generating assumptions explained in Assumption 3.1.

Soft interventions satisfy Assumption 3.3.

The causal and exogenous variables are real-valued.

The causal and exogenous variables follow a multivariate normal distribution.

Then the following statements are equivalent:

-Two LCMs M and M ′ assign the same likelihood to interventional and observational data i.e., p X , I ( x, ˜ x, i ) = p X , I ′ ( x, ˜ x, i ) .

-M and M ′ are disentangled, that is M∼ r M ′ according to Definition 3.4.

M

M

3.4 Training Objective

Consequently, there will be three latent variables in ICRL-SM:

A causal mechanism switch variable V .

The pre-intervention exogenous variables E . 286

3. The post-intervention exogenous variables ˜ E . 287

As the data log-likelihood log p ( x, ˜ x, x -˜ x ) ≡ log p ( x, ˜ x ) is intractable, we utilize an ELBO 288 approximation as training objective: 289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

The observations are encoded and decoded independently. The KLD term regularizes the encodings to share the latent intervention model p (˜ e | e, v ) p ( v ) p ( e ) that is shared across all data points. The components of this model can be interpreted as follows:

1. p ( e ) is the prior distribution over exogenous variables e .

2. p ( v ) is the prior distribution over switch variables v .

3. p (˜ e | e, v ) is a transition model that shows how the exogeneous variables change as a function of the intervention.

We factorize the posterior with a mean-field approximation q ( v, e, ˜ e | x, ˜ x ) = q ( v | ˜ x -x ) · q ( e | x ) · q (˜ e | ˜ x ) and, following our data generation model (Figure 2b), the reconstruction probability as p ( x, ˜ x | e, ˜ e, v ) = p ( x | e ) p (˜ x | ˜ e ) p (˜ x -x | v ) . The prior over latent variables is factorized as p (˜ e, e, v ) = p (˜ e | e, v ) p ( v ) p ( e ) (Figure 2b). Pre-intervention exogenous variables are mutually independent, hence, p ( e ) = Π i p ( e i ) and p ( v ) = Π i p ( v i ) . We assume p ( e i ) and p ( v i ) to be standard Gaussian. Furthermore, as we assume e i = ˜ e i for all non-intervened variables, the p (˜ e | e, v ) will be as follows:

The last equality is obtained from the Change of Variable Rule in probability theory, applied to the solution function ˜ z i = s i (˜ e i ; e /i , h i ( v )) . Furthermore, we write p (˜ z i | e, v ) = p (˜ z i | e i ) since only e i is a known parent of ˜ z i in implicit modeling. We assume p (˜ z i | e i ) to be a Gaussian whose mean is determined by e i . We implement the solution function using a location-scale noise models [12] as also practiced in [3], which defines an invertible diffeomorphism. For simplicity, in our experiments, we are only going to change the loc network in post-intervention. Therefore, h i ( v ) will be used as:

where loc i : R n -1 → R and scale i : R n -1 → R are fully connected networks calculating the first and second moments, respectively. The general overview of the model is illustrated in Figure 2a.

4 Experiments and Results

The experiments conducted in this paper address two downstream tasks; (1) Causal Disentanglement to identify the true causal graph from pairs of observations ( x, ˜ x, i ) , and (2) Action Inference to make supervised inferences about actions generated from the post-intervention samples using information about the values of the manipulated causal variables. Moreover, we conducted additional experiments designed as an ablation study, the results of which are presented in A4. All models are trained using the same setting and data with known intervention targets.

4.1 Datasets

Synthetic Dataset We generate simple synthetic datasets with X = Z = R n . For each value of n , we generate ten random DAGs, a random location-scale SCM, then a random dataset from the parameterized SCM. To generate random DAGs, each edge is sampled in a fixed topological order from a Bernoulli distribution with probability 0.5. The pre-intervention and post-intervention causal variables are obtained as:

where the loc and scale networks are changed in post intervention. The pre-intervention loc and post-intervention ˜ loc network weights are initialized with samples drawn from N (0 , 1) and N (3 , 1) , respectively. The scale is constant 1 for both pre-intervention and post-intervention samples. Both e i and ˜ e i are sampled from a standard Gaussian. The causal variables are mapped to the data space through a randomly sampled SO ( n ) rotation. For each dataset, we generate 100,000 training samples, 10,000 validation samples, and 10,000 test samples.

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

Action Datasets Causal-Triplet datasets tailored for actionable counterfactuals [19] feature paired images where several global scene properties may vary including camera view and object occlusions. Thus, the images can be viewed as outcomes of soft interventions, wherein actions affect objects alongside subtle alterations. These datasets [19] consist of: images obtained from a photo-realistic simulator of embodied agents, ProcTHOR [9], and the other contains images repurposed from a realworld video dataset of human-object interactions [8]. The former one contains 100 k images in which 7 types of actions manipulate 24 types of objects in 10 k distinct ProcTHOR indoor environments. The latter consists of 2,632 image pairs, collected under a similar setup from the Epic-Kitchens dataset with 97 actions manipulating 277 objects.Based on the nature of actions in this dataset, the causal variables should represent attributes of objects such as shape and color. As the dataset consists of images we train all the methods with ResNet encoder and decoder. For the ProcThor dataset the number of causal variables are 7. For the Epic-Kitchens dataset, we randomly chose 20 actions from the dataset as 97 causal variables will be too complex in a V AE setup.

4.2 Metrics

For the causal disentanglement task, we are going to use the DCI scores [10]. Causal disentanglement score quantifies the degree to which Z i factorises or disentangles the Z ∗ . Causal disentanglement D i for Z i is calculated as D i = (1 -H K ( P i. )) = (1 + ∑ K -1 k =0 P ik log K P ik ) where P ij = R ij ∑ K -1 k =0 R ik and R ij denotes the probability of Z i being important for predicting Z ∗ j . Total causal disentanglement is the weighted average ∑ i ρ i D i where ρ i = ∑ j R ij ∑ ij R ij . Causal Completeness quantifies the degree to which each Z ∗ i is captured by a single Z i . Causal completeness is calculated as C j = (1 -H D ( ˜ P .j )) = (1 + ∑ D -1 d =0 ˜ P dj log D ˜ P ij ) . D and K here are equal to the dimension of Z ∗ and Z which is n . For the action inference task, we will use classification accuracy as a metric. As we assume intervention targets are known, we train all models using known intervention targets for a fair comparison.

5 Results

5.1 Causal Disentanglement

We generated a dataset for the soft interventions and trained the models of ICRL-SM, ILCM, β -VAE and D-VAE for 10 different seeds, which generated 10 different causal graphs. We selected 4 causal variables to encompass complex causal structures, including forks, chains, and colliders. Table 2 displays the Causal Disentanglement and Causal Completeness scores for all models, computed on the test data.

The results in Table 2 indicate that our method ICRL-SM can identify the true causal graph in most 362 cases. The worst results are seen for graphs G 5 and G 10 . As mentioned in [27, 25], causal graphs are 363 sparse and in the G 5 case, where the graph is fully connected, the proposed method cannot identify 364 the causal variables well. Furthermore, in the next experiment we are going to examine the factors 365 affecting causal disentanglement such as the number of edges in the graph and the intensity of soft 366 intervention effect. These findings can explain why ICRL-SM cannot identify causal variables in 367 G 10 despite its sparsity. 368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

5.2 Factors Affecting Causal Disentanglement

In this experiment, we consider the graph G 3 , which has the best identifiability, and change the intensity of soft intervention and number of edges in its data generation process. To change the intensity, the post-intervention ˜ loc network weights are initialized with samples drawn from N (1 , 1) (almost similar to loc ) and N (10 , 1) (significantly different from loc ). To change the number of edges, we consider a chain and fully-connected graph.

The results in Table 4 further confirms the sparsity of causal graphs as the causal disentanglement is much worse in the fully-connected graph than the default graph of G 3 . The result for significantly different post-intervention causal mechanisms indicate that the switch variable cannot approximate intense effects of soft intervention and more supervision is required to observe V . Similar postintervention causal mechanisms also do not have sufficient variability to disentangle the causal variables as mentioned in Theory 3.5.

5.3 Action Inference

In this experiment, we show the performance of ICRL-SM in the real-world Causal-Triplet datasets. In these datasets V i.e., soft intervention effects, are not directly observable. Nevertheless, our findings suggest that incorporating soft interventions through V leads to superior performance compared to other implicit modeling approaches. Clearly, understanding the impact of soft interventions on the generative system of the dataset would result in improved outcomes.

The results in Table 3 indicate that when including all causal variables to predict actions, ICRL-SM performs at par with the baseline methods. However, including all causal variables in the action or object inference may cause spurious correlations. Therefore, we have also experimented with including only the related causal variable in action and object inference. In this setting, ICRLSM significantly outperforms the baseline methods which means that it can better disentangle the causal variables. We have also compared ICRL-SM with explicit causal representation learning methods. ENCO [16] and DDS [5] have variable topological order of causal variables during training. Furthermore, we have included a specific setting where the topological order is fixed during training. As shown in Table 4, our proposed method has superior performance to explicit models as well.

6 Conclusion

ICRL-SM, our novel model, enhances implicit causal representation learning during soft interventions by introducing a causal mechanism switch variable. Evaluations on synthetic and real-world datasets demonstrate ICRL-SM's superiority over state-of-the-art methods, highlighting its practical effectiveness. Our findings emphasize ICRL-SM's ability to discern causal models from soft interventions, marking it as a promising avenue for future research.

402

403

404

405

406

407

408

409

410

411

412

413

414

References

Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representation learning. In International Conference on Machine Learning, ICML , volume 202 of Proceedings of Machine Learning Research , pages 372-407. PMLR, 2023.

Shayan Shirahmad Gale Bagi, Zahra Gharaee, Oliver Schulte, and Mark Crowley. Generative causal representation learning for out-of-distribution motion forecasting. In International Conference on Machine Learning, ICML , volume 202 of Proceedings of Machine Learning Research , pages 31596-31612. PMLR, 2023.

Johann Brehmer, Pim de Haan, Phillip Lippe, and Taco S. Cohen. Weakly supervised causal representation learning. In NeurIPS , 2022.

Simon Buchholz, Goutham Rajendran, Elan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general nonlinear mixing, 2023.

Bertrand Charpentier, Simon Kibler, and Stephan Günnemann. Differentiable DAG sampling. In The Tenth 415 International Conference on Learning Representations, ICLR . OpenReview.net, 2022. 416

417

418

Gregory F. Cooper and Changwon Yoo. Causal discovery from a mixture of experimental and observational data, 2013.

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

Juan D. Correa and Elias Bareinboim. General transportability of soft interventions: Completeness results. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS , 2020.

Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100. Int. J. Comput. Vis. , 130(1):33-55, 2022.

Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems , 35:5982-5994, 2022.

Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations, ICLR , 2018.

Irina Higgins, Loïc Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR , 2017.

Alexander Immer, Christoph Schultheiss, Julia E. Vogt, Bernhard Schölkopf, Peter Bühlmann, and Alexander Marx. On the identifiability and estimation of causal location-scale noise models. In International Conference on Machine Learning, ICML , volume 202 of Proceedings of Machine Learning Research , pages 14316-14332. PMLR, 2023.

Amin Jaber, Murat Kocaoglu, Karthikeyan Shanmugam, and Elias Bareinboim. Causal discovery from soft interventions with unknown targets: Characterization and learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS , 2020.

Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems. CoRR , abs/2206.15475, 2022.

Sébastien Lachapelle, Pau Rodríguez, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In 1st Conference on Causal Learning and Reasoning, CLeaR , volume 177 of Proceedings of Machine Learning Research , pages 428-484. PMLR, 2022.

Phillip Lippe, Taco Cohen, and Efstratios Gavves. Efficient neural causal discovery without acyclicity constraints. In The Tenth International Conference on Learning Representations, ICLR . OpenReview.net, 2022.

Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M. Asano, Taco Cohen, and Stratis Gavves. CITRIS: causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, ICML , volume 162 of Proceedings of Machine Learning Research , pages 13557-13603. PMLR, 2022.

Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. Learning causal semantic representation for out-of-distribution prediction. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 6155-6170. Curran Associates, Inc., 2021.

Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkopf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centric causal representation learning. In Conference on Causal Learning and Reasoning, CLeaR , volume 213 of Proceedings of Machine Learning Research , pages 553-573. PMLR, 2023.

Yuejiang Liu, Riccardo Cadei, Jonas Schweizer, Sherwin Bahmani, and Alexandre Alahi. Towards robust and adaptive motion forecasting: A causal representation perspective. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages

466

17060-17071. IEEE, 2022.

Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael 467 Tschannen. Weakly-supervised disentanglement without compromises. In Proceedings of the 37th 468 International Conference on Machine Learning,ICML , volume 119 of Proceedings of Machine Learning 469 Research , pages 6348-6359. PMLR, 2020. 470

Chaochao Lu, Yuhuai Wu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal 471 representation learning for out-of-distribution generalization. In The Tenth International Conference on 472 Learning Representations, ICLR , 2022. 473

Judea Pearl. Causality , cambridge university press (2000). Artif. Intell. , 169(2):174-179, 2005. 474

Judea Pearl, Madelyn Glymour, and Nicholas P. Jewell. Causal inference in statistics: A primer . John 475 Wiley and Sons, 2016. 476

Ronan Perry, Julius von Kügelgen, and Bernhard Schölkopf. Causal discovery in heterogeneous environ477 ments under the sparse mechanism shift hypothesis. In NeurIPS , 2022. 478

Bernhard Schölkopf. Causality for machine learning. CoRR , abs/1911.10500, 2019. 479

Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh 480 Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE , 109(5):612481 634, 2021. 482

Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Weakly supervised 483 disentangled generative causal representation learning. J. Mach. Learn. Res. , 23:241:1-241:55, 2022. 484

Chandler Squires, Anna Seigal, Salil Bhate, and Caroline Uhler. Linear causal disentanglement via 485 interventions, 2023. 486

Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score-based 487 causal representation learning with interventions, 2023. 488

Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, 489 and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from 490 style. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances 491 in Neural Information Processing Systems , volume 34, pages 16451-16467. Curran Associates, Inc., 2021. 492

Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Keki´ c, Elias Bareinboim, 493 David M. Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal representations from 494 unknown interventions, 2023. 495

Liang Wendong, Armin Keki´ c, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gresele, 496 and Bernhard Schölkopf. Causal component analysis, 2023. 497

Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentan498 gled representation learning via neural structural causal models. In IEEE Conference on Computer Vision 499 and Pattern Recognition, CVPR , pages 9593-9602. Computer Vision Foundation / IEEE, 2021. 500

Shuai Yang, Kui Yu, Fuyuan Cao, Lin Liu, Hao Wang, and Jiuyong Li. Learning causal representations for 501 robust domain adaptation. IEEE Transactions on Knowledge and Data Engineering , pages 1-1, 2021. 502

Kui Yu, Xianjie Guo, Lin Liu, Jiuyong Li, Hao Wang, Zhaolong Ling, and Xindong Wu. Causality-based 503 feature selection: Methods and evaluations. ACM Comput. Surv. , 53(5), 2020. 504

Yue Yu, Jie Chen, Tian Gao, and Mo Yu. DAG-GNN: DAG structure learning with graph neural networks. 505 In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of 506 Machine Learning Research , pages 7154-7163. PMLR, 2019. 507

Jiaqi Zhang, Chandler Squires, Kristjan Greenewald, Akash Srivastava, Karthikeyan Shanmugam, and 508 Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions, 2023. 509

Jiaqi Zhang, Chandler Squires, Kristjan H. Greenewald, Akash Srivastava, Karthikeyan Shanmugam, 510 and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. CoRR , 511 abs/2307.06250, 2023. 512

Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. Dags with NO TEARS: continuous 513 optimization for structure learning. In Advances in Neural Information Processing Systems 31: Annual 514 Conference on Neural Information Processing Systems NeurIPS , pages 9492-9503, 2018. 515

Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: sparsity and beyond. In 516 NeurIPS , 2022. 517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

Appendix

A1 Proof of Identifiability Theorem

In order to prove our model is identifiable we need a two additional definitions and some previously stated assumptions.

Definition A1.1. Structural Causal Models

A structural causal model (SCM) is a tuple C = ( F , Z , E , G ) with the following components:

The domain of causal variables Z = Z 1 ×Z 2 × . . . ×Z n .

The domain of exogenous variables E = E 1 ×E 2 × . . . ×E n .

A directed acyclic graph G ( C ) over the causal and exogenous variables.

A causal mechanism f i ∈ F which maps an assignment of parent values for the parents Z pa i plus an exogenous variable value for E i to a value of causal variable Z i .

Definition A1.2. (Component-wise Transformation) Let ϕ be a transformation (1-1 onto mapping) between product spaces ϕ : Π n i =1 X i → Π n i =1 Y i . If there exist local transformations ϕ i such that ∀ i, j, ∀ x, ϕ ( x 1 , x 2 , ..., x n ) i = ϕ i ( x j ) , then ϕ is a component-wise transformation.

Definition A1.3. (Diffeomorphism) A diffeomorphism between smooth manifolds M and N is a bijective map f : M → N , which is smooth and has a smooth inverse. Diffeomorphisms preserve information as they are invertible transformations without discontinuous changes in their image.

Definition A1.4. (Pushforward measure) Given a measurable function f : A → B between two measurable spaces A and B , and a measure p defined on A , the pushforward measure f ∗ p on B is defined for measurable sets E in B as:

where ∗ denotes the pushforward operation. In other words, the pushforward measure f ∗ p assigns a measure to a set in B by measuring the pre-image of that set under f in the space A .

̸

Lemma A1.5. The transformation ϕ Z : Z → Z ′ between the causal variable of two LCMs M and M ′ defined in Definition 3.4 is a component-wise transformation, if ∀ i, j, i = j ˜ E ′ i ⊥ ⊥ ˜ E ′ j and the causal variables follow a multivariate normal distribution conditional on the pre-intervention exogenous variables where ˜ E ′ i denote the post-intervention exogenous variable of causal variable i in M ′ .

proof: We consider the case where the exogenous variables are mapped to causal variables by a

̸

.

let's add these three constants -E [ ˜ E ′ i ] ˜ loc ′ j ( e ′ /j ) , -E [ ˜ E ′ j ] ˜ loc ′ i ( e ′ /i ) , ˜ loc ′ i ( e ′ /i ) ˜ loc ′ j ( e ′ /j ) to the both 548 sides of the equality and then divide both sides by ˜ scale ′ i ( e ′ /i ) ˜ scale ′ j ( e ′ /j ) : 549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

Typically, the aforementioned equalities would be valid for any diffeomorphic solution function ˜ s i : ˜ E i → ˜ Z i . However, in this paper, we specifically focus on solution functions represented by a location-scale noise model.

Assuming that the causal variables follow a multivariate normal distribution conditional on the pre-intervention exogenous variables , cov ( ˜ Z ′ i , ˜ Z ′ j |E ′ ) = 0 would imply that ˜ Z ′ i ⊥ ⊥ ˜ Z ′ j |E ′ . Let's define ϕ E = g ′-1 ◦ g : E → E ′ where g and g ′ are the decoders in M and M ′ . As stated in Assumption 3.1, the decoders are diffeomorphism, hence, ϕ E is a diffeomorphism. Furthermore, let's denote ˜ s as the set of all solution functions in post-intervention which are also diffeomorphism as stated in Assumption 3.1. Consequently:

̸

The association between ˜ Z ′ and ˜ Z arises from their shared observation space. We know that every causal variable in M ′ depends at least on one of the causal variables in M . If one of the causal variables in M ′ depended on more than one causal variable in M , it would create dependency between two variables in M ′ and violate the above equality. Therefore, no variable in M ′ depends on more than one causal variable in M . Consequently, the transformation ϕ Z is a component-wise transformation.

Theorem A1.6. (Identifiability of latent causal models.) Let M = ( A , X , g, I ) and M ′ = ( A ′ , X , g ′ , I ) be two LCMs with shared observation space X and shared intervention targets I . Suppose the following conditions are satisfied:

1. Identical correspondence assumptions explained in 3.1.

Soft interventions satisfy Assumption 3.3. 569

3. The causal and exogenous variables are real-valued. 570

4. The causal and exogenous variables follow a multivariate normal distribution. 571

-Two LCMs M and M ′ assign the same likelihood to interventional and observational data i.e., 573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

p X M ( x, ˜ x ) = p X ′ M ′ ( x, ˜ x ) . -M and M ′ are disentangled, that is M∼ r M ′ according to Definition 3.4.

Proof We will proceed to prove the equivalence between statements 1 and 2 by showing the implication is true in each direction.

This direction is fairly straightforward. According to Definition 3.4, the fact that M ∼ r M ′ implies that ϕ E is measure preserving. Therefore, p E M ′ ( e ′ , ˜ e ′ ) = ( ϕ E ) ∗ p E M ( e, ˜ e ) . Furthermore, considering that ancestry is preserved, ϕ Z is measure preserving, and that causal variables are obtained from their ancestral exogenous variables in implicit models, we have p Z M ′ ( z ′ , ˜ z ′ ) = ( ϕ Z ) ∗ p Z M ( z, ˜ z ) . Since models are trained to maximize the log likelihood of p ( x, ˜ x, ˜ x -x ) and the latent spaces in M and M ′ have the same distribution, the decoders should yield the same observational distributions p X M ( x, ˜ x ) = p X M ′ ( x, ˜ x ) .

Let's define ϕ E = g ′-1 ◦ g : E → E ′ . Since we can express e = s -1 ( z ) , we can now define ϕ Z as

Therefore, ϕ E = s ′-1 ◦ ϕ Z ◦ s . Because g and g ′ are diffeomorphisms , ϕ E is a diffeomorphism as well. Furthermore, since p X M = p X M ′ and ϕ E is a diffeomorphism, then p E M ′ = ( ϕ E ) ∗ p E M . Consequently, ϕ E is measure-preserving. Similarly, ϕ E is measure-preserving as well since causal mechanisms are diffeomorphisms .

Step 1: Identical correspondence of edges and nodes Let's define the set U as U = {E ×E|∀ I, J ∈ I : supp p E , I M ( e, ˜ e | I ) ∩ supp p E , I M ( e, ˜ e | J ) } . Then, assuming atomic interventions and counterfactual exogenous variables , p E , I M ( U | I ) = p E , I M ( U | J ) = 0 . Therefore, we can say that p E M ( e, ˜ e ) = ∑ I ∈I p E , I M ( e, ˜ e | I ) p I M ( I ) is a discrete mixture of non-overlapping distributions p E , I M ( e, ˜ e | I ) . Similarly, we can say that p E M ′ ( e, ˜ e ) is a discrete mixture of non-overlapping distributions. It can be concluded that as ϕ E must map between these distributions, there exists a bijection that also induces a permutation ψ : [ n ] → [ n ] . Note: If we had non-atomic interventions or non-counterfactual exogenous variables, then these distributions would have some overlapping. With overlapping distributions, we can no longer claim there is a bijection mapping between these distributions.

In space Z , the interventions should also be sufficiently variable in order to have non-overlapping p Z , I M ( z, ˜ z | I ) distributions. In the case of soft interventions, ˜ z is affected by all ancestral exogenous variables which could be ancestors of other causal variables as well. Consequently, if the changes in causal mechanisms are not sufficient, the effect of ancestral exogenous variables on causal variables will share some similarities and create overlapping distributions. Similar to p E M ( e, ˜ e | I ) , we can say that there is a permutation between p Z M ( z, ˜ z | I ) as well. Furthermore, as we assume the target of interventions are known we have:

Consequently, the permutation ψ is an identity transformation. The effect of soft intervention with known targets on these conditional distributions is shown in Figure A1.

Step 2: Component-wise ϕ Z

According to Lemma A1.5, in order to prove that ϕ Z is a component-wise transformation, we need to prove that ˜ E ′ i and ˜ E ′ j are independent ∀ i, j, i = j . In implicit modeling we do not know the parents of each causal variable, hence, we assume the distribution of ˜ Z ′ i to be conditioned only on E ′ i as in Equation 5 since E ′ i is a known parent of ˜ Z ′ i . The mean of a conditional distribution can be calculated as:

̸

where ρ and σ are the correlation coefficient and variance of the random variables, respectively. On 616 the other hand, we model ˜ Z ′ i using switch mechanisms as: 617

0.6 0.4 0.20.0 0.2 0.4 0.6 0.8 1.0

X1

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

X2

Observed samples

Intervention on E1

Intervention on E2

(a)

Z1

10

0

10

20

30

40

10

0

10

20

30

40

P

0.00

0.01

0.02

0.03

0.04

0.05

M

M'

(c)

30

(d)

By using Taylor's expansion we can write above equation as: 618

Furthermore, we assume separable dependence such that: 619

An example of such a scenario could be in location-scale noise models, where a soft intervention 620 changes the location parameter of the model as: 621

Pre-Intervention

Z1

10

0

10

20

30

40

10

0

10

20

30

40

P

0.00

0.01

0.02

0.03

0.04

0.05

M

M'

(b)

Intervention on Z2

0.05

0.04

0.03

P

0.02

0.01

0.00

40

30

10

0

M

10

Z1

20

M'

0

40

10

20

10

In this example, for v ′ 0 = 0 , s i (˜ e ′ i ; e ′ /i , h i ( v ′ 0 )) = s i (˜ e ′ i ; e ′ /i ) . 622

Consequently, we can write the following equality from Equation 10: 623

By taking the partial derivative of both side with respect to ˜ E ′ j we have: 624

̸

625

626

627

628

629

630

If we did not have the causal mechanism switch variable ( h i ( V ′ )) , the equation above would only hold if s i was constant in parents, which is not the case due to the presence of soft interventions, or if ∂s i ( ˜ E ′ i ; E ′ /i ) ∂ ˜ E ′ i · ∂ ˜ E ′ i ∂ ˜ E ′ j = -∂s i ( ˜ E ′ i ; E ′ /i ) ∂ E ′ /i · ∂ E ′ /i ∂ ˜ E ′ j . The latter scenario would imply that ∂ ˜ E ′ i ∂ ˜ E ′ j = 0 , hence, ˜ E ′ i ̸⊥ ⊥ ˜ E ′ j . However, by introducing the causal mechanism switch variable V and assuming it is observed, we can account for the effects of soft interventions through h i ( V ′ ) . In this case, ∂ ˜ E ′ i ∂ ˜ E ′ j = 0 as exogenous variables are commonly assumed to be independent in practice. Consequently:

"

𝑧̃

!

"

𝑧̃

!

𝑧

𝑧̃

!

"

𝑧

𝑰

𝑰

𝜑

!

!

𝑧̃

#

"

=

𝑧̃

#

"

𝜑

!

"

𝑓

"

𝑓

#

𝑓

#

$

𝜑

!

𝑓

"

$

𝑽

𝑽

"

(a)

𝑧̃

!

𝑧̃

!

𝑧

𝑧̃

!

"

𝑰

𝑰

𝜑

!

!

=

𝑓

"

𝑓

#

𝑓

"

$

(b)

Step 3: Component-wise ϕ

E 631

Using the result from previous step that ϕ Z is a component-wise transformation, the string diagrams 632 for connections between E and E ′ will be as shown in Figure A3. ϕ E i will only depend on E A , 633 where A = anc i is the ancestors of variable i , and e i . Because s ( e ) anc i , s ( e ) i , and s ′-1 ( z ′ ) i only 634 depend on ancestors and ϕ Z is a component-wise transformation. The first equality in Figure A3 635 follows from the definition of ϕ E i . The second equality holds when we first apply ϕ Z A and then apply 636 the causal mechanisms. It can be concluded from the most right-hand side diagram in Figure A3 637 that the transformation from E ′ i ×E A →E ′ i is constant in E A . Therefore, ϕ E i is a component-wise 638 transformation. 639

̸

640

𝑓

!

" #$

𝑓

!

𝑠

"

𝑓

!

" #$

𝑓

!

#

𝑠

"

=

=

𝜑

$

!

𝜀

!

𝜀

"

𝜀

"

#

𝜀

!

𝜀

"

𝑧

!

𝑧

"

𝜑

%

"

𝜑

%

!

𝑧

"

#

𝑧

!

#

𝜀

"

#

𝜀

"

#

𝑧

"

#

𝑧

!

#

𝜀

"

#

𝜑

%

"

𝑧

!

𝜀

!

(a) Pre-Epic-Kitchens

(b) Pre-Epic-Kitchens

(c) Pre-Epic-Kitchens

(d) Pre-Epic-Kitchens

(a) Post: Valve-locked

(c)

Post: Clothes-Gathered

(d) Post: Juice-Poured

(e) Pre-ProcTHOR

(f) Pre-ProcTHOR

(g) Pre-ProcTHOR

(h) Pre-ProcTHOR

(e) Post: Cabinet-Open

(f) Post: Box-Open

(g) Post: TV-Broken

(h) Post: TV-On

A2 Soft vs. Hard intervention

In a causal model, an intervention refers to a deliberate action taken to manipulate or change one or 641 more variables in order to observe its impact on other variables within the causal model. Interventions 642

help to study how changes in one variable directly cause changes in another, thereby revealing causal 643 relationships. 644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

𝑧̃

!

𝑰

𝑧̃

"

𝑠

!

𝑒

/!

𝑠̃

"

(a)

𝑧̃

!

𝑒

/"

𝑰

𝑠̃

!

𝑠

"

𝑧̃

$

(b)

Based on the levels of control and manipulation in a causal intervention, we can have soft vs. hard interventions. A hard intervention involves directly manipulating the variables of interest in a controlled manner such as Randomized Controlled Trials (RCTs). In other words, a hard intervention sets the value of a causal variable Z to a certain value denoted as do ( Z = z ) [24].

On the other hand, soft intervention involves more subtle or less controlled manipulation of variables and changes the conditional distribution of the causal variable p ( Z | Z pa ) → ˜ p ( Z | Z pa ) which can be modeled as ˜ z i = ˜ f i ( z pa i , ˜ e i ) [7].

Looking at interventions from a graphical standpoint, a hard intervention entails that the intervened node is solely impacted by the intervention itself, with no influence coming from its ancestral nodes. Conversely, in the context of a soft intervention, the representation of the intervened node can be influenced not only by the intervention but also by its parent nodes.

As an example, suppose we are trying to understand the causal relationship between different types of diets and weight loss. The soft intervention in this scenario could be a switch from a regular diet to a low-carb diet. Switching to a low-carb diet is a voluntary choice made by the individual and there are no external forces or regulations compelling them to make this change (non-coercive).

The intervention involves a modification of the individual's diet rather than a complete disruption since they are adjusting the proportion of macronutrients (fats, proteins, and carbs) they consume, which is less disruptive than a radical change in eating habits (gradual modification). The individual has autonomy to choose and tailor their diet according to their preferences and health goals so they are empowered to make informed decisions about their dietary choices (behavioural empowerment).

Conversely, if the government or an authority were to intervene and enforce a mandatory low-carb diet through legal means, this would constitute a hard intervention . In this scenario, regulations would be implemented, prohibiting the consumption of specific carbohydrate-containing foods. Regulatory agencies would be established to oversee and ensure adherence to the low-carb diet mandate, taking actions such as removing prohibited foods from the market, restricting their import and production, and so on. Individuals caught consuming banned foods would be subject to fines, legal repercussions, or other penalties.

A3 Experiments

This section contains additional details about ICRL-SM design architectures, datasets, and experiments settings.

A3.1 Datasets

A3.1.1 Synthetic

We generate simple synthetic datasets with X = Z = R n . For each value of n , we generate ten random DAGs, a random location-scale SCM, then a random dataset from the parameterized SCM. To generate random DAGs, each edge is sampled in a fixed topological order from a Bernoulli

distribution with probability 0.5. The pre-intervention and post-intervention causal variables are 680 obtained as: 681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

where the loc and scale networks are changed in post intervention. The pre-intervention loc and post-intervention ˜ loc network weights are initialized with samples drawn from N (0 , 1) and N (3 , 1) , respectively. For ablation studies, we change the mean of these Normal distributions. The scale is constant 1 for both pre-intervention and post-intervention samples. Both e i and ˜ e i are sampled from a standard Gaussian. The causal variables are mapped to the data space through a randomly sampled SO ( n ) rotation. For each dataset, we generate 100,000 training samples, 10,000 validation samples, and 10,000 test samples.

A3.1.2 Causal-Triplet

The Causal-Triplet datasets are consisted of images containing objects in which an action is manipulating the objects shown in Figure A4. Examples of actions and objects in these datasets are given in Table A1 and A2.

ProcTHOR Dataset

Epic-Kitchens Dataset

Based on the actions and objects, we treat our causal variables as attributes of objects which can be changed by actions. Therefore, actions in these datasets are considered as interventions. Assume that z 1 corresponds to the attributes of an object, e.g. a door, the target of opening or closing (action's target) is z 1 .

We use actions' labels in these datasets to detect the targets of interventions to determine which causal variable has been intervened upon. Note that informing the model about the target of intervention is not same as informing about the action itself (See Table 3). We use 5000 images of these datasets to train all models.

A3.2 Architecture Design

Based on the ICRL-SM architecture depicted in Figure 2a, we devised a location-scale solution function (Equation 6) in which the loc i and scale i , and h i networks each comprise of fully connected networks. These networks consist of two layers each, with 64 hidden units per layer and ReLU activation functions. The encoder and decoder parameters for latents E and ˜ E are shared and we use a separate encoder and decoder with the same architecture for the latent V . For our synthetic dataset experiments, the encoder and decoder are consisted of fully connected networks with 2 hidden layers and 64 units in each hidden layer. For the Causal-Triplet datasets, we utilized ResNet-based networks. The same encoder and decoder architectures are used for all baseline models in the experiments. ResNet50 encoder, ResNet50 decoder, and classifiers with 1 hidden layer and 64 hidden units are used for predicting actions and objects for experiments in Table 4 and Table 3. ResNet18 encoder, ResNet18 decoder, and classifiers with 2 hidden layer and 2 hidden units are used for predicting actions and objects for experiments in Table A4 and Table A3.

A3.3 Training

To enforce the condition described in Equation 5 for i / ∈ I , we assign the post-intervention exogenous variables the same value as the pre-intervention exogenous variables. In mathematical terms, this translates to ∀ i / ∈ I , we set ˜ e i = e i .

In our experiments, we do not pretrain the networks, however, for the baseline models we follow the 718 training procedure in [3]. We also use consistency in our experiments to ensure that the encoder and 719 decoder are inverse of each other. Consistency regularizer is used as ∑ i E ˆ x ∼ p (ˆ x | e ) ,x ∼ p ( x ) [( x -ˆ x ) 2 ] 720 where ˆ x are the reconstructed samples. 721

722

723

724

725

726

727

728

729

For optimization, Adam optimizer is used with default hyperparamters. In the synthetic experiments, learning rate changes from 3 e -4 to 1 e -8 with a cosine scheduler. In the Causal-Triplet experiments in Table 4 and Table 3 learning rate changes from 0 . 002 to 1 e -8 with a cosine scheduler. For Table A4 and Table A3 experiments earning rate changes from 0 . 0001 to 1 e -8 with a cosine scheduler. In all experiments the batch size is set to 64. In the main Causal-Triplet experiments we train the models for 400 epochs, in the appendix Causal-Triplet experiments we train the models for 2000 epochs, and in the synthetic experiments we train the models for 100 epochs. In the appendix experiments, the graph parameters for explicit models are frozen after 1000 epochs.

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

All models are trained using Nvidia GeForce RTX4090 GPUs. Each of the Causal-Triplet experiments takes 3-8 hours to train the models and each of the synthetic experiments takes 2-3 hours to train the models.

We save the models' weights with best validation loss and evaluate them using those weights with test data.

A4 Ablation study

A4.1 Scalability

While our primary research objective centered on addressing identifiability challenges in implicit causal models under soft interventions, we also conducted an investigation into the scalability of our proposed model. To comprehensively assess its performance, we designed experiments covering a range of causal graphs, featuring 5 to 10 variables, with 10 different seeds for each variable, following a similar experimental setup as our 4-variable causal graph experiments. The outcomes of these experiments, comparing ICRL-SM and ILCM, are presented in Figure A6. By increasing the number of variables in the graph, confounding factors and ambiguities of causal relations increase as well.

Consequently, more supervision on

V

themselves on the observed variables.

Means with Standard Deviations

D4

D5

D6

D7

D8

D9

D10

Causal Variables

0.4

0.5

0.6

0.7

0.8

0.9

Causal Disentanglement Score

ICRL-SM (mean)

ICRL-SM (std)

ILCM (mean)

ILCM (Std)

A4.2 Backbone model 746

We trained the models using a simpler backbone model, ResNet18, to see how it affects performance. 747 The input image resolution is 64 × 64 and we use the intervened causal variables to predict action 748

is required to better separate the effect of causal variables

and object classes. The results are shown in Table A4 and A3. It can be seen from the results that the 749 proposed method outperforms other explicit and implicit models even with a simpler model. 750

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our contributions include identifiability of causal models with soft interventions. In the proposed methods section we give the theory and assumptions for the identifiability result and in our experiments we evaluate our method using datasets generated by soft interventions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have some strict assumptions on data generation process and model which are given in Assumptions 3.3 and 3.1 which may not be plausible to satisfy in some applications.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We give the full set of our assumptions in the proposed method section and the detailed proof in Appendix A1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer:[Yes]

Justification: We provide the full details of our model architecture and training settings in Appendix A3 and in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our anonymized codes which contains the necessary scripts and instructions to run the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the full details of our model architecture and training settings in Appendix A3 and in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In our synthetic experiments we initialized the causal graph in the dataests with different seeds. The results of these different seeds are provided in Table 2 and Figure A6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The details are given in Appendix A3.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

1059

1060

1061

1062

1063

1064

1065

1066

1067

1068

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We only have a code repository for replicating experiments and we have submitted the anonymized zip file with our submission.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies weakly supervised causal representation learning with soft interventions.\nIt assumes that pairs of observations are provided that differ by a soft intervention on one variable in the latent space.\nAdditionally, the intervention target (and the total number of latent variables) are given.\nIdentifiability up to element-wise transformations assuming linear mixing and known intervention targets (among other assumptions) is shown.\nExperiments on synthetic data compare the proposed method to ILCM, beta-VAE and D-VAE.\nAn improved D and C score in the DCI framework is shown.\nAn experiment on semi-synthetic image data is shown.\n\nSTRENGTHS:\n- The method extends previous work on weakly supervised CRL and generalizes it from hard to soft interventions, which is an important step for bringing CRL closer to real-world applications, since CRL methods often suffer from restrictive assumptions.\n- Also, I appreciate the effort of testing this CRL approach to datasets which are more similar to real-world image datasets.\n\nWEAKNESSES:\n- The mathematical notation is poorly defined and difficult to follow: objects aren't well-defined and notation inconsistently or unusually used. See questions below.\n- L136-137: \"a diffeomorphic solution function [...] deterministically maps a value for exogenous variable [...] to a value for causal variable [...]\" That's incorrect. There can be such a map from the set of all exogenous to all endogenous, but not per variable. Consider $X:= U_X; Y:= X+U_Y$, where $U_X$, $U_Y$ are exogenous. In general, there is no deterministic map from $U_Y$ to $Y$.\n- Even though the DCI framework for evaluation of the learned representation is cited, only two of the metrics (D and C) are used, whereas Informativeness (I) is omitted without comment. That seems suspicious to me.\n- The synthetic experiments test a rather simple setting on few data generating processes. The observation dimension is 4 (same as latent dimension) and the mixing is linear (as far as I can tell). We have seen in previous work (e.g. [2]) that the complexity of the mixing function is a crucial element in the recoverability of the latents. For example, the more nonlinear the function, the harder it is to recover latents. Furthermore, only 10 data generating processes are tested. That seems quite limited, given that the dimensions of the data is so small.\n\n\n**Minor:**\n\n- L133: \"decoder function\" is a bit of an odd choice for something that relates to the ground-truth data generating process. Usually, this is a part of a model. \"Mixing function\" would be more appropriate.\n- There is a period missing in L251.\n- L345: There is a follow-up version of DCI that takes more aspects of the learned representations into account [1]. It could be considered for the next iteration if time permits.\n- L363: \"As mentioned in [27, 25], causal graphs are sparse and in the G5 case, where the graph is fully connected, the proposed method cannot identify the causal variables well.\" The two references **do not say that causal graphs are sparse**. You may refer to the sparse mechanism shift hypothesis, which says that changes between environments are assumed to stem from changes in few mechanisms of the causal graph. This is a statement about sparse changes, not sparse graphs.\n\nQUESTIONS:\n- L136: $i$ isn't defined, I suppose it's the index of the causal variables. However, later it's defined as an element of the intervention targets. Which one is it?\n- L138: What is $\\mathcal{S}$ is it a set of functions $\\mathcal{E}_i\\to\\mathcal{Z}_i$ (as suggested by the first statement in the sentence) or is it a map $\\mathcal{E}\\to\\mathcal{Z}$?\n- L187: Previously, $s_i$ was introduced as a function $\\mathcal{E}_i\\to\\mathcal{Z}_i$ (L136), now it seems to be a function $\\mathcal{E}\\to\\mathcal{Z}_i$. Which one is it?\n- L204: What does this equation mean? How come you are using the same function but with a different set of arguments? What does it mean when one argument disappears? \n- Equation 2: Is the Taylor expansion necessary here, or is the expansion used elsewhere? To me, the LHS can be transformed to the RHS in any case, since $R_i$ is just some variable that is introduced for the difference. If it's not used elsewhere, I would suggest removing it since it's only a distraction and doesn't add to the method or understanding.\n- Assumption 3: I'm confused by the approximate relation. Either $g$ is linear and has no intercept term (\"truly linear\"), then $g(\\tilde{z}) g(z)=g(\\tilde{z} - z)$; or it has an intercept term (affine), then the difference between LHS and RHS can be arbitrarily large and thus writing approximately equal is inappropriate.\n- Assumption 3: What exactly is the assumption? Is it that $g$ is linear? The rest seems to (more or less) follow from linearity and the statements made before.\n- L325: \"loc and scale networks are changed in post intervention\", but from Equation (7) it seems only loc is changed?\n- Equation 7: Are the loc parameters during data generation generated by random neural networks? Do they have the same architecture as the network that estimates those parameters?\n- The experiment code seems to indicate that the following seeds were selected for synthetic experiments: \"nature_seed=5 # 1, 2, 5, 6, 7, 8, 9, 10, 11, 12\" Why were seeds 3 and 4 omitted?\n- I struggle to understand both the evaluation of the experiments in 5.3. Why weren't the same DCI metrics compared here?\n\nLIMITATIONS:\n- As far as I can tell, no latents were uncovered (theoretically or experimentally) for either nonlinear mixing or unobserved intervention targets. Therefore, it seems to me, it should be mentioned that the results are for the linear case and weakly supervised. The title and abstract make the impression as if the more general CRL problem with paired observations is solved.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approach **ICRL-SM** that performs implicit causal representation learning (mapping from noise to latent variables) by using causal mechanism switch variable to model the soft intervention effects.\n\n&nbsp;\n\n### References\n[1] Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv:2301.08230, 2023.\n\n[2] Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning: Linear and general transformations. arXiv:2402.00849, 2024.\n\n[3] Julius von Kügelgen, Michel Besserve, Wendong Liang, Luigi Gresele, Armin Kekic ́, Elias Bareinboim, David M Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal rep- resentations from unknown interventions. In Proc. Advances in Neural Information Processing Systems, New Orleans, LA, December 2023.\n\n[4] J. Brehmer, P. De Haan, P. Lippe, and T. Cohen. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, volume 35, pages 38319– 38331, 2022.\n\nSTRENGTHS:\nThe idea of implicitly modeling causal effects using switch variables is very interesting. The experiments have shown promising performance on synthetic and high-dimensional image datasets compared to several baselines.\n\nWEAKNESSES:\n1. My most significant concern is that this paper is generally not well written and, thus, pretty hard to follow. For example, the calligraphic letter $\\mathcal{Z}$ was used throughout to denote both causal variables (e.g., line 137) and its domain (e.g., line 139).\n2. The assumption of a diffeomorphic causal mechanism is pretty strong. I am aware that [4] made a similar assumption. Yet, it is not a very common assumption for latent causal models in other causal representation learning literature (e.g., [1, 2, 3]).\n3. Line 278: The Gaussianity assumption of the causal and exogenous variables might be hard to satisfy in realistic settings.\n\nQUESTIONS:\n1. Line 159: (**Terminology**) As far as I know from the causality literature, a **soft intervention** is defined on a distributional level; from this perspective, it is not necessarily correct to assume the values are strictly different pre- and post-interventions for the intervened causal variables. Also, since intervention is performed on a distributional level, one does not observe such ``paired data`` $(x, \\tilde{x}, i)$. Instead, one observes the data population before and after intervention $p(x), p(\\tilde{x})$ without the same level correspondence (see [1, 2, 3]).\n2. Line 162: Is there also a formal mathematical definition for **\"sufficient variability\"**?\n3. Line 181: I would appreciate the authors properly defining the **\"switch variables\"** upon their first occurrence, especially because the work is based on these \"switch variables.\"\n4. Line 244: \"Latent Causal Model\" is neither defined nor cited.\n\nLIMITATIONS:\nThe main paper did not discuss limitations. The authors pointed them out in the checklist, but I would have appreciated it more if they had adequately addressed the limitations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach for learning implicit causal representations through switchable mechanisms, specifically designed to handle soft interventions which are more realistic but challenging compared to hard interventions. The authors introduce a causal mechanism switch variable to model the subtle effects of soft interventions and establish the identifiability of causal models under certain assumptions. Their proposed method, ICRL-SM, demonstrates improved performance in learning identifiable causal representations over baseline methods through experiments on synthetic and real-world datasets. The paper contributes a new perspective to causal representation learning with potential applications in various domains where understanding causal relationships from observational and interventional data is crucial.\n\nSTRENGTHS:\n1. A standout feature of this paper is the introduction of a switchable mechanism to model the nuanced effects of soft interventions. Traditional causal inference methods often focus on hard interventions, which are impractical in many real-world scenarios due to the need for stringent control. The paper's approach to incorporate soft interventions through a causal mechanism switch variable is groundbreaking. It allows the model to adapt to changes in causal relationships post-intervention, providing a more realistic and flexible framework for causal representation learning.\n2. This paper proposes Augmented Implicit Causal Models, which is an innovative concept that extends the scope of implicit causal representation learning. By integrating the causal mechanism switch variable into the model's solution functions, AICMs can capture the intrinsic characteristics of each causal variable while accounting for intervention effects. This approach sidesteps the need for explicit parameterization of the causal graph, which is a complex and often intractable task. The innovation here lies in the model's ability to implicitly learn the causal structure while directly modeling the effects of interventions.\n\nWEAKNESSES:\n1. The method's effectiveness relies on several key assumptions, including knowing the targets for intervention, interventions being atomic (indivisible), and variables following a multivariate normal distribution. These assumptions might not be realistic in many real-world settings where interventions can be complex, and data may not be normally distributed.\n2. While the paper's experiments on synthetic and specific real-world (Causal-Triplet) datasets demonstrate the method's potential, it's unclear how well it generalizes to other data types or different domains.\n\nQUESTIONS:\nAs the paper itself mentions, the identifiability of causal models can become more challenging with an increase in the number of variables and complexity of causal graphs. The paper could benefit from a deeper analysis of scalability, including how the method performs as the size and complexity of the causal system grow.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work is in causal representation learning that utilizes interventional data. It involves two types of common interventions: hard interventions and soft interventions. It’s known that soft intervention is more general since it covers hard intervention. But it is also more challenging since parental relations remain. This work proposed identifiability results using a causal mechanism switch variable designed to change between different causal mechanisms by component-wise transformations. Adequate experiments were provided to verify they claim.\n\nSTRENGTHS:\nPros:\n\n1. Overall, the writing is clear and with good context.\n2. Adequate experiments were provided, including real-data experiments. Also, they offered good observations of the comparison with baselines. This is a big plus to their contribution.\n\nWEAKNESSES:\nCons:\n\n1. It's unclear what the unique theoretical contribution is to this switch variable since both types of interventions were proposed before. Could the author clarify the theoretical contribution? Especially regarding the original technical hardness compared to [1] and also [38].\n\n2. In the synthetic experiment, it seems that you use the linear decoder, which loses the purpose of training a neural network to handle non-linear issues. Did you try non-linear functions and see how it goes? Besides the switch variable utilizing interventional data, what is the difference between these settings and those classical linear unmixing methods (like linear ICA or its other assumption variants)?\n\nSome minor issues/recommendations:\n\nLine 161, the $e_i$ is not defined, if you mean causal variable, then you already have defined it as $Z_i$.\n\nLine 187, the ‘\\s’ is not defined before but later in line 195.\n\nQUESTIONS:\nI put my questions in my previous section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}