SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $\mathcal{E}_n \subseteq \mathcal{E}$  be a subset of  $n$  pairs of environments which are assumed to correspond to distinct targets in the model  $q$ , and suppose for a contradiction that this is not actually the case for the ground truth  $p$  (i.e., there are duplicate and missing interventions w.r.t.  $p$ ). Then there must be two pairs of environments  $(e_a, e_a')$ ,  $(e_b, e_b') \in \mathcal{E}_n$ , both corresponding to interventions on some  $V_i$  in  $p$ , but which
are modelled as interventions on distinct nodes  $Z_{j}$  and  $Z_{k}$  with  $j \neq k$  in  $q$ . We show that this implies that  $V_{i}$  must simultaneously be a deterministic function of only  $Z_{j}$  and only  $Z_{k}$ . Similar to the proof of Thm. 3.2, we obtain the following equations,
$$
\frac {\tilde {\tilde {q}} _ {j}}{\tilde {q} _ {j}} \left(z _ {j}\right) = \frac {\tilde {\tilde {p}} _ {i}}{\tilde {p} _ {i}} \left(\psi_ {i} (z)\right), \tag {C.77}
$$
$$
\frac {\tilde {\hat {q}} _ {k}}{\tilde {q} _ {k}} \left(z _ {k}\right) = \frac {\hat {\hat {p}} _ {i}}{\hat {p} _ {i}} \left(\psi_ {i} (\boldsymbol {z})\right). \tag {C.78}
$$
By taking partial derivatives w.r.t.  $z_{l}$  and applying assumption (A3'), we find that
$$
\frac {\partial \psi_ {i}}{\partial z _ {l}} = 0 \quad \forall l \neq j, \tag {C.79}
$$
$$
\frac {\partial \psi_ {i}}{\partial z _ {l}} = 0 \quad \forall l \neq k. \tag {C.80}
$$
Since  $j \neq k$ , this implies that  $\partial \psi_{i} / \partial z_{l} = 0$  for all  $l$  which contradicts invertibility of  $\psi$ . Thus, by contradiction, we find that  $\mathcal{E}_n$  must contain exactly one pair of intervention per node also w.r.t.  $p$ . For the remainder of the proof, we only consider  $\mathcal{E}_n$ .
W.l.o.g., for any  $(e_i, e_i') \in \mathcal{E}_n$  we now fix the intervention targets in  $p$  to  $\mathcal{I}^{e_i} = \mathcal{I}^{e_i'} = \{i\}$  and let  $\pi$  be a permutation of  $[n]$  such that  $\pi(i)$  denotes the inferred intervention target in  $q$  that by (A2') is shared across  $(e_i, e_i')$ . (We will show later that not all permutations are admissible, but only ones that preserve the partial order of  $G$ .)
The first part of the proof is similar to Case 1 in the proof of Thm. 3.2. Consider the densities in environments  $e_i$  and  $e_i'$ , which are related through the change of variable formula by:
$$
\tilde {q} _ {\pi (i)} \left(z _ {\pi (i)}\right) \prod_ {j \in [ n ] \backslash \{\pi (i) \}} q _ {j} \left(z _ {j} \mid z _ {\mathrm {p a} (j; G ^ {\prime})}\right) = \tilde {p} _ {i} (\psi_ {i} (z)) \prod_ {j \in [ n ] \backslash \{i \}} p _ {j} \left(\psi_ {j} (z) \mid \psi_ {\mathrm {p a} (j)} (z)\right) | \det  J _ {\psi} (z) |, \tag {C.81}
$$
$$
\tilde {\tilde {q}} _ {\pi (i)} \left(z _ {\pi (i)}\right) \prod_ {j \in [ n ] \backslash \{\pi (i) \}} q _ {j} \left(z _ {j} \mid z _ {\mathrm {p a} (j; G ^ {\prime})}\right) = \tilde {\tilde {p}} _ {i} \left(\psi_ {i} (\boldsymbol {z})\right) \prod_ {j \in [ n ] \backslash \{i \}} p _ {j} \left(\psi_ {j} (\boldsymbol {z}) \mid \psi_ {\mathrm {p a} (j)} (\boldsymbol {z})\right) | \det  J _ {\psi} (\boldsymbol {z}) |, \tag {C.82}
$$
where  $Z_{\mathrm{pa}(j;G^{\prime})}\subseteq Z\setminus \{Z_j\}$  denotes the parents of  $Z_{j}$  in  $G^{\prime}$ .
Taking the quotient of the two equations yields
$$
\frac {\tilde {\tilde {q}} _ {\pi (i)}}{\tilde {\tilde {q}} _ {\pi (i)}} \left(z _ {\pi (i)}\right) = \frac {\tilde {\tilde {p}} _ {i}}{\tilde {\tilde {p}} _ {i}} \left(\psi_ {i} (\boldsymbol {z})\right). \tag {C.83}
$$
Next, for any  $j \neq \pi(i)$ , taking partial derivatives w.r.t.  $z_j$  on both sides yields
$$
0 = \left(\frac {\tilde {\bar {p}} _ {i}}{\tilde {p} _ {i}}\right) ^ {\prime} (\psi_ {i} (z)) \frac {\partial \psi_ {i}}{\partial z _ {j}} (z). \tag {C.84}
$$
By assumption (A3'), the first term on the RHS is non-zero everywhere. Hence, (C.84) implies
$$
\forall j \neq \pi (i), \forall z: \quad \frac {\partial \psi_ {i}}{\partial z _ {j}} (z) = 0 \tag {C.85}
$$
from which we can conclude that
$$
V _ {i} = \psi_ {i} (Z _ {\pi (i)}) \tag {C.86}
$$
for all  $i \in [n]$ . That is,  $\psi$  is the composition of the permutation  $\pi$  with an element-wise reparametrisation.
It remains to show that  $\pi$  must, in fact, be a graph isomorphism, which is equivalent to the statement
$$
V _ {i} \rightarrow V _ {j} \quad \text {i n} \quad G \quad \Longleftrightarrow \quad Z _ {\pi (i)} \rightarrow Z _ {\pi (j)} \quad \text {i n} \quad G ^ {\prime}. \tag {C.87}
$$
$(\Rightarrow)$  Suppose for a contradiction that there exist  $(i,j)$  such that  $V_{i}\rightarrow V_{j}$  in  $G$  , but  $Z_{\pi (i)}\nrightarrow Z_{\pi (j)}$  in  $G^{\prime}$
The main idea is to demonstrate that the lack of such direct arrow implies a certain conditional independence which, by faithfulness, would contradict the unconditional dependence of  $V_{i}$  and  $V_{j}$ .
Consider environment  $e_i$  in which there are perfect interventions on  $Z_{\pi(i)}$  and  $V_i$ , which has the effect of removing all incoming arrows to  $Z_{\pi(i)}$  and  $V_i$  in the respective post-intervention graphs  $G_{\overline{Z}_{\pi(i)}}'$  and  $G_{\overline{V}_i}$ .
As a result of this and the lack of direct arrow by assumption, any d-connecting path between  $Z_{\pi(i)}$  and  $Z_{\pi(j)}$  must enter the latter via  $Z_{\mathrm{pa}(\pi(j); G')}$  [95].
It then follows from Markovianity of  $q$  w.r.t.  $G^{\prime}$  that the following holds in  $Q_{\mathbf{Z}}^{e_i}$ :
$$
Z _ {\pi (i)} \perp \perp Z _ {\pi (j)} \mid Z _ {\mathrm {p a} (\pi (j); G ^ {\prime})}. \tag {C.88}
$$
We now consider the corresponding implication for  $P_V^{e_i}$ . Define
$$
\tilde {\boldsymbol {V}} = \left\{V _ {k} = \psi_ {k} \left(Z _ {\pi (k)}\right): Z _ {\pi (k)} \in \boldsymbol {Z} _ {\mathrm {p a} \left(\pi (j); G ^ {\prime}\right)} \right\} \subseteq \boldsymbol {V} \backslash \left\{V _ {i}, V _ {j} \right\}, \tag {C.89}
$$
and note that by assumption,  $Z_{\pi (i)} \notin Z_{\mathrm{pa}(\pi (j);G^{\prime})}$  and hence  $V_{i} \notin \tilde{V}$ .
By applying the corresponding diffeomorphic functions  $\psi_{i}$  from (C.86) to (C.88), it follows from Lemma C.2 that
$$
V _ {i} \perp V _ {j} | \hat {\boldsymbol {V}} \tag {C.90}
$$
in  $P_V^{e_i}$ . However, this violates faithfulness (Asm. 2.2) of  $P_V$  to  $G$  since  $V_{i}$  and  $V_{j}$  are d-connected in  $G_{\overline{V}_i}$ .
Thus, by contradiction, we must have  $Z_{\pi (i)}\to Z_{\pi (j)}$  in  $G^{\prime}$
$(\Longleftarrow)$  Now, suppose for a contradiction that there exist  $(i,j)$  such that  $Z_{\pi(i)} \to Z_{\pi(j)}$  in  $G'$ , but  $V_i \not\to V_j$  in  $G$ .
By the same argument as before, we find that
$$
V _ {i} \perp V _ {j} \mid \mathbf {V} _ {\mathrm {p a} (j)} \tag {C.91}
$$
in  $P_V^{e_i}$ , and thus by Lemma C.2
$$
Z _ {\pi (i)} \perp Z _ {\pi (j)} \mid \tilde {Z} \tag {C.92}
$$
in  $Q_{\mathbf{Z}}^{e_i}$  where
$$
\tilde {\boldsymbol {Z}} = \left\{Z _ {\pi (k)}: V _ {k} \in \boldsymbol {V} _ {\mathrm {p a} (j)} \right\} \subseteq \boldsymbol {Z} \backslash \left\{Z _ {\pi (i)}, Z _ {\pi (j)} \right\}.
$$
However, this contradicts faithfulness of  $Q_{\mathcal{Z}}$  to  $G'$ . Hence, we must have that  $V_{i} \to V_{j}$  in  $G$ .
This shows that  $\pi$  must be a graph isomorphism, thus concluding the proof.
C.4 Proof of Thm. 4.2
Proof. First, recall that according to Defn. 4.1,
$$
\mathfrak {C} _ {i \rightarrow j} ^ {P _ {V}} := D _ {\mathrm {K L}} \left(P _ {V} \| P _ {V} ^ {i \rightarrow j}\right), \tag {C.93}
$$
where  $P_{\mathbf{V}}^{i \rightarrow j}$  denotes the interventional distribution obtained by replacing  $p_j(v_j \mid \mathbf{v}_{\mathrm{pa}(j)})$  with
$$
p _ {j} ^ {i \rightarrow j} \left(v _ {j} \mid \mathbf {v} _ {\mathrm {p a} (j) \backslash \{i \}}\right) = \int_ {\mathcal {V} _ {i}} p _ {j} \left(v _ {j} \mid \mathbf {v} _ {\mathrm {p a} (j)}\right) p _ {i} \left(v _ {i}\right) \mathrm {d} v _ {i}. \tag {C.94}
$$
Writing out the KL divergence and noting that all terms except the interved mechanism  $j$  cancel inside the log, we obtain
$$
\mathfrak {C} _ {i \rightarrow j} ^ {P V} = \int_ {\mathcal {V}} \log \left(\frac {p _ {j} \left(v _ {j} \mid \boldsymbol {v} _ {\mathrm {p a} (j)}\right)}{\int_ {\mathcal {V} _ {i}} p _ {j} \left(v _ {j} \mid \boldsymbol {v} _ {\mathrm {p a} (j)}\right) p _ {i} \left(v _ {i}\right) \mathrm {d} v _ {i}}\right) p (\boldsymbol {v}) \mathrm {d} \boldsymbol {v}. \tag {C.95}
$$
$$
\mathfrak {C} _ {\pi (i) \rightarrow \pi (j)} ^ {Q _ {\mathcal {Z}}} = \int_ {\mathcal {Z}} \log \left(\frac {q _ {\pi (j)} \left(z _ {\pi (j)} \mid z _ {\mathrm {p a} (\pi (j) ; G ^ {\prime})}\right)}{\int_ {\mathcal {Z} _ {\pi (i)}} q _ {\pi (j)} \left(z _ {\pi (j)} \mid z _ {\mathrm {p a} (\pi (j) ; G ^ {\prime})}\right) q _ {\pi (i)} \left(z _ {\pi (i)}\right) \mathrm {d} z _ {\pi (i)}}\right) q (\boldsymbol {z}) \mathrm {d} \boldsymbol {z}. \tag {C.96}
$$
Since  $\mathbf{Z} = \mathbf{P}_{\pi^{-1}} \circ \phi(\mathbf{V})$ , we have  $V_{i} = \psi_{i}(Z_{\pi(i)})$  for all  $i \in [n]$  where  $\psi = \phi^{-1}$ .
Thus, by the change of variable formula, and using the fact that  $\pi(\mathrm{pa}(i)) = \mathrm{pa}(\pi(i); G')$  since  $\pi: G \mapsto G'$  is a graph isomorphism, we have for all  $i \in [n]$ :
$$
\left. q _ {\pi (i)} \left(z _ {\pi (i)} \mid \boldsymbol {z} _ {\mathrm {p a} (\pi (i); G ^ {\prime})}\right) = p _ {i} \left(\psi_ {i} \left(z _ {\pi (i)}\right) \mid \psi_ {\mathrm {p a} (i)} \left(\boldsymbol {z} _ {\mathrm {p a} (\pi (i); G ^ {\prime})}\right)\right) \left| \frac {\mathrm {d} \psi_ {i}}{\mathrm {d} z _ {\pi (i)}} \left(z _ {\pi (i)}\right) \right|, \right. \tag {C.97}
$$
as well as for the marginal density
$$
\left. q _ {\pi (i)} \left(z _ {\pi (i)}\right) = p _ {i} \left(\psi_ {i} \left(z _ {\pi (i)}\right)\right) \left| \frac {\mathrm {d} \psi_ {i}}{\mathrm {d} z _ {\pi (i)}} \left(z _ {\pi (i)}\right) \right|, \right. \tag {C.98}
$$
$$
q (\boldsymbol {z}) = p \left(\psi \circ \boldsymbol {P} _ {\pi} (\boldsymbol {z})\right) | \det  \boldsymbol {J} _ {\psi} (\boldsymbol {z}) |. \tag {C.99}
$$
Substitution into the expression for  $\mathfrak{C}_{\pi (i)\to \pi (j)}^{Qz}$  yields:
$$
\begin{array}{l} \mathfrak {C} _ {\pi (i) \rightarrow \pi (j)} ^ {Q z} = \int_ {\mathcal {Z}} \log \left(\frac {p _ {j} \left(\psi_ {j} \left(z _ {\pi (j)}\right) \mid \psi_ {\mathrm {p a} (j)} \left(\boldsymbol {z} _ {\mathrm {p a} (\pi (j) ; G ^ {\prime})}\right)\right)}{\int_ {\mathcal {Z} _ {\pi (i)}} p _ {j} \left(\psi_ {j} \left(z _ {\pi (j)}\right) \mid \psi_ {\mathrm {p a} (j)} \left(\boldsymbol {z} _ {\mathrm {p a} (\pi (j) ; G ^ {\prime})}\right)\right) p _ {i} \left(\psi_ {i} \left(z _ {\pi (i)}\right)\right)\left| \frac {\mathrm {d} \psi_ {i}}{\mathrm {d} z _ {\pi (i)}} \left(z _ {\pi (i)}\right)\right| \mathrm {d} z _ {\pi (i)}}\right) (C.100) \\ p (\psi \circ \boldsymbol {P} _ {\pi} (z)) | \det  \boldsymbol {J} _ {\psi} (z) | d z. (C.101) \\ = \int_ {\mathcal {V}} \log \left(\frac {p _ {j} \left(v _ {j} \mid \boldsymbol {v} _ {\mathrm {p a} (j)}\right)}{\int_ {\mathcal {V} _ {i}} p _ {j} \left(v _ {j} \mid \boldsymbol {v} _ {\mathrm {p a} (j)}\right) p _ {i} \left(v _ {i}\right) \mathrm {d} v _ {i}}\right) p (\boldsymbol {v}) \mathrm {d} \boldsymbol {v} (C.102) \\ = \mathfrak {C} _ {i \rightarrow j} ^ {P V}. (C.103) \\ \end{array}
$$
where the second to last line follows by integration by substitution, applied to both integrals.
In this appendix, we describe the experiments presented in § 6 in more details (Appx. D.1), and present additional results (Appx. D.2).
D.1 Experimental Details for  $\S 6$
Synthetic Data Generating Process. We consider linear Gaussian latent SCMs of the form
$$
V _ {1} := U _ {1}, \quad V _ {2} := \alpha V _ {1} + U _ {2}, \tag {D.1}
$$
with standard normal  $U_{1}$  and  $U_{2}$ . As a mixing function, we use a three-layer multilayer perceptron (MLP),
$$
f = \sigma \circ \boldsymbol {W} _ {3} \circ \sigma \circ \boldsymbol {W} _ {2} \circ \sigma \circ \boldsymbol {W} _ {1} \tag {D.2}
$$
where  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{W}_3 \in \mathbb{R}^{2 \times 2}$  are invertible weight matrices, and  $\sigma$  is an element-wise invertible nonlinear leaky-tanh activation function used in [41]:
$$
\sigma (x) = \tanh  (x) + 0. 1 x. \tag {D.3}
$$
To compute averages of our results over multiple runs, we construct different ground truth data generating processes as follows. We generate different latent SCMs by drawing  $\alpha$  uniformly from  $[-10, -2] \cup [2, 10]$ . (We exclude  $(-2, 2)$  to avoid sampling near unfaithful models.) We generate the corresponding mixing functions by uniformly sampling each element of the weight matrices,  $(\mathbf{W}_k)_{ij} \sim U(0, 1)$ . (To avoid the sampled weight matrices being too close to singular, we reject and resample if  $|\det \mathbf{W}_k| < 0.1$ .)
Interventional Environments. In line with Thm. 3.2, for each choice of latent SCM and mixing function, we generate three environments: one observational environment and one interventional environment for each perfect single-node intervention. For  $i = 1,2$ , we model a perfect intervention on  $V_{i}$  by removing the influence of the parent variables and changing the exogenous noise by shifting its mean up or down. Specifically, we replace the corresponding assignment in (D.1) by
$$
V _ {i} := \tilde {U} _ {i}, \quad \text {w h e r e} \quad \tilde {U} _ {i} \sim \mathcal {N} (m _ {i}, 1) \tag {D.4}
$$
where the mean  $m_{i}$  of the shifted Gaussian noise is fixed per environment and sampled uniformly from  $\{\pm 2\}$ .
We label the observational environment as  $e = 0$  and the environment arising from intervention on  $V_{i}$  by  $e = i$  for  $i = 1,2$ . Samples from  $p^e$  are then generated by sampling latents  $\mathbf{v}$  from the respective (un)intervened SCM and then applying the mixing function.
Model Architecture. We use normalizing flows [93] to model observations  $x$  as the result of an invertible, differentiable transformation  $g$  of some latent (noise) variable  $z$ ,
$$
\boldsymbol {x} = g (\boldsymbol {z}). \tag {D.5}
$$
We apply a series of  $L$  such transformations  $g^l:\mathbb{R}^2\to \mathbb{R}^2$  such that  $g = g^{L}\circ \ldots \circ g^{1}$  which we refer to as flow layers. We use Neural Spline Flows [30] for the invertible transformation, with a 3-layer feedforward neural network with hidden dimension 128 and permutation in each flow layer and  $L = 12$  layers. The transformations  $g,g^{1},\dots ,g^{L}$  have learnable parameters (the weights and biases of the neural networks), which we omit to simplify notation.
Typically, simple distributions such as a uniform or isotropic Gaussian are used as base distribution  $q(z)$  in normalizing flows. Here, we instead choose a base distribution that encodes information about the latent SCM. Specifically, we model the base mechanism as
$$
q _ {1} \left(z _ {1}\right) = \mathcal {N} \left(\mu_ {1}, \sigma_ {1} ^ {2}\right), \quad q _ {2} \left(z _ {2} \mid z _ {1}\right) = \mathcal {N} \left(\hat {\alpha} z _ {1}, \sigma_ {2} ^ {2}\right), \quad q _ {2} \left(z _ {2}\right) = \mathcal {N} \left(\mu_ {2}, \hat {\sigma} _ {2} ^ {2}\right) \tag {D.6}
$$
and the intervened mechanism as
$$
\tilde {q} _ {1} \left(z _ {1}\right) = \mathcal {N} \left(\tilde {\mu} _ {1}, \tilde {\sigma} _ {1} ^ {2}\right), \quad \tilde {q} _ {2} \left(z _ {2}\right) = \mathcal {N} \left(\tilde {\mu} _ {2}, \tilde {\sigma} _ {2} ^ {2}\right). \tag {D.7}
$$
Candidate Graphs and Intervention Targets. We train a separate normalizing-flow based model for each choice of candidate graph  $G'$  and inferred intervention targets. For the bivariate case with  $n = 2$ , this gives rise to four models, depending on whether  $G'$  matches  $G$  or not, and whether the intervention targets are aligned or misaligned w.r.t. the ground truth intervention targets. To model the setting  $G' \neq G$  in which  $Z_1$  and  $Z_2$  are assumed independent, we use  $q_2(z_2)$  in place of  $q_2(z_2 \mid z_1)$  in (D.6). If the intervention targets are aligned, we use  $\tilde{q}_i$  instead of  $q_i$  in  $e = i$  for  $i = 1, 2$ . Else, if they are misaligned, we use  $\tilde{q}_2$  instead of  $q_2$  in  $e = 1$  and  $\tilde{q}_1$  instead of  $q_1$  in  $e = 2$ . By multiplying the respective mechanisms, we thus obtain three environment-specific joint base distributions  $q^e(\boldsymbol{z})$  for  $e = 0, 1, 2$ .
Learning Objective. Given multi-environment data, the parameters  $\mu_1, \sigma_1, \hat{\alpha}, \sigma_2, \mu_2, \hat{\sigma}_2, \tilde{\mu}_2, \tilde{\sigma}_2$ ,  $\tilde{\mu}_1$  and  $\tilde{\sigma}_1$  are jointly learned with the parameters of the invertible transformations  $g^l$  by maximising the log-likelihood of the data under our model, which is given by:
$$
\sum_ {e \in \mathcal {E}} \mathbb {E} _ {\boldsymbol {x} \sim p ^ {e} (\boldsymbol {x})} \left[ \log p _ {\text {m o d e l}} ^ {e} (\boldsymbol {x}) \right] = \sum_ {e \in \mathcal {E}} \mathbb {E} _ {\boldsymbol {x} \sim p ^ {e} (\boldsymbol {x})} \left[ \log q ^ {e} (h (\boldsymbol {x})) + \log | \det  J _ {h} (\boldsymbol {x}) | \right] \tag {D.8}
$$
where the encoder  $h \coloneqq g^{-1}$  is the inverse of the normalizing flow which is readily available by construction; and where the expectations are empirical averages over the respective datasets in practice.
Training and Model Selection Details. Each environment comprises a total of 200k data points. We use the ADAM optimizer [67] with cosine annealing learning rate scheduling, starting with a learning rate of  $5 \times 10^{-3}$  and ending with  $1 \times 10^{-7}$ . We train the model for 200 epochs with a batch size of 4096. We split the dataset into  $70\%$  for training, and  $15\%$  for validation and held-out test data, each sampled randomly across all environments. For each drawn data generating process, we train three versions of each model with different random initializations and select the one with the highest validation log likelihood at the end of training for evaluation.
Evaluation Metrics. We evaluate the trained models w.r.t. mean correlation coefficient (MCC) on held-out data and log-likelihood on validation data (for model selection).
- The MCC measures the extent to which there is a one-to-one correspondence between the ground truth latents  $V_{i}$  and (a permuted version of) the inferred latents  $Z_{i} = h_{i}(\mathbf{X})$ . Its maximum value of one indicates a perfect correlation between the two. MCC is thus a proxy measure for the level of identifiability up to permutation and invertible reparametrisation. We report MCC based on Pearson (linear) correlation, though we found the results based on Spearman (nonlinear monotonic) correlation to be almost identical.
- The log-likelihood, on the other hand, measures how well a model explains or fits the data. Since the ground truth is typically unknown, a reasonable procedure when training multiple models is to select the one that attains the highest likelihood. For this reason, we report the difference in log-likelihood between misspecified models (ones assuming a wrong graph or intervention targets) to the correctly specified model. Whenever this difference is larger than zero, the correct model fits the data better and would thus be selected.
D.2 Additional Results: Learning Nonlinear Latent SCMs from Partial Causal Order
In this subsection, we present an additional experiment, in which we extend the setting investigated in § 6 and Appx. D.1 along the following axes.
- We fit generative models over three instead of two variables, corresponding to the setting of Thm. 3.4.
- The ground-truth SCM is now given by nonlinear mechanisms with non-additive, non-Gaussian noise.
- The generative model, including the learnt mechanisms, is now fully nonlinear.
- Despite Thm. 3.4 formally requiring two environments per single-node intervention, we only provide one interventional environment per node.
- Rather than searching over candidate graphs, we only fix the causal order and fit the reduced form of the SCM (see § 2.1) with a second normalizing flow.
Below, we describe these differences in more detail.
Figure 4: Comparison of Correctly and Incorrectly Specified Models for  $V_{2} \gets V_{1} \rightarrow V_{3}$  with Fixed Causal Order and Nonlinear SCM. Each violinplot corresponds to one setting where the intervention target labels are permuted. The blue plot  $(123^{*})$  is the setting with correct intervention target labels. The yellow plot  $(132^{(*)})$  has the targets for the two children  $V_{2}$  and  $V_{3}$  permuted, which also corresponds to a correct causal ordering and should thus be considered equivalent. We show mean correlation coefficients (MCCs) between the learned and ground truth latents (Left) and the difference in validation model log-likelihood between the well-specified (blue) and misspecified models (Right). Each violin plot is based on 20 different ground truth data generating processes; the horizontal lines indicate the minimum, median and maximum values.
Three-Variable Graph. The unknown ground truth graph is given by
$$
V _ {2} \leftarrow V _ {1} \rightarrow V _ {3}. \tag {D.9}
$$
This is consistent with the partial ordering  $V_{1} \preceq V_{2} \preceq V_{3}$ , which is assumed for all models a priori w.l.o.g., see § 2.2. Note that, due to the encoding of causal structure in the nonparametric model explained below, we only iterate over different permutations of the intervention targets and not over latent graph configurations. Due to the causal order implied by the graph (D.9), the permutations  $(1,2,3)$  (no permutation) and  $(1,3,2)$  (permutation of the two effects) are equivalent since the latter also implies the correct causal ordering.
Nonlinear, Non-Gaussian SCM. The mechanisms in the ground-truth SCM are now given by
$$
V _ {i} := \beta f _ {i} ^ {\text {l o c}} \left(\mathbf {V} _ {\mathrm {p a} (i)}\right) + f _ {i} ^ {\text {s c a l e}} \left(\mathbf {V} _ {\mathrm {p a} (i)}\right) U _ {i} \tag {D.10}
$$
for all  $i$ , where the location and scale functions  $f_{i}^{\mathrm{loc}}$ ,  $f_{i}^{\mathrm{scale}}: \mathbb{R}^{\lfloor \mathrm{pa}(i) \rfloor} \to \mathbb{R}$  are parameterized by random 3-layer neural networks (sampled as the random mixing function in (D.2)) and the noise variables are Gaussian,  $U_{i} \sim \mathcal{N}(0,1)$ . The factor  $\beta$  controls the influence of the parent variables relative to the exogenous noise. As  $\beta$  increases, variables tend to become more dependent, as also the mean shifts as a function of the parent variables. We set  $\beta = 10$  for the experiments shown in Fig. 4.
Nonparametric Latent SCM. We use a second normalizing flow to learn a reduced form of the latent SCM via the transformation  $g^{\mathrm{SCM}}: \mathbb{R}^3 \to \mathbb{R}^3$  mapping an exogenous noise variable  $\epsilon$  to the latent variable  $z$ ,
$$
z = g ^ {\mathrm {S C M}} (\epsilon). \tag {D.11}
$$
The distribution of the exogenous noise variable  $\epsilon$  as well as the distribution of the intervened mechanisms  $\tilde{q}_i(z_i)$  for  $i = 1,2,3$  is fixed and standard (isotropic) Gaussian. The flow layers in  $g^{\mathrm{SCM}}$  have an upper triangular Jacobian and thus allow us to encode assumptions about the causal graph: by passing the variables in topological order, which we can assume w.l.o.g., we ensure that an exogenous noise variable  $\epsilon_{i}$  can only influence endogenous variables in  $\mathbf{z}$  that are descendants of  $z_{i}$ . The learned weights of the flow layers then implicitly encode which endogenous variables are connected. Therefore, only different choices of the permutations of the intervention targets need to be considered as candidate models. We use a similar architecture based on Neural Spline Flows. However, we omit permutation layers, which would violate the topological order of the variables.