SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Overview of ICA identifiability results. For the trivial DAG over  $Z_{1}$ ,  $Z_{2}$ ,  $Z_{3}$  (i.e., no edges), we summarise our identifiability results (§ 4.2) for representations learnt by maximizing the likelihoods  $\mathbb{P}_{\theta}^{k}(X)$  based on multiple interventional regimes.  $\pi[\cdot]$  denotes a permutation.
Requirement of interventions	Learned representation z = f-1(x)	Reference
1 intervention on any two nodes respectively with Asm. 4.1	[h1(z1), h2(z2), h3(z3)]	Prop. 4.6
1 intervention on z1 and 2 fat-hand interventions on (z2, z3) with Asm. 4.4	[h1(z1), h2(z2, z3), h3(z2, z3)]	Corollary 4.8
1 intervention on z1 and 4 fat-hand interventions on (z2, z3) with “variability” assumption	[h1(z1), π[h2(z2), h3(z3)]]	Prop. 4.9
1 intervention per node on any two nodes respectively with unknown order (“matched intervention target”, see Defn. E.2) with Asm. E.4	π [h1(z1), h2(z2), h3(z3)]	Prop. E.6
6 fat-hand interventions on (z1, z2, z3) with “variability” assumption	π [h1(z1), h2(z2), h3(z3)]	[21][Thm. 1]
taking the Hessian, the left-hand side of (34) vanishes: this is exploited in the proof of [21, Thm.1]. All the proof techniques of nonlinear ICA papers we are aware of rely on this vanishing left-hand side over all coordinates of  $z$ . Unfortunately, with a non-trivial graph, it is impossible to get the same after taking the partial derivatives of every coordinate of  $z$ : in fact, for each  $i \in [d]$ , the left-hand side of the  $i$ -th equation depends on the ancestors of  $z_i$ . Our proofs for the case with a nontrivial graph (Thm. 4.2 and Thm. 4.5) therefore follow different strategies, based on first derivatives alone.
G Additional theoretical results used in the design of experiments
G.1 Multi-objective and pooled objective
Our identifiability theory implies that the ground-truth latent CBN could be learned by maximizing likelihood across all interventional regimes, i.e.,
$$
\theta^ {*} = \bigcap_ {k = 0} ^ {d} \arg \max  _ {\theta} \frac {1}{N _ {k}} \sum_ {n = 1} ^ {N _ {k}} \log p _ {\boldsymbol {\theta}} ^ {k} (\mathbf {x} ^ {(n, k)}),
$$
where  $\log p_{\pmb{\theta}}^{k}(\mathbf{x}^{(n,k)})$  is defined in (10). This is a multi-objective optimization problem.
In general, multi-objective optimization is hard; however, we show that, because of our assumptions, we can equivalently optimize a pooled objective. In fact, suppose
$$
f _ {k} (\boldsymbol {\theta}) = \log p _ {\boldsymbol {\theta}} ^ {k} (\mathbf {x}), \quad f (\boldsymbol {\theta}) = \sum_ {k = 0} ^ {d} f _ {k} (\boldsymbol {\theta}).
$$
Define  $\Theta_{k} := \arg \max_{\pmb{\theta}} f_{k}(\pmb{\theta})$ . By our problem setting we know  $\bigcap_{k=0}^{d} \Theta_{k} \neq \emptyset$ .
On the one hand, for all  $\hat{\pmb{\theta}}\in \bigcap_{k = 0}^{d}\Theta_{i},\hat{\pmb{\theta}}\in \arg \max_{\pmb{\theta}}f(\pmb {\theta})$  . On the other hand, we will prove by contradiction that  $\forall \hat{\pmb{\theta}}\in \arg \max_{\pmb{\theta}}f(\pmb {\theta}),\hat{\pmb{\theta}}\in \bigcap_{k = 0}^{d}\Theta_{k}$  . Suppose there exists  $i\in [0,d]$  such that  $\hat{\pmb{\theta}}\notin \Theta_i$  . Then for all  $\pmb{\theta}^{*}\in \bigcap_{k = 0}^{d}\Theta_{k}$ $f_{i}(\hat{\pmb{\theta}}) <   f_{i}(\pmb{\theta}^{*})$  . Thus  $f(\hat{\pmb{\theta}}) = \sum_{k = 0}^{d}f_{k}(\hat{\pmb{\theta}}) <   \sum_{k = 0}^{d}f_{k}(\pmb{\theta}^{*}) = f(\pmb{\theta}^{*})$  . This yields a contradiction with  $\hat{\pmb{\theta}}\in \arg \max_{\pmb{\theta}}f(\pmb {\theta})$
We thus conclude that  $\bigcap_{k=0}^{d} \arg \max_{\pmb{\theta}} f_k(\pmb{\theta}) = \arg \max_{\pmb{\theta}} f(\pmb{\theta})$ .
G.2 Expressivity in multi-intervention learning
To learn the latent CBN, we need to learn both the latent distributions (both the unintervened causal mechanisms and the intervened ones) and mixing functions. For the latent distributions, a natural
question is whether any of them could be fixed without loss of generality. This is for example possible in the context of nonlinear ICA, where due to the indeterminacy up to element-wise nonlinear scaling of the latent variables, we can arbitrarily fix their univariate distributions w.l.o.g. The following proposition elucidates this matter for CauCA.
Proposition G.1. For a ground truth latent CBN,  $\left(G,\mathbf{f},\left(\mathbb{P}^k,\tau_k\right)_{k\in \llbracket 0,d\rrbracket}\right)$  where  $\left(\mathbb{P}^k,\tau_k\right)_{k\in \llbracket 1,d\rrbracket}$  are obtained by perfect stochastic intervention on every variable respectively, fix at most one element for each  $k$  in  $\{\mathbb{Q}_k|pa(k) = \varnothing \} \cup \{\widetilde{\mathbb{Q}}_k|k\in [d]\}$ , there exists  $\left(G,\mathbf{f}',\left(\mathbb{Q}^k,\tau_k'\right)_{k\in \llbracket 0,d\rrbracket}\right)$  s.t.  $\mathbf{f}_{*}\mathbb{P}^{k} = \mathbf{f}_{*}'\mathbb{Q}^{k},\mathbf{f}'^{-1}\circ \mathbf{f}\in S_{scaling}$ .
In practice, this means that in order to learn  $\mathbf{f}'$  and  $(\mathbb{Q}^k)_{k\in [d]}$  with  $d$  perfect stochastic interventions, even if we fix all the intervention mechanisms  $(\widetilde{\mathbb{Q}}_k)_{k\in [d]}$ , we can still learn a true latent model up to scaling functions. Equivalently, we could also fix instead  $\{\mathbb{Q}_k|\mathrm{pa}(k) = \emptyset \} \cup \{\widetilde{\mathbb{Q}}_k|\mathrm{pa}(k)\neq \emptyset \}$ .
Proof. For any  $k \in [d]$ ,  $\mathbb{P}^k = \widetilde{\mathbb{P}}_k \prod_{j \neq k} \mathbb{P}_j$ . Without loss of generality by exchanging  $\mathbb{Q}_k$  with  $\widetilde{\mathbb{Q}}_k$ , suppose we are given  $\widetilde{\mathbb{Q}}_k$ , by Lemma B.2, there are two possible diffeomorphisms for  $T_k$  st.  $(T_k)_* \widetilde{\mathbb{P}_k} = \widetilde{\mathbb{Q}_k}$ . Choose one of them arbitrarily. Set  $\mathbf{T} := (T_k)_{k \in [d]}$ .
Define  $\mathbb{Q}^0 \coloneqq \mathbf{T}_* \mathbb{P}^0$ . By Lemma B.3,  $\forall i \in [d], \forall z_i \in \mathbb{R}, \forall \mathbf{z}_{\mathrm{pa}(i)} \in \mathbb{R}^{\# pa(i)}$ ,
$$
p _ {i} \left(z _ {i} \mid \mathbf {z} _ {\mathrm {p a} (i)}\right) = q _ {i} \left(T _ {i} \left(z _ {i}\right) \mid T _ {\mathrm {p a} (i)} \left(\mathbf {z} _ {\mathrm {p a} (i)}\right)\right) \left| T _ {i} ^ {\prime} \left(z _ {i}\right) \right|.
$$
Multiply the causal mechanisms of all  $i \in [d] \setminus \{k\}$  and the intervened mechanism of  $k$ , we get
$$
\widetilde {p} _ {k} (z _ {k}) \prod_ {i \neq k} p _ {i} \left(z _ {i} \mid \mathbf {z} _ {\mathrm {p a} (i)}\right) = \widetilde {q} _ {k} \left(T _ {k} \left(z _ {k}\right)\right) | T _ {k} ^ {\prime} (z _ {k}) | \prod_ {i \neq k} q _ {i} \left(T _ {i} \left(z _ {i}\right) \mid \mathbf {T} _ {\mathrm {p a} (i)} \left(\mathbf {z} _ {\mathrm {p a} (i)}\right)\right) | T _ {i} ^ {\prime} (z _ {i}) |,
$$
which is equivalent to  $\widetilde{p}^k (\mathbf{z}) = \widetilde{q}^k (T(\mathbf{z}))|\det \mathbf{T}(\mathbf{z})|$ . This is the change of variable formula of the diffeomorphism  $\mathbf{T}$  in  $\mathbb{R}^d$ , and implies  $\mathbb{Q}^k = \mathbf{T}_*\mathbb{P}^k$ .
Define  $\mathbf{f}' \coloneqq \mathbf{T} \circ \mathbf{f}^{-1}$ , then  $\mathbf{f}'^{-1} \circ \mathbf{f} = \mathbf{T} \in S_{\mathrm{scaling}}$ .
Remark If  $G$  is non trivial, given  $(G,\mathbf{f},\mathbb{P}^0)$ , in general there does not exist  $\mathbb{Q}^0\in P_G$ ,  $\mathbf{f}'\in \mathcal{F}$  s.t.  $\mathbf{f}_{*}\mathbb{P}^{0} = \mathbf{f}_{*}'\mathbb{Q}^{0}$ ,  $\mathbf{f}^{\prime -1}\circ \mathbf{f}\in S_{\mathrm{scaling}}$ .
To see why, consider the graph  $z_1 \to z_2$ . Fix  $\mathbb{Q}_1$ , there are only two possible diffeomorphisms for  $T_1$  s.t.  $\mathbb{Q}_1 = (T_1)_* \mathbb{P}_1$  by Lemma B.2.
If  $\mathbb{Q}_2\left(Z_2\mid T_1(z_1)\right)$  is fixed, to find  $T_{2}$  s.t.
$$
\left(T _ {2}\right) _ {*} \mathbb {P} _ {2} \left(Z _ {2} \mid z _ {1}\right) = \mathbb {Q} _ {2} \left(Z _ {2} \mid T _ {1} (z _ {1})\right) \quad \forall z _ {1} \in \mathbb {R},
$$
by Lemma B.2, the only possible  $T_{2}$  are  $G(\cdot \mid T_1(z_1))^{-1} \circ F(\cdot \mid z_1)$  and  $\bar{G}(\cdot \mid T_1(z_1))^{-1} \circ F(\cdot \mid z_1)$ . In general, these two functions depend on  $z_{1}$ . For example if  $\mathbb{P}_1(Z_1) \perp \mathbb{P}_2(Z_2)$ ,  $\mathbb{Q}_2(Z_2 \mid z_1) \sim \mathcal{N}(z_1, 1)$  then  $T_{2}$  depends on  $z_{1}$ . Namely, There is no  $\mathbf{T} \in S_{\mathrm{scaling}}$  s.t.  $\mathbb{Q}^0 = \mathbf{T}_* \mathbb{P}^0$ .
G.3 Normalizing flows for nonparametric CBN
In App. I, we learn a nonlinear mixing function and nonparametric causal mechanisms  $(p_i^\phi (z_i\mid$ $\mathbf{z}_{\mathrm{pa}(i)}))_{i\in [d]}$ . We model  $p^\phi$  by a normalizing flow  $\mathbf{h}^{\phi}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  from the space of  $\mathbf{Z}$  to the space of  $\mathbf{U}$ , while fixing  $\mathbb{P}_{\mathbf{u}}$ . The goal is to learn  $\mathbf{h}^{\phi -1}$ , the reduced form of an SCM which pushes forward the exogenous variables  $\mathbf{U}$  to every causal mechanism  $(p_i^\phi (z_i\mid \mathbf{z}_{\mathrm{pa}(i)}))_{i\in [d]}$ . ( $\mathbf{h}^{\phi}$  is denoted as  $\mathbf{h}_{\phi}^{\mathrm{CBN}}$  in App. I). In the following, we prove that such  $\mathbf{h}^{\phi}$  exists in a special case.
Proposition G.2. For any CBN  $\{\mathbb{P}_i^\phi (Z_i|\mathbf{Z}_{pa(i)})\}_{i\in [d]}$  Markov and faithful to a fully connected DAG  $G$  with  $\mathbb{P}^{\phi}$  absolutely continuous and smooth in  $\mathbb{R}^d$ , and for any independent component joint distribution  $\mathbb{P}_{\mathbf{u}}$  absolutely continuous and smooth in  $\mathbb{R}^d$ , there exists a normalizing flow  $\mathbf{h}^{\phi}:\mathbb{R}^{d}\to \mathbb{R}^{d}$  such that
- there exist a permutation matrix  $\mathbf{P}$  and an autoregressive normalizing flow  $\tilde{\mathbf{h}}$  such that  $\mathbf{h}^{\phi} = \mathbf{P} \circ \tilde{\mathbf{h}}$  (autoregressive in the sense that  $D\tilde{\mathbf{h}}(\mathbf{z})$  is lower triangular for all  $\mathbf{z} \in \mathbb{R}^{d}$ );
$\mathbf{h}_*\phi \mathbb{P}^\phi = \mathbb{P}_{\mathbf{u}};$
for all  $i\in [d]$ $p_i^\phi (z_i|\mathbf{z}_{pa(i)}) = \left|\frac{\partial h_i^\phi}{\partial z_i} (\mathbf{z})\right|p_{\mathbf{u}_i}\left(h_i^\phi (\mathbf{z})\right).$
Proof. Without loss of generality by applying a permutation matrix  $\mathbf{P}$ , we can suppose that the index of  $G$  preserves the partial order induced by  $E(G)$ :  $i < j$  if  $(i,j) \in E(G)$ . In the following, we can thus replace  $\tilde{\mathbf{h}}$  by  $\mathbf{h}^{\phi}$ . Since  $\mathbb{P}^{\phi}$  is faithful to a fully connected graph,  $\mathbf{Z}_{\mathrm{pa}(i)} = \mathbf{Z}_{< i}$ .
By [40] [Sec 2.2], there exists a Darmois transformation  $\mathbf{F}:\mathbb{R}^d\to (0,1)^d$  s.t.  $\mathbf{F}_{*}\mathbb{P}^{\phi} = \mathrm{Unif}(0,1)^{d}$  and a Darmois transformation  $\mathbf{G}:\mathbb{R}^d\to (0,1)^d$  s.t.  $\mathbf{G}_*\mathbb{P}_{\mathbf{u}} = \mathrm{Unif}(0,1)^d$ . Define  $\mathbf{h}^{\phi}:= \mathbf{G}^{-1}\circ \mathbf{F}$ , then  $\mathbf{h}_*^\phi \mathbb{P}^\phi = \mathbb{P}_{\mathbf{u}}$ .
Notice that the Darmois transformations  $\mathbf{F}$  and  $\mathbf{G}$  are autoregressive normalizing flows, i.e.  $D\mathbf{F}(\mathbf{z})$  and  $D\mathbf{G}(\mathbf{z})$  are lower triangular for all  $\mathbf{z} \in \mathbb{R}^d$ . Since the autoregressive normalizing flows are closed under composition and inverse,  $D(\mathbf{h}^\phi)(\mathbf{z})$  is also lower triangular for all  $\mathbf{z} \in \mathbb{R}^d$ . By induction using Lemma B.4, it is direct to prove that for all  $i \in [d]$ ,  $\mathbf{h}_{[i]}^\phi$  is a diffeomorphism from the first  $i$  variables of  $Z$  to the first  $i$  variables of  $\mathbf{U}$ .
By the definition of  $\mathbf{h}^{\phi}$ ,  $\mathbb{P}_{\mathbf{u}} = \mathbf{h}_{*}^{\phi}\mathbb{P}^{\phi}$ . By the change of variable formula,  $p^{\phi}(\mathbf{z}) = \left|\operatorname*{det}D\mathbf{h}^{\phi}(\mathbf{z})\right|p_{\mathbf{u}}\left(\mathbf{h}^{\phi}(\mathbf{z})\right)$ .
We now prove by induction that for all  $i \in [d]$ ,  $p_i^\phi(z_i | \mathbf{z}_{\mathrm{pa}(i)}) = \left| \frac{\partial h_i^\phi}{\partial z_i}(\mathbf{z}) \right| p_{\mathbf{u}_i} \left( h_i^\phi(\mathbf{z}) \right)$ . For  $i = 1$ , by the assumption of indices in  $G$ ,  $z_1$  has no parent. Since  $\mathbf{h}_1^\phi$  is a diffeomorphism,  $p_1^\phi(z_1) = \left| \frac{\partial h_1^\phi}{\partial z_1}(z_1) \right| p_{\mathbf{u}_1} \left( h_1^\phi(z_1) \right)$ .
Suppose the property is true for  $i$ :
$$
p _ {i} ^ {\phi} \left(z _ {i} \mid \mathbf {z} _ {\mathrm {p a} (i)}\right) = \left| \frac {\partial h _ {i} ^ {\phi}}{\partial z _ {i}} (\mathbf {z}) \right| p _ {\mathbf {u} _ {i}} \left(h _ {i} ^ {\phi} (\mathbf {z})\right) \tag {46}
$$
Then for  $i + 1$ , since  $\left(\mathbf{h}^{\phi}\right)_{[i + 1]}$  is a diffeomorphism with lower triangular Jacobian, we have
$$
\prod_ {j = 1} ^ {i + 1} p _ {j} ^ {\phi} \left(z _ {j} \mid \mathbf {z} _ {\mathrm {p a} (j)}\right) = \left| \det  D \mathbf {h} _ {[ i + 1 ]} ^ {\phi} (\mathbf {z}) \right| p _ {\mathbf {u} _ {[ i + 1 ]}} \left(\mathbf {h} _ {[ i + 1 ]} ^ {\phi} (\mathbf {z})\right) = \prod_ {j = 1} ^ {i + 1} \left| \frac {\partial h _ {j} ^ {\phi}}{\partial z _ {j}} (\mathbf {z}) \right| p _ {\mathbf {u} _ {j}} \left(h _ {j} ^ {\phi} (\mathbf {z})\right) \tag {47}
$$
where the last equality is because  $D\mathbf{h}^{\phi}$  is lower triangular and  $\mathbb{P}_{\mathbf{u}}$  is independent components. By dividing eq. (47) by eq. (46), we obtain the result.
In the method above, we assume that the CBN is Markov and faithful to a fully connected graph. Devising a more general method for sparse graphs is left for future work.
We note that the concurrent work by [23] may also be used to learn nonparametric latent CBNs. One main difference is that in our normalizing flow, every coordinate of  $\mathbf{h}^{\phi -1}$  models separately a causal mechanism, which is not the case in [23]. In our objective function (10), we need to model and learn every causal mechanism individually.
H Details Experiments
H.1 Synthetic Data Generation
Directed Acyclic Graph (DAG). In order to generate data, we begin by sampling a random DAG  $G \sim \mathbb{Q}_G$ , where  $\mathbb{Q}_G$  is a distribution over DAGs. The edge density of the DAG is set to 0.5 in topological order, meaning that the edges in the DAG are constrained to follow the variable index order: an edge  $Z_i \rightarrow Z_j$  can only exist if  $j > i$ . To construct the DAG, we individually sample each
potential edge  $Z_{i} \rightarrow Z_{j}$  with  $j > i$  with a probability of 0.5, while all other edges are assigned a probability of 0. For experiments conducted in the CauCA setting with non-trivial graphs (i.e., not empty), we reject and redraw any sampled DAGs that do not contain any edges.
Causal Bayesian network (CBN). To sample data from a CBN, we start by drawing the parameters of a linear Gaussian Structural Causal Model (SCM) with additive noise:
$$
Z _ {i} := \sum_ {j \in \operatorname {p a} (i)} \alpha_ {i, j} Z _ {j} + \varepsilon_ {i}, \tag {48}
$$
where the linear parameters are drawn from a uniform distribution  $\alpha_{i,j} \sim \mathrm{Uniform}(-a,a)$  and the noise variable is Gaussian  $\varepsilon_i \sim \mathcal{N}(0,1)$ . The signal-to-noise ratio for the SCM, denoted as  $a / \mathrm{std}(\varepsilon_i) = a$ , describes the strength of the dependence of the causal variables relative to the exogenous noise. For most experiments, the signal-to-noise ratio is set to 1. In Fig. 4 (g), we explore values ranging from 1 to 10. To specify the latent CBNs, we can define the conditional distributions entailed by the SCMs defined as in eq. (48), see also App. H.2 and eq. (52).
Generating interventional datasets. For a given CBN, we generate  $d + 1$  related datasets: one unintervened (observational) dataset and  $d$  interventional datasets, where the CBN was modified by a perfect stochastic intervention on one variable (i.e., one dataset for each variable in the CBN). W.l.o.g. we assume that the  $k^{\mathrm{th}}$  variable was intervened on in dataset  $k$ ;  $k = 0$  denotes the observational dataset. The intervention is applied in the SCM by removing the influence of the parent variables and changing the exogenous noise by shifting its mean up or down. Hence, for dataset  $k$  we have
$$
Z _ {k} := \tilde {\varepsilon} _ {k}, \quad \text {w i t h} \quad \tilde {\varepsilon} _ {k} \sim \mathcal {N} (\mu , 1), \tag {49}
$$
where the mean of the noise  $\mu$  is uniformly sampled from  $\{\pm 2\}$  and fixed within each dataset. Each dataset comprises a total of 200,000 data points, resulting in  $(d + 1)\times 200,000$  data points in total for each CBN.
Mixing function. The mixing function takes the form of a multilayer perceptron  $\mathbf{f} = \sigma \circ \mathbf{A}_M \circ \dots \circ \sigma \circ \mathbf{A}_1$ , where  $\mathbf{A}_m \in \mathbb{R}^{d \times d}$  for  $m \in [1, M]$  denote invertible linear maps, and  $\sigma$  is an element-wise invertible nonlinear function. The elements of the linear maps are sampled independently  $(\mathbf{A}_m)_{i,j} \sim \mathrm{Uniform}(0, 1)$  for  $i, j \in [1, d]$ . A sampled matrix  $\mathbf{A}_m$  is rejected and re-drawn if  $|\det \mathbf{A}_m| < 0.1$  to rule out linear maps that are (close to) singular. The invertible element-wise nonlinearity is a leaky-tanh activation function:
$$
\sigma (x) = \tanh  (x) + 0. 1 x, \tag {50}
$$
as used in [12].
H.2 Model architecture
Normalizing flows. We use normalizing flows [40] to learn an encoder  $\mathbf{g}_{\theta}:\mathbb{R}^d\to \mathbb{R}^d$ . Normalizing flows model observations  $\mathbf{x}$  as the result of an invertible, differentiable transformation  $\mathbf{g}_{\theta}$  on some base variables  $\mathbf{z}$ ,
$$
\mathbf {x} = \mathbf {g} _ {\theta} (\mathbf {z}). \tag {51}
$$
We apply a series of  $L = 12$  such transformations  $\mathbf{g}_{\theta^l}^l:\mathbb{R}^d\to \mathbb{R}^d$ , which we refer to as flow layers, such that the resulting transformation is given by  $\mathbf{g}_{\theta} = \mathbf{g}_{\theta^L}^L\circ \dots \circ \mathbf{g}_{\theta^1}^1$ . We use Neural Spline Flows [10] for the invertible transformation, with a 3-layer feedforward neural network with hidden dimension 128 and a permutation in each flow layer.
Base distribution. We extend the typically used simple base distributions to encode information about the CBN. We have one distribution per dataset  $(\hat{p}_{\theta_{\mathrm{CBN}}^k}^k)_{k\in [0,d]}$  over the learned base noise variables  $\mathbf{z}$ . The conditional density of latent variable  $i$  in dataset  $k$  is given by
$$
\hat {p} _ {\boldsymbol {\theta} _ {\mathbf {C B N}} ^ {k}} ^ {k} \left(z _ {i} \mid \mathbf {z} _ {\mathrm {p a} (i)}\right) = \mathcal {N} \left(\sum_ {j \in \mathrm {p a} (i)} \hat {\alpha} _ {i, j} z _ {j}, \hat {\sigma} _ {i}\right), \tag {52}
$$
when  $i \neq k$ , i.e., when variable  $i$  is not intervened on. For  $i = k$ , we have
$$
\hat {p} _ {\boldsymbol {\theta} _ {\mathrm {C B N}} ^ {k}} ^ {k} \left(z _ {i}\right) = \mathcal {N} \left(\hat {\mu} _ {i} ^ {k}, \hat {\sigma} _ {i} ^ {k}\right). \tag {53}
$$
In summary, the base distribution parameters are the parameters of the linear relationships between parents and children in the CBN  $(\hat{\alpha}_{i,j})_{i,j\in [[1,d]]}$ , the standard deviations for each variable in the observational setting  $(\hat{\sigma}_i)_{i\in [[1,d]]}$ , and mean and standard deviation for the intervened variable in each dataset  $(\hat{\mu}_i^k,\hat{\sigma}_i^k)_{i,k\in [[1,d]]}$ .
Linear baseline model. For the linear baseline models shown in Fig. 4 (a, b, e, f) we replace the nonlinear transformations  $(\mathbf{g}_{\boldsymbol{\theta}^l})_{l\in [1,L]}$  by a single linear transformation. The base distribution stays the same.
Graph-misspecified model. In order to test the impact of providing knowledge about the causal structure, we compare the CauCA model to one that assumes the latents are independent. This is achieved by setting  $\hat{\alpha}_{i,j} = 0\forall i,j\in [[1,d]]$  in the base distribution (52).
H.3 Training and model selection
Training parameters. We use the ADAM optimizer [27] with cosine annealing learning rate scheduling, starting with a learning rate of  $5 \times 10^{-3}$  and ending with  $1 \times 10^{-7}$ . We train the model for 50-200 epochs with a batch size of 4096. The number of epochs was tuned manually for each type of experiment to ensure reliable convergence of the validation log probability.
Pooled objective. The learning objective described in § 5 is using the pooled rather than the multi-objective formulation. In App. G, we prove that for our problem the two are equivalent.
Fixing CBN parameters. As explained in Prop. G.1, we can fix some of the CBN parameters w.l.o.g. In our case, we fix the noise parameters for intervened mechanisms of non-root variables and observational mechanisms of root variables.
Model selection. For each drawn latent CBN, we train three models with different initializations and select the model with the highest validation log probability at the end of training.
Compute. Each training run takes 2-8 hours on NVIDIA RTX-6000 gpus. For the experiments shown in the main paper, we performed 450 training runs (30 per violin plot / point in Fig. 4 (g)) which sums up to around 2250 compute hours (assuming an average run time of 5 hours).
I Nonparametric Experiments
The main experiments in § 5 were made under the restriction that the ground-truth CBN and the learned latent CBN were assumed to be linear Gaussian. In the experiments shown here, we relax those restrictions. In the following, we describe the differences with respect to the main experiments. All other settings and parameters are the same as described in § 5 and App. H.
I.1 Synthetic Data Generation
Causal Bayesian network (CBN). To sample data from a CBN, we draw the parameters of a nonlinear non-additive-noise Structural Causal Model (SCM):
$$
Z _ {i} := h ^ {\text {l o c}} \left(Z _ {\mathrm {p a}} (i)\right) + h ^ {\text {s c a l e}} \left(Z _ {\mathrm {p a}} (i)\right) \varepsilon_ {i}, \tag {54}
$$
where the location and scale functions  $h^{\mathrm{loc}}$ ,  $h^{\mathrm{scale}}: \mathbb{R}^{\# \mathrm{pa}(i)} \to \mathbb{R}$  are parameterized by random 3-layer neural networks (similar to the random mixing function) and the noise variable is Gaussian  $\varepsilon_{i} \sim \mathcal{N}(0,1)$ .
I.2 Model architecture
Base distribution. We extend the typically used simple base distributions to encode information about the CBN. We train a second normalizing flow to learn a mapping  $\mathbf{h}_{\phi}^{\mathrm{CBN}}: \mathbb{R}^{d} \to \mathbb{R}^{d}$  from the latent causal variables  $\mathbf{z}$  to exogenous noise variables  $\mathbf{u}$ , which we explain in App. G.3. We encode knowledge about the causal graph in the base distribution by passing the latent variables in causal order to the invertible transformation  $\mathbf{h}^{\mathrm{CBN}}$ , whose Jacobian is lower triangular. This ensures the correct causal relationships among the elements of  $\mathbf{z}$ . During training, we fix the distribution of  $\mathbf{u}$  and the intervened upon latent variables, i.e.  $z_{k}$  for dataset  $k$ , to be standard normal. We use a similar architecture based on Neural Spline Flows, with one difference: we omit the permutation layer, which would violate the topological order of the variables.
Figure 6: Experimental results with nonparametric model. The figure shows the mean correlation coefficients (MCC) between true and learned latents for Causal Component Analysis (CauCA) experiments. The first two violin plots show the fully nonparametric model when the ground truth latent CBN is generated by a location-scale model and for the case with a linear SCM generating the ground truth CBN. Misspecified models assuming a linear encoder function class and a naive (single-environment) unidentifiable normalizing flow with an independent Gaussian base distribution (labelled i.i.d.) are compared. The misspecified models are trained on a location-scale CBN. All violin plots show the distribution of outcomes for 10 pairs of CBNs and mixing functions.
I.3 Results