SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This reasoning can only be extended to the larger class  $\mathcal{F}$  if the ground truth mixing  $f$  (for the relevant applications) can be better approximated (i.e., with smaller norm) by functions in  $\mathcal{G}$  than all spurious solutions  $\widetilde{f}$ . This is hard to verify in practice and difficult to formalize theoretically. Nevertheless, this motivates to look for identifiability results for function classes that are known to be useful representation learners and used in practice, in particular neural nets. We emphasize that alternatively one can also directly consider norms on the larger space  $\mathcal{F}$  penalizing, e.g., derivative norms, to perform model selection.
F Additional details on experimental methodology
In this appendix, we give more details about our experimental methodology. First, we derive the log-odds and prove Lemma 1 in Appendix F.1. Then, we use it to design our model and theoretically justify our contrastive approach in Appendix F.2, by connecting it to the Bayes optimal classifier. Then, we describe the NOTEARS regularizer in Appendix F.3, describe the limitations of the contrastive approach in Appendix G.1 and finally, in Appendix F.4, we describe the ingredients needed for an approach via Variational Autoencoders, the difficulties involved and how to potentially bypass them.
F.1 Proof of Lemma 1
$$
\ln p _ {X} ^ {(i)} (x) - \ln p _ {X} ^ {(0)} (x) = c _ {i} - \frac {1}{2} \lambda_ {i} ^ {2} \left(\left(f ^ {- 1} (x)\right) _ {t _ {i}}\right) ^ {2} + \eta^ {(i)} \lambda_ {i} \cdot \left(f ^ {- 1} (x)\right) _ {t _ {i}}) + \frac {1}{2} \langle f ^ {- 1} (x), s ^ {(i)} \rangle^ {2} \tag {6}
$$
Proof. We will write the log likelihood of  $X^{(i)} = f(Z^{(i)})$  using standard change of variables. As we elaborate in Appendix A.1, we have
$$
Z ^ {(i)} = \left(B ^ {(i)}\right) ^ {- 1} \epsilon + \mu^ {(i)} \text {w i t h} B ^ {(i)} = \left(D ^ {(i)}\right) ^ {- 1 / 2} \left(I - A ^ {(i)}\right), \mu^ {(i)} = \eta^ {(i)} \left(B ^ {(i)}\right) ^ {- 1} e _ {t _ {i}} \tag {114}
$$
Also, denote by  $\Theta^{(i)} = (B^{(i)})^\top B^{(i)}$  the precision matrix of  $Z^{(i)}$  (see Appendix A.1 for full derivation). By change of variables,
$$
\ln p _ {X} ^ {(i)} (x) = \ln | J _ {f ^ {- 1}} | + \ln p _ {Z} ^ {(i)} (f ^ {- 1} (x))
$$
$$
= \ln | J _ {f ^ {- 1}} | - \frac {n}{2} \ln (2 \pi) - \frac {1}{2} \ln | \Sigma^ {(i)} | - \frac {1}{2} (f ^ {- 1} (x) - \mu^ {(i)}) ^ {T} \Theta^ {(i)} (f ^ {- 1} (x) - \mu^ {(i)})
$$
where  $J_{f^{-1}}$  denotes the Jacobian matrix of partial derivatives. Let  $s^{(1)}, s^{(2)}, \ldots, s^{(d)}$  denote the rows of  $B^{(0)}$ . We then compute the log-odds with respect to  $X^{(0)}$ , the base distribution without interventions (and where  $\mu^{(0)} = 0$ ) to get
$$
\begin{array}{l} \ln p _ {X} ^ {(i)} (x) - \ln p _ {X} ^ {(0)} (x) \\ = \left(- \frac {1}{2} \ln \frac {\left| \Sigma^ {(i)} \right|}{\left| \Sigma^ {(0)} \right|} - \frac {1}{2} \left(\mu^ {(i)}\right) ^ {\top} \Theta^ {(i)} \mu^ {(i)}\right) - \frac {1}{2} \left(f ^ {- 1} (x)\right) ^ {\top} \left(\Theta^ {(i)} - \Theta^ {(0)}\right) \left(f ^ {- 1} (x)\right) + \left(f ^ {- 1} (x)\right) ^ {\top} \Theta^ {(i)} \mu^ {(i)} \\ = c _ {i} - \frac {1}{2} (f ^ {- 1} (x)) ^ {\top} (r ^ {(i)} \otimes r ^ {(i)} - s ^ {(i)} \otimes s ^ {(i)}) (f ^ {- 1} (x)) + \eta^ {(i)} \cdot (f ^ {- 1} (x)) ^ {\top} (B ^ {(i)}) ^ {\top} e _ {t _ {i}} \\ = c _ {i} - \frac {1}{2} (f ^ {- 1} (x)) ^ {\top} (\lambda_ {i} ^ {2} e _ {t _ {i}} e _ {t _ {i}} ^ {\top} - (s ^ {(i)}) (s ^ {(i)}) ^ {\top}) (f ^ {- 1} (x)) + \eta^ {(i)} \lambda_ {i} \cdot (f ^ {- 1} (x)) ^ {\top} e _ {t _ {i}} \\ = c _ {i} - \frac {1}{2} (f ^ {- 1} (x)) ^ {\top} (\lambda_ {i} ^ {2} e _ {t _ {i}} e _ {t _ {i}} ^ {\top} - (s ^ {(i)}) (s ^ {(i)}) ^ {\top}) (f ^ {- 1} (x)) + \eta^ {(i)} \lambda_ {i} \cdot (f ^ {- 1} (x)) _ {t _ {i}} \\ = c _ {i} - \frac {1}{2} \lambda_ {i} ^ {2} \left(\left(\left(f ^ {- 1} (x)\right) _ {t _ {i}}\right) ^ {2} + \eta^ {(i)} \lambda_ {i} \cdot \left(f ^ {- 1} (x)\right) _ {t _ {i}}\right) + \frac {1}{2} \langle f ^ {- 1} (x), s ^ {(i)} \rangle^ {2} \\ \end{array}
$$
for a constant  $c_{i}$  independent of  $z$ . For the second equality, we used Lemma 3.
F.2 A theoretical justification for the log-odds model
In this section, we provide more details on our precise modeling choices for the contrastive approach, and show theoretically why it encourages the model to learn the true underlying parameters.
Recall that, motivated by Lemma 1, we model the log-odds as
$$
g _ {i} \left(x, \alpha_ {i}, \beta_ {i}, \gamma_ {i}, w ^ {(i)}, \theta\right) = \alpha_ {i} - \beta_ {i} h _ {t _ {i}} ^ {2} (x, \theta) + \gamma_ {i} h _ {t _ {i}} (x, \theta) + \langle h (x, \theta), w ^ {(i)} \rangle^ {2} \tag {115}
$$
where  $h(\cdot, \theta)$  denotes a neural net parametrized by  $\theta$ , parameters  $w^{(i)}$  are the rows of a matrix  $W$  and  $\alpha_{i}, \beta_{i}, \gamma_{i}$  are learnable parameters. Moreover, we have  $g_{0}(x) = 0$  as we compute the log-odds with respect to  $X^{(0)}$ . Therefore, the cross entropy loss given by
$$
\mathcal {L} _ {\mathrm {C E}} ^ {(i)} = - \mathbb {E} _ {j \sim \mathcal {U} (\{0, i \})} \mathbb {E} _ {x \sim X ^ {(j)}} \ln \left(\frac {e ^ {\mathbf {1} _ {j = i} g _ {i} (x)}}{e ^ {g _ {i} (x)} + 1}\right). \tag {116}
$$
Let  $C(x) \in \{0,1\}$  denote the label of the datapoint  $x$  indicating whether it was an observational datapoint or an interventional datapoint respectively. By using the cross entropy loss, we are treating the model outputs as logits, therefore we can write down the posterior probability distribution using a logistic regression model as
$$
\Pr [ C (x) = 1 | x ] = \frac {\exp \left(g _ {i} \left(x , \alpha_ {i} , \beta_ {i} , \gamma_ {i} , w ^ {(i)} , \theta\right)\right)}{1 + \exp \left(g _ {i} \left(x , \alpha_ {i} , \beta_ {i} , \gamma_ {i} , w ^ {(i)} , \theta\right)\right)} \tag {117}
$$
Moreover, by using a sufficiently wide or deep neural network with universal approximation capacity and sufficiently many samples, our model will learn the Bayes optimal classifier, as given by Lemma 1. Therefore, we can equate the probabilities to arrive at
$$
\frac {\exp \left(g _ {i} \left(x , \alpha_ {i} , \beta_ {i} , \gamma_ {i} , w ^ {(i)} , \theta\right)\right)}{1 + \exp \left(g _ {i} \left(x , \alpha_ {i} , \beta_ {i} , \gamma_ {i} , w ^ {(i)} , \theta\right)\right)} = \frac {\exp \left(\ln p _ {X} ^ {(i)} (x) - \ln p _ {X} ^ {(0)} (x)\right)}{1 + \exp \left(\ln p _ {X} ^ {(i)} (x) - \ln p _ {X} ^ {(0)} (x)\right)} \tag {118}
$$
$$
\begin{array}{l} \alpha_ {i} - \beta_ {i} h _ {t _ {i}} ^ {2} (x, \theta) + \gamma_ {i} h _ {t _ {i}} (x, \theta) + \langle h (x, \theta), w ^ {(i)} \rangle^ {2} (119) \\ = c _ {i} - \frac {1}{2} \lambda_ {i} ^ {2} \left(\left(f ^ {- 1} (x)\right) _ {t _ {i}}\right) ^ {2} + \eta^ {(i)} \lambda_ {i} \cdot \left(f ^ {- 1} (x)\right) _ {t _ {i}}) + \frac {1}{2} \langle f ^ {- 1} (x), s ^ {(i)} \rangle^ {2} (120) \\ \end{array}
$$
This suggests that in optimal settings, our model will learn (up to scaling)
$$
\alpha_ {i} = c _ {i}, \quad \beta_ {i} = \frac {1}{2} \lambda_ {i} ^ {2}, \quad \gamma_ {i} = \eta^ {(i)} \lambda_ {i}, \quad w ^ {(i)} = s ^ {(i)}, \quad h (x, \theta) = f ^ {- 1} (x) \tag {121}
$$
thereby inverting the nonlinearity and learning the underlying parameters.
F.3 NOTEARS [102]
In our algorithm, using the parameters  $w^{(i)}$  as rows, we learn the matrix  $W$ , which we saw in optimal settings will be  $B = D^{-1/2}(\mathrm{Id} - A)$ . Note that the off-diagonal entries of  $B$  form a DAG. Therefore, the graph encoded by  $W_0$ , which is defined to be  $W$  with the main diagonal zeroed out, must also be a DAG.
However, in our algorithm, we don't explicitly enforce this acyclicity. There are two ways to get around this. One way is to assume a default causal ordering on the  $Z_{i}$ s, which is feasible as we don't directly observe  $Z$ . Then, we can simply enforce  $W_{0}$  to be triangular and train via standard gradient descent and  $W_{0}$  is guaranteed to be a DAG. However, this approach doesn't work because the interventional datasets are given in arbitrary order and so we are not able to match the datasets to the vertices in the correct order. So this merely defers the issue.
Instead, the other approach that we take in this work is to regularize the learnt  $W_{0}$  to model a directed acyclic graph. Learning an underlying causal graph from data is a decades old problem that has been widely studied in the causal inference literature, see e.g. [12, 102, 64, 80, 4] and references therein. In particular, the work [102] proposed an analytic expression to measure the DAGness of the causal graph, thereby making the problem continuous and more efficient to optimize over.
Lemma 12 ([102]). A matrix  $W \in \mathbb{R}^{d \times d}$  is a DAG if and only if
$$
\mathcal {R} := \operatorname {t r} \exp (W \circ W) - d = 0 \tag {122}
$$
where  $\circ$  is the matrix Hadamard product and  $\exp$  is the matrix exponential. Moreover,  $\mathcal{R}$  has gradient
$$
\nabla \mathcal {R} = \exp (W \circ W) ^ {\top} \circ 2 W \tag {123}
$$
Therefore, we add  $\mathcal{R}_{\text{NOTEARS}}(W) = \operatorname{tr} \exp(W_0 \circ W_0) - d$  as a regularization to our loss function. As in prior works, we could also consider the augmented Lagrangian formulation however it leads to additional training complexity. There have been several follow-up works to NOTEARS such as [98] (see [4] for an overview) that suggest different regularization schemes. We leave exploring such alternatives to future work.
F.4 A Variational Autoencoder approach
In this section, we describe the technical details involved with adapting Variational Autoencoders (VAEs) [39, 68] for our setting. We highlight some difficulties that we will encounter and also suggest possible ways to work around them.
Indeed, most earlier approaches for causal representation learning have relied on maximum likelihood estimation via variational inference. In particular, they have relied on autoencoders or variational autoencoders [36, 2]. In our setting, VAEs are a viable approach. More concretely, we can encode the distribution  $X^{(0)}$  to  $\epsilon$ , then apply the  $B^{(i)}$  parameter matrix before finally decoding to  $X^{(i)}$ . To handle the fact that we don't have paired interventional data as in [6], we could potentially modify the Evidence Lower Bound (ELBO) to include some divergence measure between the predicted and measured interventional data. Finally, this can be trained end-to-end as in traditional VAEs. We now describe this in more detail.
We will use VAEs to encode the observational distribution  $X^{(0)}$  to  $\epsilon$ , so the encoder will formally model  $B^{(0)} \circ (f^{-1}(.) - \mu^{(i)})$  where we use the notation from Appendix A.1. Note here that we cannot directly design the encoder to map  $X$  to  $Z$  because we do not know the prior distribution on  $Z$ . Indeed, the objective is to learn it.
Let  $I$  denote the set of intervention targets. Assume the targets are chosen uniformly at random, which can be done in practice by subsampling each interventional distribution to have the same size. Suppose we had paired counterfactual data  $\mathcal{D} = \cup_{i\in I}\mathcal{D}_i$  of paired interventional datasets, i.e.  $\mathcal{D}_i$  consists of pairs  $(X^{(0)},X^{(i)})$  with corresponding probability density denoted  $p^{(i)}(x^{(0)},x^{(i)})$ . Following [39, 101], define an amortized inference distribution as  $q(\epsilon |x^{(0)})$ . Then, we can bound the expected log-likelihood as
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} \left[ \ln p ^ {(i)} \left(x ^ {(0)}, x ^ {(i)}\right) \right] = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \left[ \ln p ^ {(i)} \left(x ^ {(0)}, x ^ {(i)}\right) \right] (124) \\ = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \ln \int_ {\epsilon} p ^ {(i)} \left(x ^ {(0)}, x ^ {(i)}, \epsilon\right) d \epsilon (125) \\ \end{array}
$$
$$
\begin{array}{l} = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \ln \int_ {\epsilon} \frac {p ^ {(i)} \left(x ^ {(0)} , x ^ {(i)} , \epsilon\right) q \left(\epsilon \mid x ^ {(0)}\right)}{q \left(\epsilon \mid x ^ {(0)}\right)} d \epsilon (126) \\ = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \ln \mathbb {E} _ {\epsilon \sim q (\epsilon | x ^ {(0)})} \frac {p ^ {(i)} \left(x ^ {(0)} , x ^ {(i)} , \epsilon\right)}{q (\epsilon | x ^ {(0)})} (127) \\ \geq \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \mathbb {E} _ {\epsilon \sim q \left(x ^ {(0)}\right)} \ln \frac {p ^ {(i)} \left(x ^ {(0)} , x ^ {(i)} , \epsilon\right)}{q \left(\epsilon \mid x ^ {(0)}\right)} \quad (\text {J e s s e n ' s i n e q u a l i t y}) (128) \\ = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \mathbb {E} _ {\epsilon \sim q \left(x ^ {(0)}\right)} \ln \frac {p ^ {(i)} \left(x ^ {(0)} , x ^ {(i)} \mid \epsilon\right) p (\epsilon)}{q \left(\epsilon \mid x ^ {(0)}\right)} (129) \\ = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\mathcal {D} _ {i}} \left(\mathbb {E} _ {\epsilon \sim q \left(x ^ {(0)}\right)} \ln p ^ {(i)} \left(x ^ {(0)}, x ^ {(i)} \mid \epsilon\right) - \mathbb {E} _ {\epsilon \sim q \left(x ^ {(0)}\right)} \ln \frac {q \left(\epsilon \mid x ^ {(0)}\right)}{p (\epsilon)}\right) (130) \\ = \mathbb {E} _ {i \sim \mathcal {U} (I)} \mathbb {E} _ {\left(x ^ {(0)}, x ^ {(i)}\right) \sim \mathcal {D} _ {i}} \left[ \mathbb {E} _ {\epsilon \sim q \left(\epsilon \mid x ^ {(0)}\right)} \left[ \ln p ^ {(i)} \left(x ^ {(0)}, x ^ {(i)} \mid \epsilon\right) \right] \right. (131) \\ \left. - \operatorname {K L} \left(q \left(\epsilon \mid x ^ {(0)}\right) \mid \mid N (0, I)\right) \right] (132) \\ \end{array}
$$
where the last term is the standard Evidence Lower Bound (ELBO). This can then be trained end-to-end via the reparameterization trick. A similar approach was taken in [6] who had access to paired counterfactual data.
However, we do not have access to such paired interventional data, but instead we only observe the marginal distributions  $X^{(i)}$ . To this end, conditioned on  $\epsilon$ , we can split the term  $\ln p^{(i)}(x^{(0)}, x^{(i)}|\epsilon) = \ln \widetilde{p}^{(0)}(x^{(0)}|\epsilon) + \ln \widetilde{p}^{(i)}(x^{(i)}|\epsilon)$  where  $\widetilde{p}^{(i)}$  denotes the density of the intervened marginal of  $p^{(i)}$ , which corresponds to using the interventional decoder  $f \circ ((B^{(i)})^{-1}(.) + \mu^{(i)})$ . Nevertheless,  $\mathbb{E}_{\epsilon \sim q(x^{(0)})}[\ln \widetilde{p}^{(i)}(x^{(i)}|\epsilon)]$  is still not tractable in unpaired settings. As a heuristic, we could consider a modified ELBO by modifying this term to measure some sort of divergence metric between the true interventional data  $X^{(i)}$  and the generated interventional data  $\widetilde{p}^{(i)}(x^{(i)}|\epsilon)$ , similar to [101]. We leave it for future work to explore this direction.
In this section, we provide additional experimental results and discussions to explore the effect of shift strength of the interventions, data scale, and noise distribution.
G.1 Limitations of the contrastive approach
Figure 5: Dependence of the performance metrics on the shift  $\eta$ . Other settings are as for nonlinear mixing functions (see Table 7) with ER(10,2) graphs and  $d' = 100$ .
As mentioned in the main part of the paper, our contrastive algorithm struggles to recover the ground truth latent variables when considering interventions without shifts. We illustrate the dependence on the shift strength in Figure 5. In this section, we provide some evidence why this is the case. Apart
Figure 6: Cross entropy loss for  $a = 1$ ,  $c = 0$ ,  $z_0 = 1$  and  $b = 0$  (left),  $b = 2$  (right) as a function of the estimated latent variable  $\hat{z}$ .
from setting the stage for future works, this also enables practitioners to be aware of potential pitfalls when applying our techniques.
Figure 7: Density of standardized ground truth latents and recovered latents for  $\eta = 0$  (top left) and  $\eta = 1$  (top right), scatter plot of the ground truth latent variables  $Z_{1}$  and recovered latent variables  $\hat{Z}_{1}$  for  $\eta = 0$  (bottom left) and  $\eta = 1$  (bottom right). Results shown for ER(10,2) graphs and  $d' = 100$ , all further parameters as in Table 7.
We suspect that the main reason for this behavior is the non-convexity of the parametric output layer. Note that this is very different from the well known non-convexity of (overparametrized) neural networks. While neural networks parametrize non-convex functions their final layer is typically a convex optimization problem, e.g., a least squares regression or a logistic regression on the features.
Moreover, it is well understood that convergence to a global minimizer using gradient descent is possible under suitable conditions, see, e.g., [47].
This is different for our quadratic log-odds expression where gradient descent is not sufficient to find the global minimizer. To illustrate this we consider the case of a single latent, i.e.,  $d = 1$ , then the ground truth parametric form of the log-odds in 6 can be expressed as
$$
\ln p _ {X} ^ {(1)} - \ln p _ {X} ^ {(0)} (x) = g (x) = h \left(f ^ {- 1} (x)\right) = a \left(f ^ {- 1} (x) - b\right) ^ {2} + c \tag {133}
$$
for some constants  $a, b, c$  and  $b = 0$  if  $\eta^{(1)} = 0$ , i.e., there is no shift. We moreover assume, for the sake of argument, that the final layer implementing  $h(z) = a(z - b)^2 + c$  is fixed to the ground truth parametric form of the log-odds (then there is also no scaling ambiguity left). Now, let us fix a point  $x_0$  and define  $z_0 = f^{-1}(x_0)$  and
$$
p = \mathbb {P} (i = 1 | X = x _ {0}) = \frac {p _ {X} ^ {(1)} \left(x _ {0}\right)}{p _ {X} ^ {(1)} \left(x _ {0}\right) + p _ {X} ^ {(0)} \left(x _ {0}\right)} = \frac {e ^ {g \left(x _ {0}\right)}}{1 + e ^ {g \left(x _ {0}\right)}}. \tag {134}
$$
Then the (sample conditional) cross entropy loss as a function of  $\hat{z} = \hat{f}^{-1}(x_0)$  for our learned function  $\hat{f}$  is given by
$$
\mathcal {L} _ {\mathrm {C E}} = - p \ln \left(\frac {e ^ {h (\hat {z})}}{1 + e ^ {h (\hat {z})}}\right) - (1 - p) \ln \left(\frac {1}{1 + e ^ {h (\hat {z})}}\right) = - p h (\hat {z}) + \ln \left(1 + e ^ {h (\hat {z})}\right) \tag {135}
$$
We here drop the intervention index, since we focus on a one dimensional illustration. We plot two examples of such loss functions for  $b = 0$  and  $b \neq 0$  in Figure 6.
To verify that this is indeed the reason for our failure to recover the ground truth latent variables, we investigate the relation between estimated latent variables and ground truth latent variables. The result can be found in Figure 7 and, as we see, when  $\eta = 0$ , the distribution of the recovered latents is skewed. Moreover, we find that we essentially recover the latent variable up to a sign, i.e.,  $\hat{Z} = |Z|$  (there is a small offset because we enforce  $\hat{Z}$  to be centered). Note that this explains the tiny MCC scores in Table 1, since  $\mathbb{E}(Z|Z|) = 0$  for symmetric distributions. This observation might also be a potential starting point to improve our algorithm. For non-zero shifts we recover the latent variables almost perfectly.
G.2 Varsortability and data scale
An important observation in causal discovery was that many benchmark settings exhibit specific correlation structures that can be exploited to infer causal directions. The most prominent example is varisortability [66] which measures how well the causal order agrees with the order induced by the magnitude of the variance. This is a measure between 0 and 1 and for values close to 0 and 1 the variances contain a lot of information about the causal order while close to 0.5 this is not the case. It was shown in [66] that typical parameter choices exhibit high varisortability  $\approx 1$  which can be exploited by trivial algorithms. A related notion is  $R^2$ -varisortability [67] which is a similar notion where the variance of the variable is replaced by the residual variance after standardizing the variables and then regressing out all other variables, i.e., the unexplained variance.
In our setting the varisortability of the latent variables  $Z$  is quite high (see [66] for the full definition), e.g., for  $d = 10$  and  $k = 2$  average varisortability is  $0.92 \pm 0.02$ . Note that it is not obvious how this can be exploited algorithmically. Indeed, we only observe a mixture  $X = f(Z)$  and the scale of  $Y$  is not identifiable. Nevertheless, to rule out that we implicitly use the scale of the variables we ran the same experiments except that we standardized the latent variables  $Z$  of the observable distribution (and apply the same scaling to the interventional distributions). The results can be found in Table 4 and are roughly similar to the result in Table 2 for the same settings without standardization. Note that  $R^2$ -sortability is not substantially different from .5 (roughly 0.4) in our settings, thus the  $R^2$ -sortability cannot be exploited to infer the causal structure, even when observing  $Z$ .
G.3 Effect of noise distribution
Our theoretical results only cover the setting of Gaussian SCMs and we even show that identifiability does not necessarily hold for uniformly distributed noise (see Appendix D.2). Nevertheless, we can
Table 4: Results for nonlinear synthetic data with  $n = 10000$  and standardized latent variables  $Z$  (up to standardization the setting is the same as in Table 2 in the paper).
Setting	Method	SHD ↓	AUROC ↑	MCC ↑	R² ↑
ER(5, 2), d' = 20	Contrastive Learning	2.6 ± 0.4	0.92 ± 0.02	0.96 ± 0.01	0.93 ± 0.01
VAE	10.0 ± 0.0	0.50 ± 0.00	0.11 ± 0.01	0.04 ± 0.01
ER(5, 2), d' = 100	Contrastive Learning	1.4 ± 0.4	0.98 ± 0.02	0.98 ± 0.00	0.97 ± 0.01
VAE	10.0 ± 0.0	0.50 ± 0.00	0.14 ± 0.03	0.10 ± 0.04
ER(10, 2), d' = 20	Contrastive Learning	9.6 ± 2.1	0.90 ± 0.03	0.90 ± 0.01	0.84 ± 0.01
VAE	18.6 ± 0.9	0.50 ± 0.00	0.27 ± 0.04	0.29 ± 0.04
ER(10, 2), d' = 100	Contrastive Learning	2.8 ± 1.4	0.99 ± 0.01	0.98 ± 0.00	0.96 ± 0.00
VAE	18.6 ± 0.9	0.50 ± 0.00	0.14 ± 0.03	0.11 ± 0.04
Table 5: Noise dependence of contrastive algorithm for  $\mathrm{ER}(5,2)$ ,  $d' = 20$ ,  $n = 10000$  and nonlinear mixing (same setting as first row of Table 2 in the paper).
Noise distribution	SHD ↓	AUROC ↑	MCC ↑	R2↑
Gaussian	1.8 ± 0.5	0.97 ± 0.01	0.97 ± 0.00	0.96 ± 0.00
Laplace	3.0 ± 0.6	0.89 ± 0.03	0.93 ± 0.01	0.89 ± 0.01
Gumbel	3.0 ± 0.9	0.92 ± 0.02	0.94 ± 0.01	0.91 ± 0.01
Uniform	3.4 ± 0.4	0.87 ± 0.03	0.96 ± 0.01	0.93 ± 0.01
Exponential	3.8 ± 0.5	0.86 ± 0.02	0.89 ± 0.02	0.86 ± 0.02
evaluate the performance of our algorithm for linear SCMs with non-Gaussian noise distribution. We rerun one of the settings in Table 2 for different noise distributions. The results can be found in Table 5, and they show a slightly worse but still reasonable performance than for Gaussian data. This is in line with the observation that using a Gaussian likelihood for non-Gaussian data gives good results in, e.g., causal discovery.
H Hyperparameters, architectures and further additional details on experiments
In this section, we give additional details on the experiments. We use PyTorch [60] for all our experiments.