SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
= \log P _ {Y} (y) - \log k (y) + \log C _ {\lambda} + \log \lambda^ {\frac {x - h (y)}{k (y)}} + \log (1 - \lambda) ^ {1 - \frac {x - h (y)}{k (y)}}
$$
$$
\log P _ {X} (x) - \log g (x) + \log C _ {\lambda} + \frac {y - f (x)}{g (x)} \cdot \log \lambda + (1 - \frac {y - f (x)}{g (x)}) \cdot \log (1 - \lambda)
$$
$$
= \log P _ {Y} (y) - \log k (y) + \log C _ {\lambda} + \frac {x - h (y)}{k (y)} \cdot \log \lambda + \left(1 - \frac {x - h (y)}{k (y)}\right) \cdot \log (1 - \lambda)
$$
$$
\log P _ {X} (x) - \log g (x) + \log C _ {\lambda} + \frac {1}{g (x)} \cdot \log \lambda \cdot y - \frac {1}{g (x)} \cdot
$$
$$
\log \lambda \cdot f (x) + \log (1 - \lambda) - \log (1 - \lambda) \cdot \frac {1}{g (x)} \cdot y + \log (1 - \lambda) \cdot \frac {1}{g (x)} \cdot f (x)
$$
$$
= \log P _ {Y} (y) - \log k (y) + \log C _ {\lambda} + \frac {1}{k (y)} \cdot \log \lambda \cdot x - \frac {1}{k (y)} \cdot
$$
$$
\log \lambda \cdot h (y) + \log (1 - \lambda) - \log (1 - \lambda) \cdot \frac {1}{k (y)} \cdot x + \log (1 - \lambda) \cdot \frac {1}{k (y)} \cdot h (y)
$$
$$
\begin{array}{l} \left(\log P _ {X} (x)\right) ^ {\prime} - \frac {1}{g (x)} \cdot g (x) ^ {\prime} - 1 \cdot \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \cdot \log \lambda \cdot y \\ - \left(\frac {1}{g (x)} \cdot \log \lambda \cdot f (x)\right) ^ {\prime} - \log (1 - \lambda) \cdot - 1 \cdot \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \cdot y \\ + \left(\log (1 - \lambda) \cdot \frac {1}{g (x)} \cdot f (x)\right) ^ {\prime} = \frac {1}{k (y)} \cdot \log \lambda - \log (1 - \lambda) \cdot \frac {1}{k (y)} \\ \end{array}
$$
$$
\begin{array}{l} - \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \cdot \log \lambda - \log (1 - \lambda) \cdot - 1 \cdot \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \\ = - 1 \cdot \frac {1}{k (y) ^ {2}} \cdot k (y) ^ {\prime} \cdot \log \lambda - \log (1 - \lambda) \cdot - 1 \cdot \frac {1}{k (y) ^ {2}} \cdot k (y) ^ {\prime} \\ \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \cdot \log \lambda - \log (1 - \lambda) \cdot \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \\ = \frac {1}{k (y) ^ {2}} \cdot k (y) ^ {\prime} \cdot \log \lambda - \log (1 - \lambda) \cdot \frac {1}{k (y) ^ {2}} \cdot k (y) ^ {\prime} \\ \end{array}
$$
$$
\frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} \cdot (\log \lambda - \log (1 - \lambda)) = \frac {1}{k (y) ^ {2}} \cdot k (y) ^ {\prime} \cdot (\log \lambda - \log (1 - \lambda))
$$
Since  $\lambda \neq 0.5$ , therefore  $\log \lambda - \log (1 - \lambda) \neq 0$
$$
\frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} = \frac {1}{k (y) ^ {2}} \cdot k (y) ^ {\prime}
$$
$$
\frac {\partial \frac {1}{g (x)}}{\partial x} = \frac {\partial \frac {1}{k (y)}}{\partial y}
$$
Table 3: Table 1 without dataset standardization.
True Noise	Gaussian(0,1)	Uniform(-1,1)
α	0.1	0.5	1	5	10	0.1	0.5	1	5	10
\(\overline{\mathbb{V}}[Y|X]\)	0.106	0.846	3.162	77.309	309.035	0.013	0.199	0.780	19.386	77.529
\(\overline{\mathbb{V}}[X|Y]\)	0.446	0.707	0.790	0.818	0.814	0.016	0.126	0.189	0.228	0.226
Percentage of Datasets With Misleading CVs	0%	90%	100%	100%	100%	40%	80%	100%	100%	100%
CAREFL-M	1.0	1.0	1.0	1.0	1.0	0.6	0.5	0.5	0.0	0.0
CAREFL-H	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	1.0	0.9
E Proof for Lemma 4.1
Proof. According to Equation (2), we have  $P_{\rightarrow, P_Z}(Y|X; f, g) = P_{Z_Y}(\frac{Y - f(X)}{g(X)}) \cdot \frac{1}{g(X)}$  and  $P_{\rightarrow, P_{Z'}}(Y|X; f', g') = P_{Z_Y'}(\frac{Y - f'(X)}{g'(X)}) \cdot \frac{1}{g'(X)}$ . Let  $Z_Y'$  be the standardized version of  $Z_Y$ , i.e.  $Z_Y' = \frac{Z_Y - \mu}{\sigma}$ .
$$
Z _ {Y} = Z _ {Y} ^ {\prime} \cdot \sigma + \mu
$$
$$
\frac {\partial Z _ {Y}}{\partial Z _ {Y} ^ {\prime}} = \sigma
$$
$$
\begin{array}{l} P _ {Z _ {Y} ^ {\prime}} \left(\frac {Y - f ^ {\prime} (X)}{g ^ {\prime} (X)}\right) \cdot \frac {1}{g ^ {\prime} (X)} = P _ {Z _ {Y}} \left(\frac {Y - f ^ {\prime} (X)}{g ^ {\prime} (X)} \cdot \sigma + \mu\right) \cdot | \det  \frac {\partial Z _ {Y}}{\partial Z _ {Y} ^ {\prime}} | \cdot \frac {1}{g ^ {\prime} (X)} \\ = P _ {Z _ {Y}} \left(\frac {Y - f ^ {\prime} (X)}{g ^ {\prime} (X)} \cdot \sigma + \mu\right) \cdot \sigma \cdot \frac {1}{g ^ {\prime} (X)} \\ \end{array}
$$
Let  $\frac{1}{g(X)} = \sigma \cdot \frac{1}{g'(X)}$  and  $f'(X) = f(X) + \mu \cdot g(X)$ , then
$$
g ^ {\prime} (X) = \sigma \cdot g (X)
$$
$$
\begin{array}{l} P _ {Z _ {Y}} \left(\frac {Y - f ^ {\prime} (X)}{g ^ {\prime} (X)} \cdot \sigma + \mu\right) \cdot \sigma \cdot \frac {1}{g ^ {\prime} (X)} = P _ {Z _ {Y}} \left(\frac {Y - \left(f (X) + \mu \cdot g (X)\right)}{\sigma \cdot g (X)} \cdot \sigma + \mu\right) \cdot \sigma \cdot \frac {1}{\sigma \cdot g (X)} \\ = P _ {Z _ {Y}} \left(\frac {Y - f (X)}{g (X)}\right) \cdot \frac {1}{g (X)} \\ \end{array}
$$
Therefore,  $P_{\rightarrow ,P_Z}(Y|X;f,g) = P_{\rightarrow ,P_{Z'}}(Y|X;f',g')$
For conditional variances, we have:
$$
\mathbb {V} _ {Z _ {Y}} [ Y | X ] = g ^ {2} (X) \cdot \mathbb {V} [ Z _ {Y} ] = g ^ {2} (X) \cdot \sigma^ {2}
$$
$$
\mathbb {V} _ {Z _ {Y} ^ {\prime}} [ Y | X ] = g ^ {\prime 2} (X) \cdot \mathbb {V} [ Z _ {Y} ^ {\prime} ] = g ^ {\prime 2} (X) \cdot 1 = \sigma^ {2} \cdot g ^ {2} (X)
$$
Therefore,  $\mathbb{V}_{Z_Y}[Y|X] = \mathbb{V}_{Z_Y'}[Y|X]$ .
F Proof for Theorem 4.2
Proof. An LSNM model defined in Equation (1) entails the following relationships:
$$
\mathbb {V} [ Y | X ] = g ^ {2} (X) \cdot \mathbb {V} [ Z _ {Y} ] = g ^ {2} (X)
$$
$$
\log P _ {\rightarrow , P _ {Z}} (Y | X) = \log P _ {Z _ {Y}} (Z _ {Y}) + \log \frac {1}{g (X)}
$$
where Lemma 4.1 shows that we can always assume  $\mathbb{V}[Z_Y] = 1$ . Also, since there is no causal relationship between  $X$  and  $Z_{Y}$  in the ground-truth data, i.e., changing  $X$  does not change  $Z_{Y}$ , we have  $\frac{\partial Z_Y}{\partial X} = 0$ ,  $\frac{\partial\log P_{Z_Y}(Z_Y)}{\partial X} = 0$  and  $\frac{\partial\log P_{Z_Y}(Z_Y)}{\partial g^2(X)} = 0$ .
$$
\begin{array}{l} \frac {\partial \log P _ {\rightarrow , P _ {Z}} (Y | X)}{\partial \mathbb {V} [ Y | X ]} = \frac {\partial \log P _ {Z _ {Y}} (Z _ {Y})}{\partial g ^ {2} (X)} + \frac {\partial \log \frac {1}{g (X)}}{\partial g ^ {2} (X)} \\ = 0 + \frac {\partial \log \frac {1}{g (X)}}{\partial g ^ {2} (X)} \\ = - \frac {1}{2 \cdot g ^ {2} (X)} \\ \end{array}
$$
Since  $g(X) > 0$ , i.e., strictly positive on the domain of  $X$ , therefore,  $\frac{\partial \log P_{\rightarrow ,P_Z}(Y|X)}{\partial \mathbb{V}[Y|X]} < 0$ .
G Proof for Corollary 4.3
Proof. To compare the data log-likelihood under the causal model  $\rightarrow$  and under the anti-causal model  $\leftarrow$ , we have:
$$
\begin{array}{l} \log P _ {\rightarrow , P _ {Z}} (X, Y) - \log P _ {\leftarrow , P _ {Z}} (X, Y) \\ = \left[ \log P _ {\rightarrow , P _ {Z}} (X) + \log P _ {\rightarrow , P _ {Z}} (Y | X) \right] - \left[ \log P _ {\leftarrow , P _ {Z}} (Y) + \log P _ {\leftarrow , P _ {Z}} (X | Y) \right]\left. \right. \\ \end{array}
$$
Theorem 4.2 shows that conditional likelihood and conditional variance are negatively related. Therefore, by increasing  $\mathbb{V}[Y|X]$  and decreasing  $\mathbb{V}[X|Y]$  in the data,  $\log P_{\rightarrow ,P_Z}(Y|X)$  decreases and  $\log P_{\leftarrow ,P_Z}(X|Y)$  increases. Therefore,  $\log P_{\rightarrow ,P_Z}(X,Y)$  decreases,  $\log P_{\leftarrow ,P_Z}(X,Y)$  increases, and their difference decreases.
H The ANM version of Lemma 4.1
Proposition H.1. Let  $M_1$  be an ANM model  $Y \coloneqq f(X) + Z_Y$ , where  $Z_Y$  is the noise with mean  $\mu$  and variance  $\sigma^2 \neq 0$ . Let  $M_2$  be another ANM model  $Y \coloneqq f'(X) + Z_Y'$ , where  $Z_Y'$  is the standardized version of  $Z_Y$ , i.e.  $Z_Y' = \frac{Z_Y - \mu}{\sigma}$ . When both  $Z_Y$  and  $Z_Y'$  are Gaussian,  $P_{\rightarrow, P_Z}(Y|X; f) \neq P_{\rightarrow, P_{Z'}}(Y|X; f')$ .
Proof. ANM is a restricted case of LSNM with  $g(X) = 1$ . Similar to Equation (2), we have  $P_{\to, P_Z}(Y|X; f) = P_{Z_Y}(Y - f(X))$  for  $M_1$  and  $P_{\to, P_{Z'}}(Y|X; f') = P_{Z_Y'}(Y - f'(X))$  for  $M_2$ .
We show that  $P_{Z_Y}(Y - f(X)) \neq P_{Z_Y'}(Y - f'(X))$  when  $Z_Y$  and  $Z_Y'$  are Gaussian.
$$
\begin{array}{l} P _ {Z _ {Y}} (Y - f (X)) = \frac {1}{\sigma \cdot \sqrt {2 \pi}} \cdot e ^ {- \frac {1}{2} \cdot (\frac {Y - f (X) - \mu}{\sigma}) ^ {2}} \\ = \frac {1}{\sigma \cdot \sqrt {2 \pi}} \cdot e ^ {- \frac {1}{2} \cdot (\frac {Z _ {Y} - \mu}{\sigma}) ^ {2}} \\ = \frac {1}{\sigma \cdot \sqrt {2 \pi}} \cdot e ^ {- \frac {1}{2} \cdot (Z _ {Y} ^ {\prime}) ^ {2}} \\ \end{array}
$$
$$
\begin{array}{l} P _ {Z _ {Y} ^ {\prime}} (Y - f ^ {\prime} (X)) = \frac {1}{1 \cdot \sqrt {2 \pi}} \cdot e ^ {- \frac {1}{2} \cdot (\frac {Y - f ^ {\prime} (X) - 0}{1}) ^ {2}} \\ = \frac {1}{\sqrt {2 \pi}} \cdot e ^ {- \frac {1}{2} \cdot (Z _ {Y} ^ {\prime}) ^ {2}} \\ \end{array}
$$
Therefore,  $P_{\rightarrow ,P_Z}(Y|X;f)\neq P_{\rightarrow ,P_{Z'}}(Y|X;f')$  in the case of Gaussian unless  $\sigma = 1$ . Empirical results are provided in Table 4b and Appendix P.
Table 4: Comparison between the accuracy and ground-truth noise variance in LSNMs and ANMs.  $N = 10,000$ . We used  $N(0,1)$  as model prior. All the datasets are standardized to have mean 0 and variance 1. As implied by Lemma 4.1, ML model selection in LSNMs is insensitive to noise variance, when the form of the noise distribution is correctly specified. Proposition H.1 shows that ML model selection in ANMs is sensitive to noise variance, even when the form of the noise distribution is correctly specified.
(a) Accuracy over 10 datasets generated by SCM LSNM-sine-tanh (definition in Appendix M).
True Noise	N(0,1)	N(0,4)	N(0,25)	N(0,100)	N(0,400)
\(\overline{\mathbb{V}}[Y|X]\)	0.834	0.950	0.997	0.999	0.999
\(\overline{\mathbb{V}}[X|Y]\)	0.793	0.810	0.784	0.781	0.780
Percentage of Datasets With Misleading CVs	70%	100%	100%	100%	100%
CAREFL-M	1.0	1.0	1.0	1.0	1.0
LOCI-M	1.0	1.0	1.0	1.0	1.0
(b) Accuracy over 10 datasets generated by SCM ANM-sine (definition in Appendix P).
True Noise	N(0,1)	N(0,4)	N(0,25)	N(0,100)	N(0,400)
\(\overline{\mathbb{V}}[Y|X]\)	0.632	0.865	0.991	0.999	0.999
\(\overline{\mathbb{V}}[X|Y]\)	0.756	0.979	0.999	0.999	0.999
Percentage of Datasets With Misleading CVs	0%	0%	0%	50%	40%
CAREFL-ANM-M	1.0	1.0	1.0	0.6	0.5
CAM	1.0	1.0	0.7	0.5	0.6
I CAREFL-M
CAREFL-M [11] models an LSNM in Equation (1) via affine flows  $\mathbf{T}$ . Each sub-flow  $T_{k} \in \mathbf{T}$  is defined as the following:
$$
X = t _ {1} + e ^ {s _ {1}} \cdot Z _ {X}
$$
$$
Y = t _ {2} (X) + e ^ {s _ {2} (X)} \cdot Z _ {Y} \tag {6}
$$
where  $X$  is the putative cause and  $Y$  is the putative effect in  $X \rightarrow Y$  direction.  $t_1$  and  $s_1$  are constants.  $t_2$  and  $s_2$  are functions parameterized using neural networks. Without loss of generality,  $X$  is assumed to be a function of latent noise variable  $Z_{X}$ . If  $t_1 = 0$  and  $s_1 = 0$ , then  $X = Z_{X}$ . The exponential function  $e$  ensures the multipliers to  $Z$  are positive without expression loss. Similarly, for the backward direction  $X \gets Y$ :
$$
Y = t _ {1} ^ {\prime} + e ^ {s _ {1} ^ {\prime}} \cdot Z _ {Y}
$$
$$
X = t _ {2} ^ {\prime} (Y) + e ^ {s _ {2} ^ {\prime} (Y)} \cdot Z _ {X} \tag {7}
$$
where  $Y$  is the putative cause and  $X$  is the putative effect in  $X \gets Y$  direction.  $t_1'$  and  $s_1'$  are constants.  $t_2'$  and  $s_2'$  are functions parameterized using neural networks.
Given Equation (6), the joint log-likelihood of  $(x,y)$  in  $X\to Y$  direction is:
$$
\begin{array}{l} \log P _ {\rightarrow , P _ {Z}} (x, y) \\ = \log P _ {Z _ {X}} \left(e ^ {- s _ {1}} \cdot \left(x - t _ {1}\right)\right) + \log P _ {Z _ {Y}} \left(e ^ {- s _ {2} (x)} \cdot \left(y - t _ {2} (x)\right)\right) - s _ {1} - s _ {2} (x) \tag {8} \\ \end{array}
$$
Similarly for the  $X \leftarrow Y$  direction. Note that the priors  $P_Z = \{P_{Z_X}, P_{Z_Y}\}$  in Equation (8) may mismatch the unknown ground-truth noise distribution  $P_Z^* = \{P_{Z_X}^*, P_{Z_Y}^*\}$ . Both CAREFL-M and
Algorithm 2 CAREFL-M
1: Input: data pairs  $D \coloneqq (X, Y)$ , and the flow estimator  $\mathbf{T}$  with prior  $P_Z$
2: Output: estimated causal direction dir
3: Split  $D$  into training set  $D_{train} \coloneqq (X_{train}, Y_{train})$  and testing set  $D_{test} \coloneqq (X_{test}, Y_{test})$
4: Optimize  $\mathbf{T}_{\widehat{t}_1,\widehat{s}_1,\widehat{t}_2,\widehat{s}_2}(D_{train};P_Z)$  in  $X\to Y$  direction via ML
5: Compute the likelihood  $\widehat{L}_{\rightarrow ,P_Z}(D_{test};\mathbf{T}_{\widehat{t}_1,\widehat{s}_1,\widehat{t}_2,\widehat{s}_2})$  in  $X\to Y$  direction
6: Optimize  $\mathbf{T}_{\hat{t}^{\prime}_1,\hat{s}^{\prime}_1,\hat{t}^{\prime}_2,\hat{s}^{\prime}_2}(D_{train};P_Z)$  in  $X\gets Y$  direction via ML
7: Compute the likelihood  $\widehat{L}_{\leftarrow ,P_Z}\big(D_{test};\mathbf{T}_{\hat{t}_1,\hat{s}_1',\hat{t}_2,\hat{s}_2'}\big)$  in  $X\gets Y$  direction
8: if  $\widehat{L}_{\rightarrow ,P_Z}(D_{test};\mathbf{T}_{\widehat{t}_1,\widehat{s}_1,\widehat{t}_2,\widehat{s}_2}) > \widehat{L}_{\leftarrow ,P_Z}(D_{test};\mathbf{T}_{\widehat{t'}_1,\widehat{s'}_1,\widehat{t'}_2,\widehat{s'}_2})$  then
9: dir :=  $X \to Y$
10: else if  $\widehat{L}_{\rightarrow ,P_Z}(D_{test};\mathbf{T}_{\widehat{t}_1,\widehat{s}_1,\widehat{t}_2,\widehat{s}_2}) < \widehat{L}_{\leftarrow ,P_Z}(D_{test};\mathbf{T}_{\widehat{t}_1,\widehat{s}_1,\widehat{t}_2,\widehat{s}_2})$  then
11: dir := X ← Y
12: else
13: dir := no conclusion
14: end if
Algorithm 3 CAREFL-H (Between Residuals)
1: Input: data pairs  $D \coloneqq (X, Y)$ , the flow estimator  $\mathbf{T}$  of CAREFL-M with prior  $P_Z$ , and an HSIC estimator
2: Output: estimated causal direction dir
3: Split  $D$  into training set  $D_{train} \coloneqq (X_{train}, Y_{train})$  and testing set  $D_{test} \coloneqq (X_{test}, Y_{test})$
4: Optimize  $\mathbf{T}_{\hat{t}_1,\hat{s}_1,\hat{t}_2,\hat{s}_2}(D_{train};P_Z)$  in  $X\rightarrow Y$  direction via ML to estimate  $\hat{t}_1$ $\hat{s}_1$ $\hat{t}_2$  and  $\hat{s}_2$
5: Compute the residuals  $\widehat{Z}_{X,\rightarrow}:= \frac{X_{test} - \widehat{t}_1}{e^{\widehat{s}1}}$  and  $\widehat{Z}_{Y,\rightarrow}:= \frac{Y_{test} - \widehat{t}_2(X)}{e^{\widehat{s}2(X)}}$
6: Optimize  $\mathbf{T}_{\widehat{t}_1, \widehat{s}_1', \widehat{t}_2, \widehat{s}_2'}(D_{train}; P_Z)$  in  $X \gets Y$  direction via ML to estimate  $\widehat{t}_1'$ ,  $\widehat{s}_1'$ ,  $\widehat{t}_2'$  and  $\widehat{s}_2'$ .
7: Compute the residuals  $\widehat{Z}_{Y,\leftarrow}:= \frac{Y_{test} - \widehat{t^{\prime}}_{1}}{e^{\widehat{s}\widehat{t}_{1}}} \text{ and } \widehat{Z}_{X,\leftarrow}:= \frac{X_{test} - \widehat{t^{\prime}}_{2}(Y)}{e^{\widehat{s}\widehat{t}_{2}(Y)}}$
8: if HSIC  $(\widehat{Z}_{X,\rightarrow},\widehat{Z}_{Y,\rightarrow}) <   HSIC(\widehat{Z}_{X,\leftarrow},\widehat{Z}_{Y,\leftarrow})$  then
9: dir :=  $X \to Y$
0: else if  $HSIC(\widehat{Z}_{X,\rightarrow},\widehat{Z}_{Y,\rightarrow}) > HSIC(\widehat{Z}_{X,\leftarrow},\widehat{Z}_{Y,\leftarrow})$  then
1: dir := X ← Y
2: else
3: dir := no conclusion
4: end if
CAREFL-H optimize Equation (8) for each direction over the training set. For CAREFL-M, it chooses the direction with ML Score  $L$  over the testing set as the estimated causal direction. Detailed procedure for CAREFL-M is given in Algorithm 2. To map the parameters  $\theta, \psi, \zeta, \theta', \psi'$  and  $\zeta'$  in LSNM (Equation (2)) to the flow estimator  $\mathbf{T}$  of CAREFL (Equations (6) and (7)), we have  $f_{\theta} \equiv t_2$ ,  $g_{\psi} \equiv e^{s_2}$ ,  $P_{X,\zeta} \equiv \{t_1, e^{s_1}\}$ ,  $f_{\theta'} \equiv t_2'$ ,  $g_{\psi'} \equiv e^{s_2'}$  and  $P_{Y,\zeta'} \equiv \{t_1', e^{s_1'}\}$ .
J CAREFL-H alternative independence testing
In Algorithm 1, CAREFL-H tests independence between the putative cause and the residual of the putative effect in each direction, i.e., between  $X$  and  $\widehat{Z}_Y$  in  $X \to Y$  direction, and between  $Y$  and  $\widehat{Z}_X$  in  $X \gets Y$  direction. An alternative way of testing independence is to test between the residual of the putative cause and the residual of the putative effect [6], i.e., between  $\widehat{Z}_X$  and  $\widehat{Z}_Y$  in both directions. Please see Algorithm 3 for complete steps.
Although in our experiments the two algorithms often produce the same estimation of causal direction, we prefer Algorithm 1, since it relies on few estimations of the residuals.
K Suitability theory
With a consistent HSIC estimator, Mooij et al. [14] show that an IT method consistently selects the causal direction for ANMs if the regression method is suitable. A regression method is suitable if the expected mean squared error between the predicted residuals  $\widehat{E}$  and the true residuals  $E$  approaches 0 in the limit of  $N\to \infty$ :
$$
\lim  _ {N \rightarrow \infty} \mathbb {E} _ {D, D ^ {\prime}} \left[ \frac {1}{N} | | \widehat {E} _ {1 \dots N} - E _ {1 \dots N} | | ^ {2} \right] = 0 \tag {9}
$$
where  $D$  and  $D^{\prime}$  denote training set and testing set, respectively. Hence, with enough data a suitable regression method reconstructs the ground-truth noise.
If an HSIC estimator is consistent, the estimated HSIC value converges in probability to the population HSIC value.
$$
\widehat {H S I C} (X, Y) \xrightarrow {P} H S I C (X, Y).
$$
With a consistent HSIC estimator and a suitable regression method, the consistency result for ANMs in Mooij et al. [14] extends naturally to LSNMs (see Appendix K.2 for a proof outline).
Proposition K.1. For identifiable LSNMs with an independent noise term in one causal direction only, if an IT method is used with a suitable regression method for LSNMs and a consistent HSIC estimator, then the IT method is consistent for inferring causal direction for LSNMs.
It is important to note that, while the suitability of regression methods is a valuable property for IT-based methods aiding in the determination of causal directions using independence testing, it does not guarantee a higher likelihood for the causal direction compared to the anti-causal direction under noise misspecification. Therefore, suitability does not offer the same robustness guarantee for ML-based methods.
K.1 Suitability: empirical results
Table 5: Suitability of the flow estimator  $\mathbf{T}$  of CAREFL-M in the causal direction under noise misspecification and misleading CVs.  $\mathbf{T}$  is trained with a Laplace prior. The original dataset with size  $2N$  is split into two:  $50\%$  as training set and  $50\%$  as testing set.  $\overline{\mathbb{V}}[Y|X] > \overline{\mathbb{V}}[X|Y]$  indicates misleading CVs in the dataset. LSNM definition is in Appendix M.
(a) LSNM-tanh-exp-cosine and continuousBernoulli(0.9) noise.  $\overline{\mathbb{V}}[Y|X]$  vs.  $\overline{\mathbb{V}}[X|Y]$ : 0.324 vs. 0.291.
[SZX,SZY]	N=50	N=500	N=1000	N=5000
[0.02406,0.01283]	[0.00194,0.00204]	[0.00094,0.00092]	[0.0002,0.00018]
(b) LSNM-sine-tanh and uniform(-1,1) noise. V[Y|X] vs. V[X|Y]: 0.422 vs. 0.367.
[SZX,SZY]	N=50	N=500	N=1000	N=5000
[0.0081,0.00285]	[0.00039,0.00031]	[0.0002,0.00017]	[0.00003,0.00003]
(c) LSNM-sigmoid-sigmoid and exponential(1) noise. V[Y|X] vs. V[X|Y]: 0.927 vs. 0.657.
[SZX,SZY]	N=50	N=500	N=1000	N=5000
[0.00979,0.00443]	[0.00074,0.00058]	[0.00045,0.00026]	[0.00005,0.00005]
Let suitability value  $S$  be the left-hand side of Equation (9). Table 5 shows an empirical evaluation of  $S$ , for the flow estimator  $\mathbf{T}$  in the causal direction. We generate data from 3 synthetic LSNMs, and evaluate  $\mathbf{T}$  under noise misspecification and misleading CVs. We find that as the sample size grows,  $S$  approaches 0. In other words,  $\mathbf{T}$  is empirically suitable under noise misspecification and misleading CVs. Therefore, Proposition K.1 entails that CAREFL-H based on  $\mathbf{T}$  and a consistent HSIC estimator is empirically consistent for inferring causal direction in LSNMs under these conditions.
Because  $\mathbf{T}$  in CAREFL-M [11] uses neural networks to approximate the observed data distribution, it is difficult to provide a theoretical guarantee of suitability. Therefore, although our experiments indicate that  $\mathbf{T}$  is often suitable in practice, we do not claim that it is suitable for all LSNMs. For
example, it may not be suitable in the low-noise regime when the LSNMs are close to deterministic. It is well-known that since neural network models are universal function approximators, we can expect them to fit many regression functions, although it is difficult to prove suitability theoretically. For these reasons, we do not claim a priori that flow models are suitable, but we provide empirical evidence that with careful training and hyperparameter selection, we can expect them to be suitable in practice.
K.2 Proof outline for Proposition K.1
For data pairs  $(X,Y)$  generated by an LSNM model  $X\to Y$  , we have:
1. Under suitability, the reconstructed noise  $\widehat{Z}_Y$  approaches the ground-truth noise  $Z_Y$ .
2. By employing a consistent HSIC estimator, the estimated HSIC value  $\widehat{HSIC}$  converges to the true HSIC value  $HSIC$ .
3. In the ground-truth LSNM,  $HSIC(X, \widehat{Z}_Y) = 0$  and  $HSIC(Y, \widehat{Z}_X) \neq 0$  (i.e. identifiability of LSNMs).
Therefore,  $\widehat{HSIC}(X, \widehat{Z}_Y) = 0$  and  $\widehat{HSIC}(Y, \widehat{Z}_X) \neq 0$  for LSNMs. We only provide a proof outline here, since a formal proof is analogous to the proof for ANMs in Appendix A of Mooij et al. [14].
L Default and alternative hyperparameter values used in Section 7
We use the reported hyperparameter values in CAREFL-M [11] for the Tübingen Cause-Effect Pairs benchmark [14] as the default hyperparameter values in all our experiments:
- The flow estimator  $\mathbf{T}$  is parameterized with  $\underline{4}$  sub-flows (alternatively: 1, 7 and 10).
- For each sub-flow,  $f, g, h$  and  $k$  are modelled as four-layer MLPs with 5 hidden neurons in each layer (alternatively: 2, 10 and 20).
- Prior distribution is Laplace (alternatively: Gaussian prior).
- Adam optimizer [12] is used to train each model for 750 epochs (alternatively: 500, 1000 and 2000).
- L2-penalty strength is  $\underline{0}$  by default (alternatively: 0.0001, 0.001, 0.1).
Although we also observe that LOCI-H is more robust than LOCI-M under noise misspecification and misleading CVs, we omit their results because: (1) CAREFL often outperforms LOCI (see Tables 1 and 2). (2) LOCI is fixed as a Gaussian distribution, whereas CAREFL can specify different prior distributions. (3) LOCI uses different set of hyperparameters than CAREFL. So their results cannot be merged into the same figures.
M Synthetic SCMs used in Section 7.1
We use the following SCMs to generate synthetic datasets:
$$
\begin{array}{l} \text {L S N M - t a n h - e x p - c o s i n e :} \quad Y := \tanh  (X \cdot \theta_ {1}) \cdot \theta_ {2} + e ^ {\cos (X \cdot \psi_ {1}) \cdot \psi_ {2}} \cdot Z _ {Y} \\ \text {L S N M - s i n e - t a n h :} \quad Y := \sin \left(X \cdot \theta_ {1}\right) \cdot \theta_ {2} + (\tanh  (X \cdot \psi) + \phi) \cdot Z _ {Y} \tag {10} \\ \end{array}
$$
$$
\text {L S N M - s i g m o i d - s i g m o i d :} \quad Y := \sigma (X \cdot \theta_ {1}) \cdot \theta_ {2} + \sigma (X \cdot \psi_ {1}) \cdot \psi_ {2} \cdot Z _ {Y}
$$
where  $Z_{Y}$  is the ground-truth noise sampled from one of the following distributions: continuousBernoulli(0.9), uniform(-1,1), exponential(1), beta(0.5,0.5), Gaussian(0,1) and Laplace(0,1).  $\sigma$  is the sigmoid function. Although we did not prove identifiability with beta(0.5,0.5) noise in Theorem C.2, empirically we find it is identifiable. Following [33, 34], each  $\theta$  and  $\psi$  are sampled uniformly from range  $[-2, -0.5] \cup [0.5,2]$ .  $\phi$  is sampled uniformly from range [1,2] to make the tanh function positive. The number of data pairs in each synthetic dataset is  $N \in \{500,5000\}$ .
N Hyperparameter values of CAREFL-H for Table 2b
Following Khemakhem et al. [11], we use a single set of hyperparameters for all 99 datasets, which is found by grid search. To acquire the result of CAREFL-H in Table 2b, the hyperparameter values used are as follows:
- Number of hidden neurons in each layer of the MLPs: 2
Number of sub-flows: 10
- Training dataset = testing dataset = 100%.
The rest of the hyperparameter values are identical to the default ones.
O Running time