SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Despite these complications, we advocate the use of IT-based methods over ML-based methods under noise misspecification and misleading CVs in LSNMs, because they are more reliable in such settings.
On synthetic datasets, we find that across different hyperparameter choices the IT method (CAREFL-H) produces much higher accuracy than the ML method (CAREFL-M) in the difficult settings with noise misspecification and misleading CVs, and produces comparable accuracy in the easier settings without noise misspecification or misleading CVs. On real-world data where the ground-truth noise distribution is unknown, the IT method is also more robust across different hyperparameter choices.
For all experiments, we start with the same default hyperparameter values for both CAREFL-M and CAREFL-H and alter one value at a time. The default hyperparameter values are those specified in CAREFL-M [11] for the Tübingen Cause-Effect Pairs benchmark [14]. Please see Appendix L for more details on default and alternative hyperparameter values. Previous work [14, 9] reported that ML methods perform better with data splitting (split data into training set for model fitting and testing set for model selection) and IT methods perform better with data recycling (the same data is used for both model fitting and selection). Therefore, we use both splitting methods: (i) CAREFL(0.8):  $80\%$  as training and  $20\%$  as testing. (ii) CAREFL(1.0): training = testing =  $100\%$ . The training procedure is not supervised: no method accesses the ground-truth direction.
Figure 3: Weighted accuracy over 99 datasets from Tübingen Cause-Effect Pairs benchmark.
Table 2: Accuracy of methods on the SIM and Tübingen Cause-Effect Pairs benchmarks. For methods other than CAREFL-M and CAREFL-H, we use the best results reported in Immer et al. [9].
(a) Accuracy with SIM benchmarks
LOCI-M	LOCI-H	GRCI	BQCD	HECI	CAM	RESIT	CAREFL-M	CAREFL-H
SIM	0.52	0.79	0.77	0.62	0.49	0.57	0.77	0.55	0.80
SIM-c	0.50	0.83	0.77	0.72	0.55	0.60	0.82	0.58	0.85
SIM-ln	0.79	0.73	0.77	0.80	0.65	0.87	0.87	0.84	0.83
SIM-G	0.78	0.81	0.70	0.64	0.56	0.81	0.78	0.82	0.79
(b) Weighted accuracy with Tübingen Cause-Effect Pairs benchmark
LOCI-M	LOCI-H	GRCI	BQCD	HECI	CAM	RESIT	CAREFL-M	CAREFL-H
Tübingen Cause-Effect Pairs	0.57	0.60	0.82	0.77	0.71	0.58	0.57	0.73	0.82
We use a consistent HSIC estimator with Gaussian kernels [21]. A summary of experimental datasets is provided in Appendix Table 7. All the datasets are standardized to have mean 0 and variance 1. The datasets in Section 7.1 are generated by LSNMs and the datasets in Sections 7.2 and 7.3 are not. Consequently, the results demonstrate that the proposed method performs well not only when the ground-truth SCMs are strictly LSNMs, but also when the ground-truth SCMs are unknown or not strictly LSNMs.
7.1 Results on synthetic LSNM data
See Appendix M for the definition of the ground-truth LSNM SCMs and details on how synthetic datasets are generated from them. The sample sizes in each synthetic dataset are 500 or 5,000. As shown in Appendix Table 7, most synthetic datasets generated by such SCMs have misleading CVs. Based on the analysis of Section 4 we formulate the following hypotheses. (i) CAREFL-M should be accurate given a correct specification on the form of noise distribution, with or without misleading CVs. (ii) With noise misspecification and mild misleading CVs, the accuracy of CAREFL-M should be reduced. (iii) With noise misspecification and severe misleading CVs, the accuracy should be very low, often below  $50\%$ . Overall, the results from the experiments confirm our hypotheses.
7.1.1 Noise misspecification
We evaluate CAREFL-M and CAREFL-H against data generated with uniform  $(-1, 1)$ , exponential(1), continuous Bernoulli(0.9) or beta(0.5, 0.5) noise, covered by our identifiability Theorem C.2 (except beta(0.5, 0.5)). Khemakhem et al. [11] empirically find that CAREFL-M with a Laplace model prior is robust with similar ground-truth noise distributions such as Gaussian and Student's t. We show that it may fail remarkably with a dissimilar distribution.
We summarize findings from the 336 settings here; the detailed results are given in Appendix Figures 6 to 17. In 289 settings (86.01%), both CAREFL-M(0.8) and CAREFL-M(1.0) select the correct causal
direction with less than  $50\%$  random accuracy. Furthermore, in 110 settings  $(32.74\%)$  both CAREFL-M(0.8) and CAREFL-M(1.0) fail catastrophically with an accuracy of  $0\%$ . These experiments also show that the accuracy of CAREFL-M often decreases as  $N$  increases. In contrast, CAREFL-H(1.0) achieves better accuracy than CAREFL-M in 333 settings  $(99.11\%)$ . The accuracy of CAREFL-H(1.0) goes below  $50\%$  in only 6 settings  $(1.79\%)$ . The results demonstrate the robustness of the IT method under noise misspecification and misleading CVs, across different hyperparameter choices.
Appendix Table 7 and Appendix Figure 17 show that with severe misleading CVs, the accuracy of CAREFL-M is close to  $0\%$ . This is much lower than the corresponding cases with mild misleading CVs in Appendix Figures 9 and 13.
7.1.2 Correct specification on the form of the noise distribution
These experiments show that CAREFL-H is comparable with CAREFL-M under correct form of noise specification, with or without misleading CVs, especially on larger datasets, as long as the affine model capacity is sufficient. We evaluate CAREFL-M and CAREFL-H against data generated with Gaussian(0,1) and Laplace(0,1) noise. The detailed results are in Appendix Figures 18 to 23. We find that CAREFL-M is more sample efficient than CAREFL-H when the model prior matches the data, which is a general pattern for ML vs. IT methods [23]. Consistent with the suitability results in Appendix K.1, the accuracy of CAREFL-H improves with more data. For example, with  $N = 500$ , there are 49 out of 84 settings (58.33%) where CAREFL-M outperforms CAREFL-H(1.0). However, with  $N = 5,000$ , CAREFL-H(1.0) achieves similar accuracy as CAREFL-M on all datasets (except LSNM-sigmoid-sigmoid with Laplace(0,1) noise.) In addition, CAREFL-H(1.0) may underperform CAREFL-M when the number of hidden neurons, sub-flows or training epochs is low. An IT method requires more model capacity to fit the LSNM functions and produce a good reconstruction of the noise.
7.2 Results on synthetic benchmarks
Similar to Tagasovska et al. [29], Immer et al. [9], we compare CAREFL-M and CAREFL-H against the SIM benchmark suite [14]. SIM comprises 4 sub-benchmarks: default (SIM), with one confounder (SIM-c), low noise levels (SIM-ln) and Gaussian noise (SIM-G). In this benchmark, most datasets do not have misleading CVs (Appendix Table 7), which favors ML methods. Each sub-benchmark contains 100 datasets and each dataset has  $N = 1000$  data pairs. As shown in Appendix Figure 24, CAREFL-M and CAREFL-H(1.0) achieve similar accuracy on SIM-ln and SIM-G across different hyperparameter choices. For SIM and SIM-c, CAREFL-H, especially CAREFL-H(1.0), outperforms CAREFL-M by  $20\% -30\%$  in all settings. The accuracy of CAREFL-M is only about random guess  $(40\% -60\%)$  on SIM and SIM-c.
Table 2a compares CAREFL-H with SOTA methods. For each CAREFL method, we report the best accuracy obtained in Appendix Figure 24 without further tuning. CAREFL-H achieves the best accuracy on SIM and SIM-c, and achieves competitive accuracy on SIM-ln and SIM-G.
7.3 Results on real-world benchmarks: Tübingen Cause-Effect Pairs
The Tubingen Cause-Effect Pairs benchmark [14] is commonly used to evaluate cause-effect inference algorithms [11, 30, 9]. To be consistent with previous work [29, 28, 9], we exclude 6 multivariate and 3 discrete datasets (#47, #52-#55, #70, #71, #105, #107) and utilize the remaining 99 bivariate datasets. As recommended by Mooij et al. [14], we report weighted accuracy.  $40\%$  of datasets in the benchmark feature misleading CVs. CAREFL-H(1.0) outperforms CAREFL-M in all configurations by large margins  $(7\%-30\%)$ ; see Figure 3. We also compare CAREFL-H with SOTA methods (see Appendix N for hyperparameters). Table 2b shows that CAREFL-H achieves the SOTA accuracy  $(82\%)$  and is  $9\%$  more accurate than CAREFL-M [11].
Furthermore, LSNM methods (i.e. LOCI, GRCI, BQCD, HECI, CAREFL) perform better than ANM methods (i.e. CAM, RESIT). The reason is that LSNM is a weaker assumption than ANM and allows for heteroscedastic noise.
8 Conclusion and future work
We identified a failure mode of maximum-likelihood (ML) methods for cause-effect inference in location-scale noise models (LSNMs). Our analysis shows that the failure occurs when the form of the noise distribution is misspecified and conditional variances are misleading (i.e., higher in the causal direction). Selecting causal models by independence tests (IT) is robust even in this difficult setting. Extensive empirical evaluation compared the ML method and a new IT method based on affine flows, on both synthetic and real-world datasets. The IT flow method achieves better accuracy under noise misspecification and misleading CVs, with robust performance across different hyperparameter choices. Future directions include improving the sample efficiency of IT methods, and improving the robustness of ML methods by learning the noise distribution instead of using a fixed prior.
9 Acknowledgments
This research was supported by a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC).
[1] Patrick Blöbaum, Dominik Janzing, Takashi Washio, Shohei Shimizu, and Bernhard Scholkopf. Cause-effect inference by comparing regression errors. In International Conference on Artificial Intelligence and Statistics, pages 900–909. PMLR, 2018.
[2] Peter Buhlmann, Jonas Peters, and Jan Ernest. Cam: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics, 42(6):2526, 2014.
[3] Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294-321, 2012.
[4] Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.02195, 2022.
[5] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pages 63-77. Springer, 2005.
[6] Yue He, Peng Cui, Zheyan Shen, Renzhe Xu, Furui Liu, and Yong Jiang. Daring: Differentiable causal discovery with residual independence. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 596-605, 2021.
[7] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Scholkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.
[8] Biwei Huang. Diagnosis of autism spectrum disorder by causal influence strength learned from resting-state fmri data. In Neural Engineering Techniques for Autism Spectrum Disorder, pages 237-267. Elsevier, 2021.
[9] Alexander Immer, Christoph Schultheiss, Julia E Vogt, Bernhard Scholkopf, Peter Buhlmann, and Alexander Marx. On the identifiability and estimation of causal location-scale noise models. In International Conference on Machine Learning, pages 14316-14332. PMLR, 2023.
[10] Markus Kalisch and Peter Buhlman. Estimating high-dimensional directed acyclic graphs with the pc-algorithm. Journal of Machine Learning Research, 8(3), 2007.
[11] Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International conference on artificial intelligence and statistics, pages 3520-3528. PMLR, 2021.
[31] Kun Zhang and Lai-Wan Chan. Extensions of ICA for causality discovery in the hong kong stock market. In International Conference on Neural Information Processing, pages 400-409. Springer, 2006.
[32] Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv:1205.2599, 2012.
[33] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
[34] Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics, pages 3414-3425. PMLR, 2020.
A Extension to multivariate settings
To extend a bivariate cause-effect inference method to multivariate, the most common approach is to first use the PC algorithm [27] (or other algorithms) to learn a completed partially directed acyclic graph (CPDAG) that may contain undirected edges. Then, orient the undirected edges using the cause-effect inference method. This approach can be found in Tagasovska et al. [29], Khemakhem et al. [11], Xu et al. [30], Strobl and Lasko [28].
B Derivation of Equation (2)
Lemma B.1. For an LSNM model  $X \to Y$  defined in Equation (1), we have  $P_{Y|X}(Y|X) = P_{Z_Y}(\frac{Y - f(X)}{g(X)}) \cdot \frac{1}{g(X)}$ , where  $P_{Z_Y}$  is the noise distribution.
For the data distribution in the  $X\to Y$  direction:
$$
P _ {\rightarrow , P _ {Z}} (X, Y) = P _ {X} (X) \cdot P _ {Y | X} (Y | X) = P _ {X} (X) \cdot P _ {Z _ {Y}} \left(\frac {Y - f (X)}{g (X)}\right) \cdot \frac {1}{g (X)}
$$
Similarly, in the  $X \leftarrow Y$  direction:
$$
P _ {\leftarrow , P _ {Z}} (X, Y) = P _ {Y} (Y) \cdot P _ {X | Y} (X | Y) = P _ {Y} (Y) \cdot P _ {Z _ {X}} (\frac {X - h (Y)}{k (Y)}) \cdot \frac {1}{k (Y)}
$$
Proof for Lemma B.1. For an LSNM model defined in Equation (1), we have  $Z_{Y} = \frac{Y - f(X)}{g(X)}$ . Hence,  $\frac{\partial Z_{Y}}{\partial Y} = \frac{1}{g(X)}$ . Since  $g(X) > 0$ , so  $|\frac{\partial Z_{Y}}{\partial Y}| = |\frac{1}{g(X)}| = \frac{1}{g(X)}$ .
(via change of variables)
$$
\begin{array}{l} P _ {Y \mid X} (Y \mid X) \\ = P _ {Z _ {Y}} \left(Z _ {Y} | X\right) \cdot | \det  \frac {\partial Z _ {Y}}{\partial Y} | \\ = P _ {Z _ {Y}} \left(Z _ {Y}\right) \cdot | \det  \frac {\partial Z _ {Y}}{\partial Y} | \\ = P _ {Z _ {Y}} \left(\frac {Y - f (X)}{g (X)}\right) \cdot \left| \frac {\partial Z _ {Y}}{\partial Y} \right| \\ = P _ {Z _ {Y}} \left(\frac {Y - f (X)}{g (X)}\right) \cdot \frac {1}{g (X)} \\ \end{array}
$$
C Identifiability of LSNMs with correct noise distribution
Strobl and Lasko [28], Immer et al. [9] prove the identifiability of LSNMs, assuming a correctly specified noise distribution. That is, given that the data generating distribution  $(X,Y)$  follows an LSNM in the direction  $X\rightarrow Y$ , the same distribution with equal likelihood cannot be induced by an LSNM in the backward direction  $X\gets Y$ , except in some pathological cases. In terms of our notation, direction identifiability means that if  $(\to ,P_Z)$  is the data generating model, then
$$
P \left(\left[ L _ {\rightarrow , P _ {Z}} (D) - L _ {\leftarrow , P _ {Z}} (D) \right] > 0\right)\rightarrow 1 \text {a s} N \rightarrow \infty
$$
Immer et al. [9] prove the following identifiability result:
Theorem C.1 (Theorem 1 from [9]). For data  $(X,Y)$  that follows an LSNM in both direction  $X\to Y$  and  $X\gets Y$ , i.e.,
$$
\begin{array}{l} Y = f (X) + g (X) \cdot Z _ {Y}, \text {w h e r e} X \perpline Z _ {Y} \\ X = h (Y) + k (Y) \cdot Z _ {X}, \text {w h e r e} Y \perpleftarrow Z _ {X} \\ \end{array}
$$
The following condition must be true:
$$
\begin{array}{l} \left(\log p (y)\right) ^ {\prime \prime} + \frac {g ^ {\prime} (x)}{G (x , y)} \cdot \left(\log p (y)\right) ^ {\prime} + \frac {\partial^ {2}}{\partial y ^ {2}} \cdot \nu_ {X | Y} (x | y) + \frac {g (x)}{G (x , y)} \cdot \frac {\partial^ {2}}{\partial y \partial x} \cdot \nu_ {X | Y} (x | y) \tag {5} \\ + \frac {g ^ {\prime} (x)}{G (x , y)} \cdot \frac {\partial}{\partial y} \cdot \nu_ {X | Y} (x | y) = 0 \\ \end{array}
$$
where  $G(x,y) = g(x)\cdot f'(x) + g'(x)\cdot [y - f(x)]\neq 0$  and  $\nu_{X|Y}(x|y) = \log p_{Z_X}(\frac{x - h(y)}{k(y)}) - \log k(y)$ .
They state that Equation (5) will be false except for "pathological cases". In addition, Khemakhem et al. [11] provide sufficient conditions for LSNMs with Gaussian noise to be identifiable. Our next theorem provides identifiability results for some non-Gaussian noise distributions:
Theorem C.2. Suppose that the true data-generating distribution follows an LSNM model in both  $X \to Y$  and  $X \gets Y$  directions:
1. If the noise distribution is uniform  $(a,b)$ , then both  $g(X)$  and  $k(Y)$  are constant functions.
2. If the noise distribution is continuous Bernoulli  $(\lambda \neq 0.5)^{2}$  or exponential  $(\lambda)$ , then one of the following conditions holds:
$\frac{1}{g(X)}$  and  $\frac{1}{k(Y)}$  are constant functions.
-  $\frac{1}{g(X)}$  and  $\frac{1}{k(Y)}$  are linear functions with the same coefficients on  $X$  and  $Y$ , respectively.
The proof is in Appendix D. Essentially, the theorem shows that for uniform, exponential, and continuousBernoulli noise distributions, the true LSNM model can be identified unless it degenerates to (i) a homoscedastic additive noise model or (ii) a heteroscedastic model with the same linear scale in both directions.
D Identifiability proofs
In this section, we prove Theorem C.2.
If the data  $(x,y)$  follows an LSNM in the forward (i.e. causal) model:
$$
y := f (x) + g (x) \cdot z _ {Y}
$$
where  $z_{Y}$  is the noise term,  $x \perp \perp z_{Y}$ ,  $g(x) > 0$  for all  $x$  on its domain. We assume  $f(\cdot)$  and  $g(\cdot)$  are twice-differentiable on the domain of  $x$ .
If the data  $(x,y)$  follows an LSNM in the backward (i.e. anti-causal) model:
$$
x := h (y) + k (y) \cdot m _ {X}
$$
where  $m_X$  is the noise term,  $y \perp m_X$ ,  $k(y) > 0$  for all  $y$  on its domain. We assume  $h(\cdot)$  and  $k(\cdot)$  are twice-differentiable on the domain of  $y$ .
$z_{Y}$  and  $m_{X}$  follow one of uniform  $(a,b)$ , exponential  $(\lambda)$  or continuous Bernoulli  $(\lambda)$  distribution accordingly.
Proof for uniform  $(a,b)$  noise. For the causal model, according to Lemma B.1, we have  $P_{Y|X}(y|x) = P_{Z_Y}(\frac{y - f(x)}{g(x)})\cdot \frac{1}{g(x)} = P_{Z_Y}(z_Y)\cdot \frac{1}{g(x)} = \frac{1}{b - a}\cdot \frac{1}{g(x)} = \frac{1}{(b - a)\cdot g(x)}$ . Similarly, for the backward model, we have  $P_{X|Y}(x|y) = \frac{1}{(b - a)\cdot k(y)}$ .
The joint likelihood of the observation  $(x,y)$  in the causal model is:
$$
P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = P _ {X} (x) \cdot P _ {Y | X} (y | x) = P _ {X} (x) \cdot \frac {1}{(b - a) \cdot g (x)}
$$
2The case of continuousBernoulli  $(\lambda = 0.5)$  is equivalent to uniform(0,1)).
The joint likelihood of the observation  $(x,y)$  in the backward model is:
$$
P _ {\leftarrow , P _ {M _ {X}}} (x, y) = P _ {Y} (y) \cdot P _ {X | Y} (x | y) = P _ {Y} (y) \cdot \frac {1}{(b - a) \cdot k (y)}
$$
If the data follows both models:
$$
P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = P _ {\leftarrow , P _ {M _ {X}}} (x, y)
$$
$$
P _ {X} (x) \cdot \frac {1}{(b - a) \cdot g (x)} = P _ {Y} (y) \cdot \frac {1}{(b - a) \cdot k (y)}
$$
$$
P _ {X} (x) \cdot \frac {1}{g (x)} = P _ {Y} (y) \cdot \frac {1}{k (y)}
$$
Take the derivative of both sides with respect to  $x$ :
$$
P _ {X} (x) \cdot (\frac {1}{g (x)}) ^ {\prime} + (P _ {X} (x)) ^ {\prime} \cdot \frac {1}{g (x)} = 0
$$
$$
P _ {X} (x) \cdot - 1 \cdot \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} + 0 \cdot \frac {1}{g (x)} = 0
$$
$$
P _ {X} (x) \cdot - 1 \cdot \frac {1}{g (x) ^ {2}} \cdot g (x) ^ {\prime} = 0
$$
$$
P _ {X} (x) > 0
$$
$$
\frac {1}{g (x) ^ {2}} > 0
$$
$$
T h e r e f o r e, g (x) ^ {\prime} = 0
$$
Similarly, if we take the derivative of both sides with respect to  $y$  instead, we have:
$$
k (y) ^ {\prime} = 0
$$
These imply that both  $g(x)$  and  $k(y)$  are constant functions.
Proof for exponential  $(\lambda)$  noise. For the causal model, according to Lemma B.1, we have  $P_{Y|X}(y|x) = P_{Z_Y}(\frac{y - f(x)}{g(x)}) \cdot \frac{1}{g(x)} = \lambda \cdot e^{-\lambda \cdot \frac{y - f(x)}{g(x)}} \cdot \frac{1}{g(x)} = \frac{\lambda}{g(x)} \cdot e^{-\frac{\lambda}{g(x)} \cdot (y - f(x))}$ . Similarly, for the backward model, we have  $P_{X|Y}(x|y) = \frac{\lambda}{k(y)} \cdot e^{-\frac{\lambda}{k(y)} \cdot (x - h(y))}$ .
$$
P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = P _ {X} (x) \cdot P _ {Y | X} (y | x) = P _ {X} (x) \cdot \frac {\lambda}{g (x)} \cdot e ^ {- \frac {\lambda}{g (x)} \cdot (y - f (x))}
$$
$$
\log P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = \log P _ {X} (x) + \log \lambda - \log g (x) - \frac {\lambda}{g (x)} \cdot (y - f (x))
$$
$$
P _ {\leftarrow , P _ {M _ {X}}} (x, y) = P _ {Y} (y) \cdot P _ {X | Y} (x | y) = P _ {Y} (y) \cdot \frac {\lambda}{k (y)} \cdot e ^ {- \frac {\lambda}{k (y)} \cdot (x - h (y))}
$$
$$
\log P _ {\leftarrow , P _ {M _ {X}} \left. \right.} (x, y) = \log P _ {Y} (y) + \log \lambda - \log k (y) - \frac {\lambda}{k (y)} \cdot (x - h (y))
$$
$$
\begin{array}{l} \log P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = \log P _ {\leftarrow , P _ {M _ {X}}} (x, y) \\ \log P _ {X} (x) + \log \lambda - \log g (x) - \frac {\lambda}{g (x)} \cdot (y - f (x)) \\ = \log P _ {Y} (y) + \log \lambda - \log k (y) - \frac {\lambda}{k (y)} \cdot (x - h (y)) \\ \log P _ {X} (x) - \log g (x) - \lambda \cdot \frac {1}{g (x)} \cdot y + \lambda \cdot \frac {1}{g (x)} \cdot f (x) \\ = \log P _ {Y} (y) - \log k (y) - \lambda \cdot \frac {1}{k (y)} \cdot x + \lambda \cdot \frac {1}{k (y)} \cdot h (y) \\ \end{array}
$$
$$
\left(\log P _ {X} (x)\right) ^ {\prime} - \left(\log g (x)\right) ^ {\prime} - \lambda \cdot y \cdot \left(\frac {1}{g (x)}\right) ^ {\prime} + \lambda \cdot \left(\frac {1}{g (x)} \cdot f (x)\right) ^ {\prime} = - \lambda \cdot \frac {1}{k (y)}
$$
Take the derivative of both sides with respect to  $y$ :
$$
\begin{array}{l} - \lambda \cdot \left(\frac {1}{g (x)}\right) ^ {\prime} = - \lambda \cdot \left(\frac {1}{k (y)}\right) ^ {\prime} \\ \frac {\partial \frac {1}{g (x)}}{\partial x} = \frac {\partial \frac {1}{k (y)}}{\partial y} \\ \end{array}
$$
They can be equal only if both sides are constants. Therefore,  $\frac{1}{g(x)}$  and  $\frac{1}{k(y)}$  are both constants or both linear functions with the same coefficient on  $x$  and  $y$ , respectively.
Proof for continuousBernoulli  $(\lambda \neq 0.5)$  noise. Please refer to uniform for continuousBernoulli  $(\lambda = 0.5)$ , which equals to uniform(0,1). For the causal model, according to Lemma B.1, we have  $P_{Y|X}(y|x) = P_{Z_Y}(\frac{y - f(x)}{g(x)}) \cdot \frac{1}{g(x)} = C_\lambda \cdot \lambda^{\frac{y - f(x)}{g(x)}} \cdot (1 - \lambda)^{1 - \frac{y - f(x)}{g(x)}} \cdot \frac{1}{g(x)}$ , where  $C_\lambda$  is the normalizing constant of the continuous Bernoulli distribution. Similarly, for the backward model, we have  $P_{X|Y}(x|y) = C_\lambda \cdot \lambda^{\frac{x - h(y)}{k(y)}} \cdot (1 - \lambda)^{1 - \frac{x - h(y)}{k(y)}} \cdot \frac{1}{k(y)}$ .
$$
\begin{array}{l} P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = P _ {X} (x) \cdot P _ {Y | X} (y | x) = P _ {X} (x) \cdot \frac {1}{g (x)} \cdot C _ {\lambda} \cdot \lambda^ {\frac {y - f (x)}{g (x)}} \cdot (1 - \lambda) ^ {1 - \frac {y - f (x)}{g (x)}} \\ \log P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = \log P _ {X} (x) - \log g (x) + \log C _ {\lambda} + \log \lambda^ {\frac {y - f (x)}{g (x)}} + \log (1 - \lambda) ^ {1 - \frac {y - f (x)}{g (x)}} \\ \end{array}
$$
$$
P _ {\leftarrow , P _ {M _ {X}}} (x, y) = P _ {Y} (y) \cdot P _ {X | Y} (x | y) = P _ {Y} (y) \cdot \frac {1}{k (y)} \cdot C _ {\lambda} \cdot \lambda^ {\frac {x - h (y)}{k (y)}} \cdot (1 - \lambda) ^ {1 - \frac {x - h (y)}{k (y)}}
$$
$$
\log P _ {\leftarrow , P _ {M _ {X}} \left. \right.} (x, y) = \log P _ {Y} (y) - \log k (y) + \log C _ {\lambda} + \log \lambda^ {\frac {x - h (y)}{k (y)}} + \log (1 - \lambda) ^ {1 - \frac {x - h (y)}{k (y)}}
$$
$$
\log P _ {\rightarrow , P _ {Z _ {Y}}} (x, y) = \log P _ {\leftarrow , P _ {M _ {X}}} (x, y)
$$
$$
\log P _ {X} (x) - \log g (x) + \log C _ {\lambda} + \log \lambda^ {\frac {y - f (x)}{g (x)}} + \log (1 - \lambda) ^ {1 - \frac {y - f (x)}{g (x)}}
$$