Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing
Authors: Xiangyu Sun
Oliver Schulte
Venue: NeurIPS 2023 poster (2023)
Keywords: Causal Discovery
Cause-Effect Inference
Location-Scale Noise Models
TL;DR: We propose a new independence-based cause-effect inference method that is robust in the settings where likelihood-based methods fail.

Abstract: A fundamental problem of causal discovery is cause-effect inference, to learn the correct causal direction between two random variables. Significant progress has been made through modelling the effect as a function of its cause and a noise term, which allows us to leverage assumptions about the generating function class. The recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified. However, through an extensive empirical evaluation, we demonstrate that the accuracy deteriorates sharply when the form of the noise distribution is misspecified by the user. Our analysis shows that the failure occurs mainly when the conditional variance in the anti-causal direction is smaller than that in the causal direction. As an alternative, we find that causal model selection through residual independence testing is much more robust to noise misspecification and misleading conditional variance.

Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing
Xiangyu Sun Simon Fraser University x xiangyu_suna caatyyussn@sfuucco
Oliver Schulte Simon Fraser University oschulte@cs, sfu. ca
Abstract
A fundamental problem of causal discovery is cause-effect inference, learning the correct causal direction between two random variables. Significant progress has been made through modelling the effect function of its cause and noise term, which allows us to leverage assumptions about the generating function class. The recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified. However, through an extensive empirical evaluation, we demonstrate that the accuracy deteriorates sharply when the form of the noise distribution is misspecified by the user. Our analysis shows that the failure occurs mainly when the conditional variance in the anti-causal direction is smaller than that in the causal direction. As an alternative, we find that causal model selection through residual independence testing is much more robust to noise misspecification and misleading conditional variance.
Introduction
Distinguishing cause and effect is fundamental problem in many disciplines, such biology, healthcare and finance [31, 88 13). Randomized controlled trials (RCTs) are the gold standard for finding causal relationships. However, it may be unethical, expensive or even impossible to perform RCTs in real-world domains 200 [8] Causal discovery algorithms aim to find causal relationships given observational data alone. Traditional causal discovery algorithms can only identify causal relationships up to Markov equivalence classes (MECs) (26 TO 3. To break the symmetry in MEC, additional assumptions are needed 20 22, such as the type of functional dependence of effect on cause. Structural causal models (SCMs) specify functional class for the causal relations in the data [7 220 In this work, we focus on one particular type of SCM called location-scale noise
models (LSNMs) 29
30 28 9
Y f(X) + 9(X) Zy
where X is the cause, Y the effect, written X Y. and Zy is a latent noise variable independent of X (i.e., X IL Zy). The functions f and are twice-differentiable on the domain of X, and is strictly positive on the domain of X. LSNMs generalize the widely studied additive noise models (ANMs), where g(x) = for all x. ANMs assume constant conditional variance V[Y [X] for different values of X, which can limit their usefulness when dealing with heteroscedastic noise in real-world data. LSNMs allow for heteroscedasticity. meaning that V[ can vary depending on the value of X. our experiments on real-world data, methods assuming an LSNM outperform those assuming ANM.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We follow previous work on cause-effect inference and focus on the bivariate setting 129 There are known methods for leveraging bivariate cause-effect inference the multivariate setting (see Appendix A). Two major methods for selecting causal SCM direction in cause-effect inference are maximum likelihood (MI.) and independence testing (IT) of residuals vs. the (putative) cause [14]. Both have been recently shown good accuracy with LSNMs [1 especially when the f and functions are estimated by neural networks. Immer et al. O note, however, that ML can be less robust than IT, in the sense that accuracy deteriorates when the noise distribution is not Gaussian.
11 30
In this paper we investigate the robustness of ML vs. IT for LSNMs. Our analysis shows that ML cause-effect inference performs poorly when two factors coincide: (1) Noise Misspecification: the ML method assumes a different form of the noise distribution from the true one. (2) Misleading Conditional Variances (CVs): V[ >> V[X in the data generated by causal direction X x Y. For example, the experiment on synthetic datasets shown in Table beloow, (i) changing the true noise distribution from Gaussian to uniform and (ii) manipulating W V while keeping other settings equal, can decrease the rate of identifying the true causal direction from 100% to 10%. In contrast, IT methods maintain a perfect 100% accuracy. The difference occurs with and without data standardization, and with increasing sample sizes.
Both conditions (1) and (2) often hold in practice. For real-world domains, assumptions about the noise distribution can be hard to determine or verify. It is also common to have misleading CVs in real-world datasets. For example, in the Tübingen Couse-Effect Pairs benchmark (14), about 40% of the real-world datasets exhibit a misleading CV (see Table ii in the appendix).
We make the following contributions to understanding location-scale noise models (LSNMs):
Describe experiments and thcoretical analysis to show that ML methods succeed when the form the noise distribution is known. In particular, ML methods are then robust to incorrectly specifying noise variance parameter.
Demonstrate empirically that ML methods often fail when the form of the noise distribution is misspecified and CV is misleading, and analyze why.
Introduce a new IT method based affine flow LSNM model.
Demonstrate, both theoretically and empirically, that our IT method is robust to noise misspecification and misleading CVs.
The paper is structured as follows. We discuss related works and preliminaries in Section 2 and Sec- tion 3 respectively. Section Macioon//ee examines when and why ML methods fail. Section 5 demonstrates the robustness of the IT method. Experiments with 580 synthetic and 99 real-world datasets are given in Section 7
The code and scripts reproduce all the results are given online
2 Related works
Causal Discovery. Causal discovery methods have been widely studied in machine learning 26 10 3 Assuming causal sufficiency and faithfulness, they find causal structure up MEC. To identify causal relations within a MEC, additional assumptions are needed 20 22). SCMs exploit constraints that result from assumptions about the functional dependency of effects on causes. Functional dependencies are often studied in the fundamental case of two variables X and Y, the simplest MEC. Mooij al. [14 provide an extensive overview and evaluation of different cause-effect methods in the ANM context. We follow this line of work for LSNMss with possibly misspecified models.
Structural Causal Models. Assuming linear non non- Gau-saasiinn acyelic model (LiNGAM) 24 25 the causal direction was proved to be identifiable. The key idea in DirectLiNGAM 25) is that in the true causal direction X Y, the model residuals independent of X. We refer to methods derived from the Directl iNGAM approach as independence testing (IT) methods. The more general ANMs 7 19 allow for nonlinear cause-effect relationships and generally identifiable, except for some special cases. There are other identifiable SCMs, such as causal additive models (CAM) 2 post-nonlinear models [32] and Poisson generalized linear models [16]
LSNM Identifiability. There are several identifiability results for the causal direction in LSNMs. Xu et al. 30 prove identifiability for LSNMs with linear causal dependencies. Khemakhem et al.

---

Noise Misspecification?
Yes
Misleading CVs?
Misleading CVs?
Yes
No
Yes
No
LSNM or ANM?
LSNM or ANM?
LSNM or ANM?
LSNM or ANM?
LSNM
ANM
LSNM
ANM
LSNM
ANM
LSNM
ANM
Low
Low
High
High
High
Low
High
High
Figure 1: Summary of the accuracy results for MI. methods for LSNMs and ANMs with respect to noise misspecification and misleading CVs.
show that nonlinear LSNMs are identifiable with Gaussian noise. Strobl and Lasko (88] Immer et al. ( prove LSNMs are identifiable except in some pathological cases (see Appendix
Effse-Effect Inference in LSNMs. The generic model selection blueprint is to fit two LSNM models for each direction X + Y and X <- Y and select the direction with higher model score. HECI 30 bins putative cause values and selects a direction based on the Bayesian information criterion. BQCD 129 uses nonparamatric quantile regression to approximate minimum description length to select the direction. GRCI (28) finds the direction based on mutual information between the putative cause and the model residuals. LOCI [9] models the conditional distribution of effect given cause with Gaussian natural parameters. Then, it chooses the direction based on either likelihood (LOCI-M) independence (LOCI-H). CAREFL-M [11] fits affine flow models and scores them by likelihood. CAREFL-M is more general than LOCII since LOCI uses a fixed Gaussian prior, whereas CAREFL-M can utilize different prior distributions. DECI generalizes CAREFL-M multivariate cases for ANMs. Unlike our work, none of these papers provide theoretical analysis of noise misspecification and mislcading CVs LSNMs.
Noise Misspecification. Schultheiss and Bühlmann present a theoretical analysis of model misspecification pitfalls when the data are generated by ANM. Their results focus model selection with Gaussian likelihood scores, which use Gaussian noise distributions. They suggest that IT-based methods may be more robust model misspecification briefly discuss the complication of fitting data generated by ground-truth ANM model with an LSNNM model. Since the backward model of ANM often takes the form of an LSNM, fitting data generated by ground -ruth ANM with an LSNM model may result in an incorrect causal direction. While their analysis examines ANM data, the ground truth model in our work an LSNM. The g() scale term in Equation ① which is absent in ANMs, plays a critical role in LSNMs.
ANMs. The authors also
Conditional Variance and Causality. To the best of our knowledge, the problem of misleading CVs has not been identified nor analyzed in previous work for LSNMs. Prior studies that have focused on CVs were based on ANMs, and stated assumptions about how CVs relate to causal structure, to be leveraged in cause-effect inference [5]. We do not make assumptions about CVs, but analyze them to understand why noise misspecification impairs cause-effect inference in LSNNss.
3 Preliminaries
In this section, we define the cause-effect inference problem and review the LSNM data likelihood.
3.1 Problem definition: cause-effect inference
Cause-effect inference takes as input dataset over two random variables X, with observa-
tional data pairs (X, Y) {x11 11), (I2, 12)..... ... (I,, DN)} generated by ground-truth LSNM in Equation (T) The binary output decision indicates whether X causes Y (X Y) or Y causes X
(X + Y). We assume no latent confounders, selection bias or feedback cycles (14 29 9 30).

---

Table 1: Accuracy over 10 datasets generated by SCM LSNM-sine-tanh (definition in Appendix M
with = 10, 000 samples. The task a binary decision whether X causes Y or Y causes X.
denotes the ground-truth cause and Y denotes the ground-truth effect. We rewrite Equation
as
Y [(X) + g(X) Zy, where a is scale factor to alter the CVs. CVs are computed by binning the putative cause. We used Gaussian(O,, 1) as model prior for both CAREFL and LOCI. The suffix
-M denotes a ML method. The suffix -H denotes the corresponding IT method (see Section All
the datasets are standardized have mean and variance
True Noise
Gaussian (0, 1)
Uniform (1,
a
0.1
0.5
10
0.1
0.5
10
V[
0.166
0.615
0.834
0.990
0.997
0.044
0.404
0.673
0.975
0.994
V
0.455
0.709
0.793
0.821
0.817
0.047
0.375
0.566
0.681
0.677
Percentage of Datasets With
30%
50%
70%
100%
100%
30%
90%
100%
100%
100%
Misleading CVs
CAREFL-M
1.0
1.0
1.0
1.0
1.0
0.7
0.6
0.7
0.1
0.1
LOCI-M
1.0
1.0
1.0
1.0
1.0
0.7
0.6
0.7
0.1
0.1
CAREFL-H
1.0
1.0
1.0
1.0
1.0
0.9
1.0
1.0
1.0
1.0
LOCI-H
1.0
1.0
1.0
1.0
1.0
0.7
1.0
1.0
1.0
1.0
3.2
Definition of maximum likelihood for LSNMs
An LSNM model for two variables (X, Y) is a pair (-x, P2). The + represents the direction X - Y with parameter space f fe.g 9 Px Px.5. The represents the direction X Y with parameter space h = hg- k = Kew. Py = Py, For notational simplicity, we treat the model model parameters and omit reference to their parameterizations. For example,
functions directly we write f = So for a function / that is implemented by neural network with weights 0. We refer Pz as the model prior noise distribution.
parameterized LSNM model defines a data distribution as follows 9 (Appendix) B):
Y f(x)
1
(X, = Px((X)
Pzr
g(X)
g(X)
(2)
X x-h(),
P.P.((X.YYYAKBBB))))) Pzx
k(Y)
k(Y)
For conciseness, we use shorter notation P-,pz (X,X) Y) and P., (X, Y) in later sections. The ML method computes the MI. parameter estimates and uses them to score the (or <-) model:
f.g. Px arg max
II P- P (1,, M. Px)
(3)
Sx i=1
I (D) = II P-.ry (m. Px)
4 Non-identifiability of LSNMs with noise misspecification
In this section, we show that ML methods fail under noise misspecification and misleading CVs, and analyse why. Figure F summarizes the diverse settings considered below.
Existing identifiability results for LSNM ML methods (Appendix C require knowing the ground-truth noise distribution. We conducted simple experiment to show that with noise misspecification. MIL model selection can fail badly even on large sample sizes. Table shows results for CAREFL-M [TT) and LOCI-M [9] with model prior distribution Gaussian (0, 1). For the left half of the table, the data

---

was generated with correctly specified Gaussian(O, 1) noise. In this case, both CAREFL-M and LOCI-M work well, and increasing CV in the causal direction does not affect their accuracy. For the right half of the table, the data was generated with misspecified uniform( (-1,1) noise. With noise misspecification, the accuracy of both CAREFL.-M and L.CC-MM decreases to 70%. With both noise misspecification and misleading CVs, their accuracy becomes even lower. For example, in the last column when V 0.994 and V[X|+| = 0.677, they give an accuracy of just 10%.
The following results help explain why ML methods fail under noise misspecification and misleading CVs.
Lemma 4.1. For an LSNM model Y == f(X) + gX) Zy where Zy is the noise with mean i and variance a² + 0, there exists a likelihhoodeeqiivlennl LSNM Model Y == ['(X) + g' (X) Zy such
that Z - (Y|x: = P- XY|; ,'), and Zzy [V] V zz [
The proof is in Appendix E Lemma 4.1 shows that for any (non-deterministicc LSNM model there is an equivalent standardized LSNM model with V [Zy] = 11
Theorem 4.2. a Olog P-. (YIX) aV X] < for LSNM models.
The proof is in Appendix F Theorem 4.2 shows that the model likelihood P-,Ps, (Y and the conditional variance V [Y
are negatively related (similarly for P-,Pz 2 (X (XlY) and V [X(Y]). Corollary 4.3, Let -> denote the causal model and +- denote the anti-causal model. When V [Y increases and V [X Y decreases in the data, the log-likelihood model difference log P-, ()-Y)-
log P2 (X, Y) decreases.
The proof is in Appendix G Under the identifiability assumption [1 that ML is consistent under correct model specification, Lemma 4.1 also implies that ML remains consistent under misspecifying the variance of the noise distribution, as long as the form of the prior distribution matches that of the noise distribution. Table 4a in the appendix demonstrates this relationship empirically.
The overall relationship between likelihood and CV with respect to noise specification in LSNMs is as follows, illustrated in Figure 2 with actual datasets:
1. (Theorem CV and likelihood are negatively related (Figures 2a and
2b
with either
correctly incorrectly specified noise distribution form.
2. (Identifiabilitt) When the form of the noise distribution is correctly specified, the causal model always has higher likelihood (Figure
3. (Corollary 4.3) When the form of the noise distribution is misspecified, the anti-causal model can have a higher likelihood under misleading CVs (Figure 2b
ANM vs. LSNM. The relationship (2) is in general different for ANMs, when the noise variance misleading (see Figure [ and Appendix PP). Proposition H.I shows that the standardization lemma Lemma 4.1 does not apply to ANMs (confirmed by empirical results in Table 4b in the appendix).
5 Robustness of independence testing
This section describes IT methods for cause-effect learning in LSNMs, including a new IT method based on affine flows. Theoretical results below explain why IT methods are robust noise misspeci- fication and misleading CVs in LSNMs.
5.1 The independence testing method
Inspired by the breakthrough DirectLiNGAM approach 25). independence testing has been used
in existing methods for various SCMs 19 28 9. Like ML methods (Equation (4), IT methods
fit the model parameters in both directions, typically maximizing the data likelihood (Equation (3). The difference is in the model selection step, where IT methods select the direction with the highest degree of independence between the fitted model residuals and the putative cause.
Algorithm T is the pseudo-code for our IT method CAREFL-H. We the functions I and in Equation with the affine flow estimator T from CAREFL-M [I]] implemented using neural
5

---

2.00
0.8
1.75
0.5
1.00
0.0
0.25
-0.1
0.00
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.4
0.6 0.8 1.0 1.2 1.4 1.6
Mean CV. Causal Anti-Causal
Mean CV: Causal / Ant- Causal
(a) Log-likelihood difference under correct
(b) Log-likelihood difference under noise
noise specification: Caussian(O, 1) noise
misspecification: uniform( -1, 1) noise
HSC Valae -11--U
0.000
Aascoali Caual AIHOQU
-0.005
-0,0005
-0,0010
-0.015
-0.0015
-0.0020
-0.0025
0.025
0.0030
0.2 0.4 0.6 0.8 1.0 1.2
1.4
0.4 0.6 0.8 1.0 1.2 1.4 1.6 Mean CV: Causal Anti Causal
Mean CV: Causal / Anti Causal
(c) HSIC difference under correct noise spec-
(d) HSTC difference under noise misspecifi- cation: uniform(-1,1) 1) noise
ification: Gaussian(O, 1) noise
Figure 2: Visualization of Table First row 2bx: ML methods. Second row 12 2dh: IT methods. Y-axis < 0.0: A ML method returns the incorrett anti-causal direction. Y-axis > 0.0: An IT method returns the incorrect anti-causal direction. ML methods may fail under misspecification and when X-axis >> / ). IT methods are more robust 2d)
misleading CVs 2b
networks. For details on CAREFL-M and learning the flow transformation T see Appendix This combination of affine flow model with IT appears be Another IT method is to test the independence of both residuals [6] (Appendix[). We found that this performs similarly to CAREFL-H and therefore report results only for the more common DireetLiNGAM-styl method.
As in previous work [4,, we use Hilbert-Schmidt independence criterion (HSIC) [5] to measure (in) dinedeeene throughout the paper. HSIC measures the squared distance between the joint proba- bility the two variables and the product their marginals embedded in the reproducible kernel Hilbert space. We have HSIC(U, V) 0 if and only if U IL V.
5.2 Theoretical comparison between IT and ML under noise misspecification in LSNMs
The intuition for why independence testing works that if LSNM training consistent, testing the independence of the putative cause and the residual will indicate the correct causal model. The next theorem provides a formal statement of this intuition.
For data pairs (X, Y) generated by an LSNM model X + let f(X) denote the conditional mean estimator and g(X) denote the conditional standard deviation estimator. Under the consistency conditions that fXX = E and g(X) = STD(Y the reconstructed noise is independent of the cause, i.e. Zy 11. X, even under noise misspecification.
Theorem 5.1.
Proof. According to Equation (
f(x) = X| = (X) + x Zy = -x/) f(X) + 9(X) E[
O(X) STD(Y X STD(S(D))) + (X)) X [x( STDD g(X) STD STD
Thus, Zy y--(x)
Zy-E(Zy] where E and S70|2r]
7(X)
STD[Zy] > are constants. Therefore, Zy and Zy are identical up shift and scale. Since Zy X, therefore, Zy 11 X.
Note that the proof does not involve the specification of the noise prior distribution Pzy, which shows that under the consistency condition, the conclusion Zy holds whether the noise prior distribution is misspecified or not.

---

Algorithm CAREFL-H
1: Input: data pairs D = (X,Y), Y), the flow estimator T of CAREFL-M with prior Pz. and an
HSIC estimator
2: Output: estimated causal direction dir
3: Split D into training set D Derasa = (X troin; Ytrain) and testing set Drest == (Xtest, Ytsst)
4: Optimize T 0,0.2 (Drrain; Pz) in X + + Y direction via ML to estimate faadd g
5: Compute the residual B = Yun-jor j(((na) ((Xmn))
6: Optimize T 80 2.00 23 (D Draini Pz) in X + direction via ML to estimate and
7: Compute the residual Zx Xec Xiue-i(Yua))
R(Yiew))
8: if HSIC(X test, Zx) HHSTCCV test, 2x) then
9: dir := X
10:
else if HSIC teats Zx) HSIC((Yests 2x) then
11:
dir X
12: else
13:
dir = no conclusion
14:
end if
In contrast, the likelihood of causal model depends on the specification of the noise prior distribution Pzy (Equation (2)). This explains why IT methods are more robust than ML methods under noise misspecification in LSNMs.
In the context of ANMs, Mooij et 14 provide another sufficient condition for Zy IL X called suitability. In Appendix K we show how the suitability concept can be adapted for LSNMs to provide another argument for the robustness of methods.
Limitation of IT-based methods
Limitations of IT- based methods include the following: 1) When the noise prior distribution is correctly specified, ML-based methods can utilize this information to be more sample efficient. Conversely, since I'T-aaeed methods are less sensitive to the noise prior distribution, they require larger sample size to infer causal direction accurately. 2) IT-based methods have higher computational cost, especially with nonlinear independence testing. 3) IT-based methods require high-capacity function estimators to model the unknown functions f and for good noise reconstruction.
Despite these complications, we advocate the use of ITT-based methods over ML-based methods under noise misspecification and misleading CVs in LSNMs, because they are more reliable in such settings.
Experiments
On synthetic datasets, we find that across different hyperparameter choices the IT method (CAREFL- H) produces much higher accuracy than the ML method (CAREFL-M) in the difficult settings with noise misspecification and misleading CVs, and produces comparable accuracy the easier settings without noise misspecification misleading CVs. On real-world data where the ground-truth noise distribution is unknown, the IT method also more robust across different hyperparameter choices.
For all experiments, we start with the same default hyperparameter values for both CAREFL-M and CAREFL-H and alter one value time. The default hyperparameter values are those specified in CAREFL-M [11 for the Tübingen Cause-Effect Pairs benchmark [14]. Please see Appendix L for more details on default and alternative hyperparameter values. Previous work [4] reported that ML methods perform better with data splitting (split data into training set for model fitting and testing set for model selection) and IT methods perform better with data recycling (the same data is used for both model fitting and selection). Therefore, we use both splitting methods: (i) CAREFL((.8)) 80% as training and 20% as testing. (ii) CAREFL training testing 100%. The training procedure not supervised: method accesses the ground-truth direction.
7

---

Accuracy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
10
20
10
500
750
1000
2000
0.0
0.0001
0.001
0.1
laplace
gaussian
Number Hidden
Number
Number Epochs
12-Penalty
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 3: Weighted accuracy over 99 datasets from Tübingen Cause-Effect Pairs benchmark.
Table 2:
Accuracy of methods the SIM and Tübingen Cause-Effect Pairs benchmarks. For methods other than CAREFL-M and CAREFL-H, we use the best results reported in Immer et al. 99
(a) Aceuracy with STM benchmarks
LOCI-M
LOCI-H
GRCI
BQCD
HECI
CAM
RESIT
CAREFL-M
CAREFL-H
SIM
0.52
0.79
0.77
0.62
0.49
0.57
0.77
0.55
0.80
SIM-c
0.50
0.83
0.77
0.72
0.55
0.60
0.82
0.58
0.85
SIM-In
0.79
0.73
0.77
0.80
0.65
0.87
0.87
0.84
0.83
SIM-G
0.78
0.81
0.70
0.64
0.56
0.81
0.78
0.82
0.79
(b) Weighted accuracy with Tübingen Cause Efffct Pairs benchmark
LOCI-M
LOCI-H
GRCI
BQCD
HECI
CAM
RESIT
CAREFL-M CAREFL-H
Tübingen Cause-Effect Pairs
0.57
0.60
0.82
0.77
0.71
0.58
0.57
0.73
0.82
We use a consistent HSIC estimator with Gaussian kernels [1] A summary of experimental datasets provided in Appendix Table 7 All the datasets standardized have mean 0 and variance 1. The datasets in Section 77. are generated by LSNMs and the datasets in Sections 7.2 and 773 are not. Consequently, the results demonstrate that the proposed method performs well not only when the ground-truth SCMs are strictly LSNMs, but also when the ground-truth SCMs are unknown or not strictly LSNMs.
7.1 Results on synthetic LSNM data
See Appendix M for the definition of the ground-truth LSNM SCMs and details on how synthetic datasets are generated from them. The sample sizes in each synthetic dataset 500 5,000. As shown in Appendix Table most synthetic datasets generated by such SCMs have misleading CVs. Based on the analysis of Section 4 accurate given a correct specification on the form of noise distribution, with or without misleading CVs. (ii) With noise misspecificatioon and mild misleading CVs. the accuracy of CAREFL-M should be reduced. (iii) With noise misspecification and severe misleading CVs, the accuracy should be very low, often below 50%. Overall, the results from the experiments confirm our hypotheses.
we formulate the following hypotheses. (i) CAREFL-M should be
7.1.1 Noise misspecification
We evaluate CAREFL-M and CAREFL-H against data generated with uniform 1), BontinuousBerroolli
or beta(0.5, 0.5) noise, covered by our identifiabil- ity Theorem C.2 (except (et5, 0.5)). Khemakhem al. [II] empirically find that CAREFL-M with Laplace model prior is robust with similar ground- ground-truth noise distributions such as Gaussian and Student's We show that may fail remarkably with dissimilar distribution.
crponential 1),
summarize findings from the 336 settings here; the detailed results are given in Appendix Figures 6 to 17 In 289 settings (86.01% %), both CAREFL-M(0.8) and CAREFL-M(1.0) select the correct causal

---

direction with less than 50% random accuracy. Furthermore, in 1 10 settings (32.74%) both CAREFL- M(0.8) and CAREFL-M(1.0 1.0) fail catastrophically with an accuracy of 0%. These experiments also show that the accuracy of CAREFI -MM often decreases as N increases. In contrast, CAREFL-HH(1.)) achieves better accuracy than CAREFL--M in 333 settings (99.11%) The accuracy of CAREFL.- H(1.0) goes below 50% in only 6 settings (1.79%). The results demonstrate the robustness of the IT method under noise misspecification and misleading CVs, across different hyperparameter choices.
Appendix Table 7 and Appendix Figure 17 show that with severe misleading CVs, the accuracy of CAREFL-M close to 0%. This is much lower than the corresponding cases with mild misleading
CVs in Appendix Figures 9 and 13
7.1.2 Correct specification on the form of the noise distribution
These experiments show that CAREFL-H is comparable with CAREFL-M under correct form of noise specification, with or without misleading CVs, especially on larger datasets, as long as the affine model capacity is sufficient. We evaluate CAREFL-M and CAREFL-H against data generated with Gaussian (0, 1) and Laplace(O, 1) noise. The detailed results are in Appendix Figures 18 to 23 We find that CAREFL-M is more sample efficient than CAREFL-H when the model prior matches the data, which is a general pattern for MI vs. IT methods 23 Consistent with the suitability results in Appendix [ the accuracy of CAREFL-H improves with more data. For example, with N = 500, there are 49 out of 84 settings (58.33%) where CAREFL-M outperforms CAREFL-H((OO) However, with N = 5, 000, CAREFL-H(1.0) achieves similar accuracy as CAREFL-M on all datasets (except LSNM-sigmoidssigooi with Laplace (0, 1) noise.) In addition, CAREFL-H(1.0 may underperform CAREFL-M when the number of hidden neurons, sub-flows or training epochs is low. An IT method requires more model capacity to fit the LSNM functions and produce good reconstruction of the noise.
7.2 Results on synthetic benchmarks
Similar to Tagasovska al. (29, Immer et al. 19, we compare CAREFL-M and CAREFL-H against the SIM benchmark suite [14] SIM comprises 4 sub- sub-benchmarks: default (SIM), with one confounder (SIM-c), low noise levels (SIM-In) and Gaussian noise (SIM-G). In this benchmark, most datasets do not have misleading CVs (Appendix Table which favors ML methods. Each sub-benchmark 1000 data pairs. As shown in Appendix Figure 24
contains 100 datasets and each dataset has CAREFL-M and CAREFL-H(1 .0) achieve similar accuracy on SIM-In and SIM-G across different hyperparameter choices. For SIM and SIM-c, CAREFL-H, especially CAREFL-H(1.0) outperforms CAREFL-M by 20%-30% in all settings. The accuracy of CAREFL-M is only about random guess (40%-60%) on SIM and SIM-c.
Table 2a compares CAREFL-H with SOTA methods. For each CAREFL method, we report the best accuracy obtained in Appendix Figure 24 without further tuning. CAREFL-H achieves the best accuracy on SIM and SIM-c, and achieves competitive accuracy on SIM-In and SIM-G.
7.3 Results on real-world benchmarks: Tübingen Cause-Effect Pairs
The Tübingen Cause- Efffect Pairs benchmark 14 is commonly used to evaluate cause-effect inference
To be consistent with previous work (29. 28 [ we exclude 6 multivariate and 3 discrete datasets (#47, #52-#55, #70, #71, #105, #107) and utilize the remaining 99 bivariate datasets. As recommended by Mooij et al. [14. we report weighted accuracy. 40% of datasets in the benchmark feature misleading CVs. CAREFL-H(1.O) 110) outperforms CAREFL-M in all configurations by large margins (7%-30%); see Figure 3 We also compare CAREFL-H with SOTA methods (see Appendix N for hyperparameters). Table 2b shows that CAREFL-H achieves the SOTA accuracy (82%) and is 9% more accurate than CAREFL-M [TT.
algorithms M 30 9
Furthermore, LSNM methods (i.e. LOCI, GRCI, BQCD, HECI, CAREFL) perform better than ANM methods (i.e. CAM, RESIT). The reason is that LSNNM a weaker assumption than ANM and allows for heteroscedastic noise.

---

8 Conclusion and future work
We identified failure mode of maximun-likclihood (ML) methods for cause-effect inference in location-scale noise models (LSNMs). Our analysis shows that the failure occurs when the form of the noise distribution misspecified and conditional variances are misleading (i.e., higher in the causal direction). Selecting causal models by independence tests (IT) is robust even in this difficult setting. Extensive empirical evaluation compared the ML method and new IT method based on affine flows, on both synthetic and real-world datasets. The IT flow method achieves better accuracy under noise misspecification and misleading CVs, with robust performance across different hyperparameter choices. Future directions include improving the sample efficiency of IT methods, and improving the robustness of ML methods by learning the noise distribution instead of using a fixed prior.
9
Acknowledgments
This research was supported by a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC).
References
[1] Patrick Blöbaum, Dominik Janzing, Takashi Washio, Shohei Shimizu, and Bernhard Schölkopf. Cause-effect inference by comparing regression errors. In International Conference on Artificial Intelligence and Statistics, pages 900-909. PMLR, 2018.
[2] Peter Bühlmann, Jonas Peters, and Jan Emnest. Cam: Causal additive models, high sigh-dimensional order search and penalized regression. The Annals of Statistics, 42(6):2526, 2014.
[3] Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics, pages 294-321, 2012.
[4] Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.02/995 2022.
[5] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmide norms. In International conference on algorithmic learning theory, pages 63-77. Springer, 2005.
[6] Yue He, Peng Cui, Zheyan Shen, Renzhe Xu, Furui Liu, and Yong Jiang. Daring: Differentiable causal discovery with residual independence. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 596-605, 2021.
[7] Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21, 2008.
[8] Biwei Huang. Diagnosis of autism spectrum disorder by causal influence strength learned from resting- state fmri data. Neural Engineering Techniques for Autism Spectrum Disorder, pages 237-267. Elsevier, 2021.
[9] Alexander Immer, Christoph Schultheiss, Julia E Vogt, Bernhard Schölkopf, Peter Bühlmann. and Alexander Marx. On the identifiability and estimation of causal location-scale noise models. In International Conference Machine Learning, pages 14332. PMLR, 2023.
[10] Markus Kalisch and Peter Bühlman. Estimating high-dimensional directed acyclic graphs with the pc-algorithm. Journal of Machine Learning Research, 8(3), 2007.
[11] Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International conference on artificial intelligence and statistics, pages 3520-3528. PMLR, 2021.
10

---

[12] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[13] Mehrdad Mansouri, Sahand Khakabimamaghani, Leonid Chindelevitch, and Martin Ester. Aristotle: stratified causal discovery for omics data. BMC bioinformatics, 23(1):1-18, 2022.
[14] Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1):1 103-1204, 2016.
[15] Gunwoong Park. Identifiability of additive noise models using conditional variances. The Journal of Machine Learning Research, 21(1):2896 2929, 2020.
[16] Gunwoong Park and Sion Park. High-dimensional poisson structural equation model learning via Iell_l-reullarieed regression. Mach. Learn. Res., 20:95-1, 2019.
[17] Judea Pearl. Causality, Cambridge university press, 2009,
[18] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[19] Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15:2009-2053, 2014.
[20] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- tions and learning algorithms. The MIT Press, 2017.
[21] Niklas Pfister, Peter Bühlmann, Bernhard Schölkopf, and Jonas Peters. Kernel-based tests for joint independence. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 80(1):5-31, 2018.
[22] Bernhard Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The Works of Judea Pearl, pages 765-804. 2022.
[23] Christoph Schultheiss and Peter Bühlmann. On the pitfalls of gaussian likelihood scoring for causal discovery. Journal of Causal Inference, 11(1):0022066,, 2023.
[24] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-Gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
[25] Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik Hoyer, Kenneth Bollen, and Patrik Hoyer. DirectLiNGAM: A direct method for learning linear non-Gaussian structural equation model. Journal of Machine Learning Research-IMLR, 12(Ap)): 201
[26] Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 62):62272, 1991.
[27] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
[28] Eric V Strobl and Thomas Lasko. Identifying patient-specific root causes with the het- eroscedastic noise model. Journal of Computational Science, 72:102099, 2023.
[29] Natasa Tagasovska, Valérie effect using quantiles: Bivariate quantile causal discovery. In International Conference Machine Learning, pages 1-333.... PMLR, 2020.
and Thibault Vatter. Distinguishing cause from
[30] Sascha Xu, Osman A Mian, Alexander Marx, and Jilles Vreeken. Inferring cause and effect in the presence of heteroscedastie noise. International Conference on Machine Learning, pages 24615-24630. PMLR. 2022.

---

[31] Kun Zhang and Lai-Wan Chan. Extensions of ICA for causality discovery in the hong kong stock market. In International Conference on Neural Information Processing, pages 400-409. Springer, 2006.
[32] Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv: 1205:2205.2599 2012.
[33] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems, 31, 2018.
[34] Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametric dags. In International Conference Artificial Intelligence and Statistics, pages 114-3425. PMLR, 2020.
12

---

A Extension to multivariate settings
To extend bivariate cause-effect inference method to multivariate, the most common approach is to first use the PC algorithm [27] (or other algorithms) to learn a completed partially directed acyclic graph (CPDAG) that may contain undirected edges. Then, orient the undirected edges using the cause-effect inference method. This approach can be found in Tagasovska et al. [29], Khemakhem al. [T. Xu al. [30), Strobl and Lasko [28]
B Derivation of Equation 2
Lemma B.1. For an LSNM model X - Y defined in Equation Pzy ( (-((X)) a(X) s(Xy where Pzy is the noise distribution.
we have Py)x(YIx)
For the data distribution in the X + Y direction:
(
P-,ps (X,Y) = Px(X) Prxx = Px(X) Par
g(X) g(X)
Similarly, in the X t- Y direction:
X X-1(Y)
Pe,P2 = -x)) Pr(Y) Pax ( k(Y)
k(Y)
Proof for Lemma B.1
For an LSNM model defined in Equation
we have Zy Y-F(X), Hence,
9(X)
BZy
Since ((X) > 0, so x = |aXx xx
0Y
a(X).
Pr|x(VIx)
dZy =P2y (2x| det
0Y
(via change of variables)
8By
-Pay det
OY
Y f(X)
8Zy
-Pay
g(X)
0Y
-x())
1
=Pay g(X)
9(X)
C Identifiability of LSNMs with correct noise distribution
Strobl and Lasko 28 Immer et al. prove the identifiability of L.NMss, assuming a correctly specified noise distribution. That is, given that the data generating distribution (X, Y) follows an LSNM in the direction X -+ Y. the same distribution with equal likelihood cannot be induced by an LSNM in the backward direction X +- Y. except in some pathological cases. In terms of our notation, direction identifiability means that (-, Pz)
data generating model, then
P(LL,,22 L. Pa (D) > 0) aas 00
Immer et al. 9 prove the following identifiability result:
Theorem C.1 (Theorem 1 from 9).. For data (X,Y) that follows an LSNM in both direction X + Y and X + i.e.,
Y f(X) + g(X) Zy. where X 11 Zy
X = h(Y) + k(Y)- Zx, where Y JL Zx
13

---

The following condition must be true:
g(xx)
0²
8² G(a.y) dydr "x(((())))
g(x)
(logp(y)))
(logg(v))
"xjy (xly)
G(r.y)
Qy²
(5)
+ g'(x) 0
Vx (xly) (=ly)= =
G(a,y)
dy
where G(x,y) g(0) f' + ('))) (y- and Pxjy (aly) log PZx
(- k(g)
log k(y).
They state that Equation (5) will be false except for "pathological cases". addition, Khemakhem
al. [1] provide sufficient conditions for LSNMs with Gaussian noise to be identifiable. Our next theorem provides identifiability results for some non-Gaussian noise distributions:
Theorem C.2. Suppose that the true data- cata-generating distribution follows an LSNM model both X Y and X Y directions:
1. If the noise distribution is wniform(a. b), then both g(X) and k(Y) are constant functions.
2. If the noise distribution is continnuosBermmulliii. # 0.5 1.50or exponential(X), then one of the following conditions holds:
and ø(
are constant functions.
YY
x and E linear functions with the same coefficients X and Y, respectively.
The proof is Appendix D Essentially, the theorem shows that for uniform, exponential, and continuousBernoulli noise distributions, the true LSNM model can be identified unless it degenerates to (i) a homoscedastic additive noise model or (ii) a heteroscedastic model with the same linear scale in both directions.
D Identifiability proofs
In this section, we prove Theorem the data (x, y) follows an LSNM in the forward (i.e. causal) model:
y f(z) + g(z) zy
where zy is the noise term, I IL zy, g(x) > 0 for all æ on its domain. We assume f(-) and g(-) are wiie-diffeeenniable on the domain of
the data (x, y) follows an LSNM in the backward (i.e. anti-causal) model:
h(y) + k(y) mx
where m is the noise term, IL mx, k(g) > for all y on its domain. We assume h(-) and k(-) twieeddifferentaabe on the domain of y.
zy and mx follow one of uniform(a, b), exponential(A) or contimmouuBerrooull (A) distribution accordingly.
Proof for uniform (a, b) noise. For the causal model, according to Lemma
B.
we have
Pylx(ulæ) Pzy (=((T)
[(m], - Pzy (zy) [(क) b-a a(a) (6 (b=)) Similarly, for the
s(*)
backward model, we have Px (zly) = (b-a))-(y)*
The joint likelihood of the observation (z, y) in the causal model is:
Pa,Pay (x, 8) = Px P Px(x) (b a) g(x)
The of continnous 0.5) equivalent to uniform(O. 1)).
14

---

The joint likelihood of the observation (z, y) in the backward model is:
1
Pa. (2, v) (v) (xpr(x))) Py(v) (b- a) k(y)
If the data follows both models:
P+, (I, P-.Paex (I,4)
1
1
Px(x). (b a) -)))
Py(y) (b-a) a) k(y)
1
Py( 1
Px(x)- g(x)
k(y)
Take the derivative of both sides with respect to a:
Px(x)) : )' + (Px(F))
g(z)
g(x)
1
Px(x) -1 g00/2 1(()) +0
((x)
Px(x) -1 1
g(x)
g(x)2
Px(x) > 0
g(x)²
Therefore, g(x)'
Similarly, if we take the derivative of both sides with respect to y instead, we have:
k(y) 0
These imply that both g(x) and k(y) are constant functions.
Proof for exponential(A) noise. For the causal model, according Lemma B.1 have Pylx(y|z) Pzy ( (==((
5(x) g(x) X w=f(m)
को. (v-)(w)) Similarly,
v(x) g(z)
for the backward model, we have Px)y((ly)) M
e R (x (x-b(v))
The joint likelihood of the observation (x, y) in the causal model is:
P..Pz (x, Px(x)) (ala)-Prxx(uu Pyx((1)))
Px(x). g(x) कों (v-(0))
A
log P..P2y (x,1) = log (x(x) log A log gxx)
(y f(x))
g(x)
The joint likelihood of the observation (x,y) in the backward model is:
Pe.Paa (x, y) Pr- (xly = Pr A
k(y) e मों (x-h-))
log P- Pe.Pray (x, = log Py (y) - log log k(y)
k(y) (x h(v))
15

---

If the data follows both models:
log P-y (2, y) log PM (2.
log Px (z) + log X log logg(z)
(y f()))
g(z)
A
=log Pr(y) + log log k(y)
(x h(y))
k(y)
log Px (x) log ((z) - x g(x)
y + X
f(x)
g(z)
1
log Py (v) log k(y) A.
x X k(y)
h(v)
k(y)
Take the derivative of both sides with respect to
1
f(x)) = = 1
Px()) - (log - y.
y' + x
g(z)
g(x)
k(y)
Take the derivative of both sides with respect to
g(z)
k(y)
8 k(y)
Oy
They can be equal only if both sides constants. Therefore, g(x) and both linear functions with the same coefficient on æ and y, respectively.
k(y) are both constants
Proof for continnousBBernoollii 0.5) noise. Please
refer to uniform for
continuousBernoulll = 0.5), which equals to uniform (0, 1). For the causal model, according
to Lemma B.1 have Prix (v)=)) Pzy ( (5-S(m) f(r)
g(z) g(z) Cx A 9(e)
v-f(e)
(1 (1- 9(e) g(z)*
-
v-r(e)
where Cλ Is the normalizing constant of the continuous Bernoulli distribution. Similarly, for the
backward model, we have (xly) Cx
k(g)
(1- (1- x(p)
k(2)
h(y)
kty)
The joint likelihood of the observation (æ, y) in the causal model is:
P-,P2y (x. v) (x) Py (y)x)) Px (x)
Cx X "
p(o)
(1 ~1- 1-
g(x)
log P.,p (I. v) = log Px (x) logg(z) + log C2 log X f=t
+ log(1 2)2-"=11
The joint likelihood of the observation (z. y) in the backward model is:
z-h(s)
I-hivl ri9)
Pr,PMy (x,v)= = Pv) (x Py(v) k(y)
Ox A
kios (1 - 2) -
log(1 2) - z-h(z)
log P- Pary (I,y) log (1) log k(y) + log C'A log
F137
16

---

If the data follows both models:
log P.,P2y (x. log PMx (5,
log Px(x) - g()) + log C3 + log x
v_((
vi2)
+ +og((1 X) 9(2)
x=hfa) F(S)
log([ -)) 2-h(s)
llog Py(2) log k(y) + log C\ log
log Px(x) log g(x) log C2 + y f(x)
log X (1 - 1/ f(x);
) log 2)
g(x)
g(x)
r -- h(v)
log X + (1 I h(y), loog1 A)
= =log Py log k(y) + log Cx
k(y)
k(y)
log Px (z) - log g(x) + log Cx + ((x)
log X y g(z)
1
1
log X f(x) log(1 A) logg(1 A).
y log(1
X)
f(x)
g(z)
g(x)
1
=log (y) log kyy + log C2 + k(y)
log A x k(y)
1
1 h(w)
log X hly) + log(1 A) loggl A)
log(1
A)
k(y)
k(y)
Take the derivative of both sides with respect to z:
(log Px(x)) g(x) -(x)) -1 g(x)2 g(x) llgg) y
1 log X. f(x))' - loog(1 - A) -11 1
g(0)3 9(x)" y
g(x)
f())) 1
1
+ (log(1 - A) g(x)
log log(1 - A) k(y))
k(y)
Take the derivative of both sides with respect to y:
g(0)² g(x)' log A - -log(1 A) . -1
g g(x))² 9(x)
--1 1
1
k(y)² k(y) log A -log(1 A) -1 k(y)² k(n)
9xx)² g(x)' log \ - log(1 - -A) g(x)2 ((x)
1
k(y)² k(y) log \ log(1 A) k(y)² k(y)'
g(x) . (log A - log(1 - X)) k(y)²
k(y)' (log \ - -lo(1 - A))
g(x)2
Since X + 0.5, therefore log x - -log(1 - A) + 0
g(z)' 1
k(y))
((x)2
k(y)²
θ 01
g(m)
Oy
1 and I They can be equal only if both sides are constants. Therefore, 0(2) k(y) are both constants or both linear functions with the same coefficient and y. respectively.
17

---

Table 3: Table 1
without dataset standardization.
True Noise
Gaussion (0,
Uniform (-1,))
a
0.1
0.5
5
10
0.1
0.5
5
10
V[
0.106
0.846
3.162
77.309
309.035
0.013
0.199
0.780
19.386
77.529
V[
0.446
0.707
0.790
0.818
0.814
0.016
0.126
0.189
0.228
0.226
Percentage of Datasets With
0%
90%
100%
100%
100%
40%
80%
100%
100%
100%
Misleading CVs
CAREFL-M
1.0
1.0
1.0
1.0
1.0
0.6
0.5
0.5
0.0
0.0
CAREFL-H
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1.0
0.9
E Proof for Lemma 4.1
Proof. According to Equation
we have P-,P2 (Y|x Pzy
(--(())
Ø[x] and
Pai (YIX;S's9) , Pzy ( T(Fj. Let Z3 be the standardized version of Zy, i.e.
Z'y Zr-
Zy Zry o +
0Zy
00y
V - -F((),
Y ((()
-0 p) -dctt 87y
Pzy
Pzr gX)
g((X) g'(X)
ozy x(X)
Y -x((X)
1
Pzy gx)
o + p) a. g'(X)
Let g(X] = a x(X) and ('(X)= = ((X) + p g(X), then
g((X) = a. x9(X)
Pzy Y ('(X)
. - u) g(() = -Pzy Y - (f(X) + p. g(X))
. + 1
g(X)
σ g()
o x
Y f(-((x)
= P2r (
g(X) g(X)
Therefore, P_,p2 (Y|;; = P.,p2 (Y1:;'','))
For conditional variances, have:
V2y |Y|]] (X) V g² o²
Vz, [Y|]] (X) V[Zrl -x((X) 1
Therefore, Vzy [XX]= = Vzy [X]
F Proof for Theorem 4.2
Proof. An LSNM model defined in Equation
entails the following relationships:
v Y X g² (x) v [Zy] (x(
log P P.,Pz
XX = log Pzy (Zx) + log
gXX)
18

---

where Lemma 4.1 shows that we can always assume V [2y] = 1. Also, since there is no causal relationship between and Zy in the ground truth data, i.e., changing does not change Zy, have 02y 0,
Bog Pay (Zy) 0 and bog (2y
0.
0X
ox
(g
alog (M
olog (Zx) (x)
θ Olog g(X)
0 [
(x)
θ log 9(X]
= =0+
(x)
1
2 (X)
Since g(X) 0, i... strictly positive the domain of X, therefore, 2log P.t. (Y(x) 0.
G
Proof for Corollary 4.3
Proof. To compare the data log-likelihood under the causal model -> and under the anti-causal model
we have:
log P-, Px, - log Px.
= [log P= (X) + log Pog P- (Y - [lo P-.P% (V) + log P+.rz (X()))
Theorem shows that conditional likelihood and conditional variance are negatively related. There- fore. by increasing V [Y]]] and decreasing V [ in the data, log P+.pz (Y decreases and log P-.rz (X increases. Therefore, log P_, (X, Y) decreases, log P Pt.Pz (X, Y) increases, and their difference decreases.
H The ANM version of Lemma 4.1
Proposition H.1. Let M1 be an ANM model == S(X) + Zv, where Zy is the noise with mean μ and variance o² * 0. Let M2 be another ANM model Y f'( + Zyy where Zy is the standardized version of Zv, i.e. Z'y ZY- When both Zy and Z are Gaussian,
P=,Pz (YI * P.,P2 (Y|:;
Proof. ANM is a restricted case LSNM with g(X) Similar to Equation (2) we have Pa,pa (Y| f) Pzy (Y - f(X) for M1 and P-,P (Y|X; f') Pz; f'(X) for M2.
We show that Pzy (Y ((X)) # Pzy -Y- ((()) when Zy and Zy are Gaussian.
c 1 Y-1(x)
Pzy (Y f(())
22T
1 2y
2
e 1.12))²
2T
- --2 --
P2, (Y ((X))
1. 2m
- (222)
v 2m
Therefore, (Y # P XY f') in the case of Gaussian unless a 1. Empirical results are provided in Table 4bland Appendix P
19

---

Table 4: Comparison between the accuracy and ground-truth noise variance in LSNMs and ANMs. N = =00, 000. We used (0, 1) as model prior. All the datasets are standardized to have mean 0 and variance 1. As implied by Lemma 4.1 ML model selection in LSNMs is insensitive to noise variance, when the form of the noise distribution is correctly specified. Proposition H.] shows that ML model selection in ANMs sensitive noise variance, even when the form of the noise distribution is correctly specified.
(a) Accuracy over datasets generated by SCM LSNM-sine-tannh (definition in Appendix M
True Noise
N (0, 1)
NN 4)
N(0,25)
N 100)
N 400)
[[
0.834
0.950
0.997
0.999
0.999
V[XY]
0.793
0.810
0.784
0.781
0.780
Percentage of Datasets With
70%
100%
100%
100%
100%
Misleading CVs
CAREFL-M
1.0
1.0
1.0
1.0
1.0
LOCI-M
1.0
1.0
1.0
1.0
1.0
(b) Accuracy over 10 datasets generated by SCM ANM-sine (definition Appendix P
True Noise
.V (0, 1)
N (0, 4)
N (0,25)) 25)
N N0,, 100)
N (0,400)
V
0.632
0.865
0.991
0.999
0.999
V[X[Y]
0.756
0.979
0.999
0.999
0.999
Percentage of Datasets With
0%
0%
0%
50%
40%
Misleading CVs.
CAREFL-ANM-N
1.0
1.0
1.0
0.6
0.5
CAM
1.0
1.0
0.7
0.5
0.6
CAREFL-M
CAREFL-M [11 models LSNM in Equation
via affine flows T. Each sub-flow T₁ T is
defined as the following:
X + Zx
(6)
Y tx) xx()) Zy
where X the putative cause and Y the putative effect in X Y direction. I and 81 are constants. l2 and 82 are functions parameterized using neural networks. Without loss of generality, X is assumed to function of latent noise variable Zx. If = 0 and 8 then Zx The exponential function ensures the multipliers to Z are positive without expression loss. Similarly, for the backward direction X f- Y:
Y= E + s Zy
X x= +0 Zx
where Y is the putative cause and X is the putative effect in X + Y direction. ti and s are constants. ig and s/2 are functions parameterized using neural networks.
Given Equation
(6) the joint log-likelihood of (x, y) in X - direction is:
log P., (z. y)
=log Pzx (e- (x (1)) + log Pzy (e-sa(a) (y 12(x;)) 81 =2)
Similarly for the X +- Y direction. Note that the priors Pz = PPzx. Pzy in Equation { may mismatch the unknown ground-truth noise distribution P {P> 122x P
}. }. Both CAREFL- M and
20

---

Algorithm 2 CAREFL-M
1: Input: data pairs D = (X,Y), Y), and the flow estimator T with prior Px
2: Output: estimated causal direction dir
3: Split D into training set Derain == (X Xaain, Ytrai)) and testing set Drest = (Xtests tests Ytest) direction via MI.
4: Optimize 22 ,2,52 ( Pz) in
5: Compute the likelihood L (Dess T3 3,,,3, in X Y direction
6: Optimize T2,,2,, 11.31.P2..3 (Diriin P2) in X t- direction via ML
7: Compute the likelihood L (Dessti
Y direction in X <
8: if (L,,,P>(Dee Tú 1., L. Px Desti 1 then
9:
dir dir X
10:
else if T, (restt T 6..3)) < Tu (Drst: T P.....2.3.) then
11:
dār = X +
12:
else
13: dir conclusion
14: end if
Algorithm 3 CAREFL-H (Between Residuals)
Input: data pairs
(X,Y), Y). the flow estimator T of CAREFL-M with prior Pz. and an
HSIC estimator
2: Oufput: estimated causal direction dir
3: Split D into training set Drrai := (Xtnain Ytrai)) and testing set Diest == (Xtesty Ytest)
4: Optimize T t2 (D P2) in X + Y direction via ML to estimate f1. 81, &y and
5: Compute the residuals 7x,- Xua-Iz and Zy You- (XX) e²2(X)
6a1
6: Optimize Tp T (Dtrains Px) in X &- Y direction via ML to estimate 7, 21, D2 and
Z2
7: Compute the residuals Zy, == Yuu-Pr and xx,- N P2(0) oo's
8: if HSIC (2x,,, ?v.,,) HSIC 2x.,, then
9:
dir = X =x-Y
10: else if ASSTC(xx... -x,-)>> HSTC 17x, ?x,+)) then
d= X
12: else
13:
dir == no conclusion
14: end if
CAREFL-H optimize Equation 8 for each direction over the training set. For CAREFL-M. chooses the direction with ML Score L over the testing set as the estimated causal direction. Detailed procedure for CAREFL-M is given in Algorithm 2 To map the parameters 0, th Ç, 0', v and in LSNM (Equation
2) to the flow estimator T of CAREFL (Equations (6) and 77), we have So = l2.
Go = = Px,ç-
{t1. e for = t.y, Iv = = and Pv. = {tg e
J CAREFL-H alternative independence testing
In Algorithm 1 CAREFL-H tests independence between the putative cause and the residual of the putative effect in each direction, i.e., between X and Zy in X + direction, and between and Zx in + direction. An alternative way of testing independence is to test between the residual of the putative cause and the residual of the putative effect (6, i..., between 2x and Z, in both directions. Please see Algorithm for complete steps.
Although in our experiments the two algorithms often produce the same estimation of causal direction, prefer Algorithm I since it relies on few estimations of the residuals.
21

---

K
Suitability theory
With a consistent HSIC estimator, Mooij al. [14 show that an IT method consistently selects the causal direction for ANMs if the regression method is suitable. A regression method is suitable if the expected mean squared error between the predicted residuals E and the true residuals E approaches in the limit of N - oo:
lim EDDDD N
J N
II - FI...N
where D and D' denote training and testing set, respectively. Hence, with enough data a suitable regression method reconstructs the ground-truth noise.
If an HSIC estimator is consistent, the estimated HSIC value converges in probability the population HSIC value.
HSIC P, HXIC(X,Y).
With a consistent HSIC estimator and suitable regression method, the consistency result for ANMs in Mooij et al. [14] extends naturally to LSNMs (see Appendix K.2 for a proof outline).
Proposition K.1. For identifiable LSNMS with an independent noise term one causal direction only, if an IT method is used with suitable regression method for LSNMs and consistent HSIC estimator, then the method is consistent for inferring causal direction for LSNMs.
It is important to note that, while the suitability of regression methods is valuable property for IT-based methods aiding in the determination of causal directions using independence testing, does not guarantee a higher likelihood for the causal direction compared to the anti-causal direction under noise misspecification. Therefore, suitability does not offer the same robustness guarantee for ML-based methods.
K.1 Suitability: empirical results
Table 5: Suitability of the flow estimator T of CAREFL-M in the causal direction under noise misspecification and misleading CVs. T trained with a Laplace prior. The original dataset with size 2N is split into two: 50% as training set and 50% as testing set. V > V[X[]] indicates misleading CVs in the dataset. LSNM definition is in Appendix M
(a) LSNM-tanb-exp-cosinee and continnousBernooulli0 noise. V[r[X] vs. V[X Y]: 0.324 vs. 0.291.
N=50
N=500
N=1000
N=5000
[Szx Szy
[0.02406, 0.01283)
[0.00194, 0.00204]
[0.00094, 0.00092]
[0.0002, 0.00018
(b) LSNM-sine-tanh and uniform(-1.. -1, 1) 1)
noise. V[[[X] vs. V[x| 0.422 vs. 0.367.
N=50
N=500
N=1000
N=5000
[8zx, Szyl
0..0081, 0.00285]
[0.00039, 0.00031]
[0.0002, 0.00017]
[0.00003, 0.00003]
(c) LSMMsiigmooddssggmdii and exponential(1) noise. V[ vs. W[x|r]] 0.927 vs. 0.657.
N=50
N=500
N=1000
N=5000
[8zx Szy
[0.00979, 0.00443]
[0.00074, 0.00058]
1.00045,0000028
[0.00005, 0.00005]
Let suitability value S be the left-hand side of Equation 99. Table shows an empirical evaluation of S. for the flow estimator T the causal direction. We generate data from synthetic LSNMs, and evaluate T under noise misspecification and misleading CVs. We find that the sample size grows, S approaches O. In other words, is empirically suitable under noise misspecification and misleading CVs. Therefore, Proposition K. entails that CAREFL-H based on T and consistent HSIC estimator empirically consistent for inferring causal direction LSNMs under these conditions.
Because T in CAREFL-M || uses neural networks to approximate the observed data distribution, is difficult to provide a theoretical guarantee of suitability. Therefore, although our experiments indicate that T is often suitable in practice, we do not claim that it is suitable for all LSNMs. For
22

---

example, it may not be suitable in the low-noise regime when the LSNMs are close to deterministic. well-known that since neural network models are universal function approximators, we can expect them to fit many regression functions, although it is difficult to prove suitability theoretically. For these reasons, we do not claim a priori that flow models are suitable, but we provide empirical evidence that with careful training and hyperparameter selection, we can expect them to be suitable in practice.
K.2 Proof outline for Proposition K.1
For data pairs (X, Y generated by an LSNM model X + Y,, we have:
1. Under suitability, the reconstructed noise Zy approaches the ground-truth noise Zy.
2. By employing a consistent HSIC estimator, the estimated HSIC value HSIC converges to the true HSIC value HSTC.
3. In the ground-truth LSNM, HSIC(X, 2x) = and HSTC(Y, 2x) * (i.e. identifiability of LSNMs).
Therefore, HSIC(X, 2y) = 0 and (Y Zx) + 0 for LSNMs. We only provide proof outline here, since a formal proof analogous the proof for ANMs in Appendix A of Mooij al. [14
L Default and alternative hyperparameter values used in Section 7
We the reported hyperparameter values in CAREFL-M [1] for the Tübingen Cause-Effect Pairs benchmark
[4] as the default hyperparameter values in all our experiments:
The flow estimator parameterized with 4 sub-flows (alternatively: 1, and 10).
For each sub-flow, f. g. h and k are modelled four-layer MI.Ps with 5 hidden neurons in each layer (alternatively: 2, 10 and 20).
Prior distribution is Laplace (alternatively: Gaussian prior).
Adam optimizer 12 used train each model for 750 epochs (alternatively: 500, 1000
and 2000).
L2-pennllt strength is 0 by default (alternatively: 0.0001, 0.001, 0.1).
Although also observe that LOCI-H is robust than LOCI-M under noise misspecification and misleading CVs, we omit their results because: (1) CAREFL often outperforms LOCI (see Tables] ] 2)d (2) LOCI is fixed
Gaussian distribution, whereas CAREFL can specify different prior distributions. (3) LOCI uses different set of hyperparameters than CAREFL. So their results cannot be merged into the figures.
M Synthetic SCMs used in Section 7.1
We use the following SCMs generate synthetic datasets:
LSNM-tanh-exp-cosine:
Y == tanh(X X 01) 02 + co)(X-ve1) Zy
L.SNM-sine-tanh: = sin(X 01) O2 + (tanh(X (b) + ②)) Zy
(10)
L. L.SM-sigmoiddsigmoode
Y a(X 0,) 02 + ax w1) 1/2 Zr
where Zy the ground-truth noise sampled from one of the following distributions: BerinuuullBBnooullll
uniform(-1, 1), expomential(1), beta(0.5. 0.5). Gaussian(0. (0, 1) and Laplace(O, (0, 1). a the sigmoid function. Although we did not prove identifiability with beta bela(0.5. 0.5) noise in Theorem C.2 empirically we find it is identifiable. Following (33 34 each 0 and are sampled uniformly from range [-2, - [0.5, 2). & sampled uniformly from range [1, 2) to make the tanh function positive. The number of data pairs in each synthetic dataset is N E (500, 5000).
23

---

N
Hyperparameter values of CAREFL-H for Table 2b
Following Khemakhem et al. M. use a single set of hyperparameters for all 99 datasets, which is found by grid search. To acquire the result of CAREFL-H in Table 2b the hyperparameter values used are as follows:
Number of hidden neurons in each layer of the MLPs; 2
Number of sub-flows: 10
Training datasct testing dataset 100%.
The of the hyperparameter values identical the default ones.
O Running time
The running time of CAREFL-H is slightly longer than CAREFL-M, due to the additional indepen- dence tests, i.e. HSIC(X test, Zv) and HSIC test, Zx). Please see Table 6 The running time is measured on a computer running Ubuntu 20.04.5 LTS with Intel Core i7-6850K 3.60GHz CPU and 32 GB memory. No GPUs are used.
Table 6: Accuracy running time in seconds over 10 datasets generated by SCM LSNM-sine-tanh (definition in Appendix M) with N = 10, 000 samples. a = 1. Ground-truth noise distribution is uniform(-L. 1). Model prior distribution is Gaussian (0, 1).
Method
Average Running Time in Seconds
CAREFL-M
33.23
CAREFL-H
39.38
P Additive noise models under misleading conditional variances
We show the accuracy of ANM ML methods with ANM data under misleading CVs, and compare with LSNM ML methods. The ANM data is generated by the following SCMs:
ANM-sine:
== sin X 01) 0g + Zy
ANM-tanh:
Y tanh 01) 02 Zy
(11)
ANM-sigmoid:
:= o(x 01) 0₂ + Zy
where Zy is the ground-truth noise sampled from one the following distributions: sniform(-1. 1)
and Gaussian (0, 1). is the sigmoid function. Following 33 34 each is sampled uniformly
from range 2-2, 0.55 U [0.5, 2].
CAREFL-ANM-M. is the ANM counterpart of CAREFL-M (see Appendix m. The LSNM flows (see Equation 6 in CAREFL-M are changed to model ANMs:
X X Zx
Y (X) + Zy
Similar to LSNM ML methods, the accuracy of ANM ML methods is also negatively related to misleading CVs, as shown in Figure Unlike LSNM ML methods, which suffer from misleading CVs under noise misspecification (see and annuresZland[]]. an ANM ML method suffers from misleading CVs with either noise misspecification correct specification, as illustrated in Figure 5 Figure summarizes the differences between ANM and LSNM ML methods.
24

---

1.0
1003
0.8
0.8
sccuracy
0.6
Accurasy
0.6
0.4
0.4
0.2
100
30
50
70
80
90
100
30
40
50
60
70
80
90
100
Percentage of Datosets with Misleading CVs (%)
Percentage of Datosets With Misteading CVs (%)
(a)
LSNM-sine-tanh and uniform(-he
(b) LSNM-sine- -anh and Gaussian 1) noisc
noise
1.0
1.0 X
0.8
0.8
Sccuracy
accuracy
0.4
0.2
0.2
0.0
0.0
100
20
30
40
50
60
70
80
100
30
40
60
Percentage of Datasets With Misleading CVs (%)
Percentage of Datasets With Misleading CVs (%)
(c)
LSNM-tani SaMM--tth-epp--osiiee
(d)
LSNM ASNM-tann-eee maan-exp-cosingg
and
uniform f
1, 1) noise
Gaussian (0, 1) noise
0.8
X
0.6
Rccunecy
0.6
0.4
0.4
0.2
0.2
60
70
80
90
100
80
90
100
Percentage of Detesets With Misleading CVs (%)
Percentage of Detasels With Misleading CVs (%)
(e)
SSNM-sigmoid-signont
and
(l) Gaussian (0, 1) noise
LSNM WWMM-ssgmidd-simmoie
and
uniform( (-1, 1) noise
Figure 4: We rewrite Equations 10 as == f
+ (aaaagg[]] (X) Zy- Altering changes the variance
of noise, and creates datasets with difference CVs. 10 datasets are generated with each a. CVs computed by binning the true cause. All the methods in the figure assume Gaussian (0, 1) whereas the ground-truth noise is either uniform(-1 1) Gaussian 1). All the datasets are standardized to have mean 0 and variance 1.
25

---

1.0
PM
90 1.0 X
CAN
sccuracy
0.6
Accuracy
0.6
X
0.4
0.4
0.2
0.2
0.0
60 70 Percentage of Datasets With Misleading CVs (%)
10
50
Percentage of Datosets With Misleading CVs (96
(a) ANM-sine and uniform(- 1) noise
(b) ANM-sinc and Gaussion 1) noisc
1.0 X
1.0
0.8
0.8
Scc.racy
Accuracy
0.4
0.4
0.2
0.2
0.0
0.0
0 10 20 30 40 50 60 70 80 90 100
0
10
20
30
40
50
60 70
Percentage of Datasets With Misleading CVs (%)
Percentage of Datasets With Mis Mislending CVs (%6
(c) ANM-tanb tanh and uniform 1) noise
(d) ANM-tanh and Gaussian (0, 1) noise
1.0
1.0 8
0.8
0.8
Accurscy
0.6
iccurass
X X
0.4
0.4
0.2
0.0
0.0
0 10 20 30
40 50 60 70 80 90 100
Percentage With Misleeding
10
20
50 60 70
Percentage of Datasets With Mis Misieading CVs (96)
(e) ANM-sigmoid and uniform(-1..) noise
ANM-sigmoid and Gaussian 1) noise
Figure 5: We rewrite Equations II as Y == f(X) + a Zy. Altering a changes the variance of noise, and creates datasets with difference CVs. 10 datasets are generated with each CVs are computed by binning the true cause. All the methods in the figure assume Gaussian (0, 1) whereas the ground-truth noise is either uniform(- 1) Gaussian (0, 1). All the datasets standardized to have mean 0 and variance 1.
26

---

Table 7: (Best viewed in color) Summary of datasets. Settings Section 7.1 are color-coded. (1)
Blue: with noise misspecification; (2) Red: with more than 50% of datasets having misleading CVs; (3) Brown: both (1) and (2). The datasets in Section 7.1 are generated by LSNMs and the datasets
in Sections 7.2 and 7.3 are not.
Type
Name
Noise
Number of Datasets
Percentage of Datasets With
Misleading CVs
(%)
Uniform ()
10
60
(0.5, 0.5)
10
80
LSNM-tanb-exp-ooinne
Continuous
10
70
Exponentiaa(
10
20
Gaussian 00, 1)
10
40
Laplace (0, 1)
10
40
Uniform 1) (1
10
100
Beta (0.5, 0.5)
10
100
Synthetic (Section 7.1
LSNM-sine-tanh
CondonuovsBerroool((....)))
10
60
Eaponertlial
10
20
Gaussian (0,
10
60
Laplace (0, (0,
10
80
Uniform (-1,1)
10
90
Beta 0.5)
10
100
LSNM-sigmoid-signnoii
ContinuoouSBernoull
80
Exponential
10
70
Gaussian (0, 1)
10
90
Laplace 1)
10
100
SIM
100
40
Synthetic (Section 7.2
SIM-c
N/A
100
46
SIM-In
100
23
SIM-G
100
29
Real-World
Tübingen Cause-Effect Pairs
N/A
99
40
(Section 7.3
10
10
10
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000 2000
0.0001 0.001
0.1
laplace Priors
Number of Hidden Neurons
Number of Sub Fub-Flows
Number Epochs
12-Penalty
(a) N 500
1.0
1.0
1.0
1.0
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000
2000
0.0001 0.001
laplace gaussian
Number of Hidden Neurons
Number of Sub-Flows
Number of Epochs
L2-Penalty
Priors
(b) N = 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure Accuracy over 10 datasets: LSNM-tanh--xppcosine and uniform -1, 1) noise.
27

---

1.0
1.0
1.0
1.0
10
Accuracy
0.0
0.0
0.0
0.0 laplace gaussian Prions
Number of Hidden Neurons
Number of Sub Flows
Number of Fpochs
12 12-Penalty
N
500
1.0
1.0
1.0
10 1.0
Accurocy
0.0
0.0
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub-Flows
Number Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H
CAREFL-H (1.0)
CAREFL-H
CAREFL-H (1.0)
Figure 7: Accuracy over 10 datasets: LSNM-ttan--exp-oosinee and beta 0.5) noise.
1.0
10
10-
10
10-
Accuracy
0.0
0.0
0.0
0.0
0.0
20
10
500
750
1000
2000
0.0
0.0001 0.001
0.1
laplace gaussian
Number Hidden
Number Sub-Flows
Number
L2-Penalty
Priors
(a) N 500
1.0
1.0
1.0
1.0
10
Accuracy
0.0
0.0
0.0
0.0
0.0
10
20
10
500
750
1000
2000
0.0
0.0001 0.001
0.1
laplace
gaussian
Number Hidden
Number Sub-Flows
Number
L2-PenaIty
Priors
(b)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 8: Accuracy over 10 datasets: LSNM-tanb-exp-oom and continnousBernoulif.. noise.
28

---

10.
100
10-
10
10-
Accuracy
0.5
0.5
0.5
0.5 0.5
gaussian Priors
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
N
500
1.0
1.0
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub Flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure Accuracy over 10 datasets: LSNM-tanh-exp-cosine and exponential(1) noise.
10
10
10
10
10-
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000
2000
0.00010.001 L2-Penalty
0.1
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Hlows
Number of Epochs
(a) N 500
1.0
1.
1.0
10
10
Accuracy
0.5
0.5
0.5
0.5
0.5
gaussian
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 10: Accuracy over 10 datasets: LSNM-sine-fanh and uniform( -1,
(-1. 1) noise.
29

---

10
1.0
10-
10-
10
Accuracy
0.5
0.5
0.5
0.5 0.5
gaussian Priors
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
N
500
1.0
1.0
1.0
1.0
1.0
Accuracy
0.5
0.5 0.5
0.5
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub Flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 11: Accuracy over datasets: LSSM-ssne-tanh and (0.a, 0.5)
noise.
10
10
10-
10
10-
Accuracy
0.5
0.5
0.5
0.5
0.5
1000 2000
0.0001
gaussian
Number of Hidden Neurons
Number of Sub-Hlows
Number of Epochs
L2-Penalty
Priors
(a) N 500
1.0
1.0
1.0
1.0
1.0
Accuracy
0.5
0.5
0.5
0.5
0.5
gaussian
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 12: Accuracy over 10 datasets: LSNM-sine-tanh and continnoussermoullf00..00 noise.
30

---

10
1.0
10-
10- 10-
Accuracy
0.5
0.5
0.5
0.5 0.5
gaussian Priors
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
N
500
1.0
1.0
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.0
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub
Sub-Flows Flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 13: Accuracy over 10 datasets: LSNM-sine-tanh and exponential(1 noise.
10
10
10-
10
10-
Accuracy
0.5
0.5
0.5
0.5
0.5
2000
0.0001
gaussian
Number of Hidden Neurons
Number of Sub-Hlows
Number of Epochs
L2-Penalty
Priors
(a) N 500
1.
10
1.0
1.0
1.0
Accuracy
0.5
0.5
0.5
0.5
0.5
gaussian
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 14: Accuracy over 10 datasets: LSNM-siggmoid-sigmoodd and uniform (- 1) noise.
31

---

1.-
1.0
1.0
1.0
10
Accuracy
0.5 0.5
0.0
0.0
0.0
0.0 laplace gaussian
Number of Hidden Neurons
Number of Sub Flows
Number Fpochs
12 12-Penalty
Prions
N
500
1.0
1.0
1.0
1. 1.0
Accuracy
0.0
0.0
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub-Flows
Number Epochs
L2-Penalty
Priors
(b) 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 15: Accuracy over 10 datasets: LSNMssigmoid-sigmmud and beta 0.5) noise.
1.0
1.0
10-
10-
10-
Accuracy
0.0
0.0
0.0
0.0
0.0
20
10
500
750
1000
2000
0.0
0.0001 0.001
0.1
laplace gaussian Priors
Number Hidden
Number Sub-Flows
Number
L2-Penalty
(a) N 500
1.0
1.0
1.0
1.0
10
Accuracy
0.0
0.0
0.0
0.0 -4
0.0
20
10
500
750
1000
2000
0.0
0.0001 0.001
0.1
laplace gaussian
Number Hidden
Sub-Flows
Number
12-PenaIty
Priors
(b)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 16: Accuracy over 10 datasets: LSN LSNM-sigmoid-sigmolll and continnousBernoulif.. noise.
32

---

10-
10
10
10-
10-
Accuracy
0.5
0.5
0.5
0.5 0.5
gaussian Priors
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
N
500
1.0
1.0
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub Flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 17: Accuracy over 10 datasets: L.SM-sigmoid-sigmoid and exponential(1) noise.
10
10
10-
10-
10
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000
2000
0.0001 0.00100001 L2-Penalty
0.1
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Hlows
Number of Epochs
(a) N 500
1.
1.
1.0
10
10
Accuracy
0.5
0.5
0.5
0.5
0.5
gaussian
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 18: Accuracy over 10 datasets: LSNM-tanh-sxp-oosnne and Gaussian 1) noise.
33

---

10
100
10-
10-
10
Accuracy
0.5
0.5
0.5
0.5 0.5
gaussian Priors
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
(a) N 500
1.0
1.0
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub Flows
Sub-Flows
Number of Epochs
12-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 19: Accuracy over 10 datasets: LSNM-tanh-exp-coiine and Laplacc(0, 1) noise.
10
10
10
10-
10
Accuracy
0.5
0.5
0.5
0.5
0.5
1000
2000
0.00010.001 L2-Penalty
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Hlows
Number of Epochs
(a) N 500
1.0
1.0
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.5
gaussian
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 20: Accuracy over datasets: LSNM-sine-tanh and Gaussian 00, 1) noise.
34

---

10
10
10-
100
10
Accuracy
0.5
0.5
0.5
0.5 0.5
gaussian Priors
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
N
500
1.0
1.0
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.0
0.0
laplace gaussian
Number of Hidden Neurons
Number of Sub Flows
Number of Epochs
12-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 21: Accuracy over datasets: LSNM-sine-tanh and Laplace(O, 1) noise.
10
10-
10-
10-
Accuracy
0.5
0.5
0.5
0.5
0.5
1000 2000
0.00010.001 L2-Penalty
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Hlows
Number of Epochs
(a) N 500
1.
1.
1.0
10
10
Accuracy
0.5
0.5
0.5
0.5
0.5
laplace gaussian
Number of Hidden Neurons
Number of Sub-Flows
Number of Epochs
L2-Penalty
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 22: Accuracy over 10 datasets: LSNMsiggmoiddsigmood and Gaussian 1) noise.
35

---

10
10-
10
10
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000
2000
0.0001
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Flows
Number of Epochs
L2-Penalty
(a) N 500
10
10
10
10
Rccuracy
0.5
0.5
0.5
0.5
0.5
1000 Number of Epochs
0.00010.001 L2-Penalty
laplace gaussian
Number of Hidden Neurons
Number of Sub-Hlows
Priors
(b) N 5000
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 23: Accuracy over 10 datasets: L.SM-sigmooidssigmoidd and Laplace(0, 1) noise.
36

---

1.0
1.0
1.0
1.0
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000
2000
0.0001 0.001
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Flows
Number of Epochs
L2-Penalty
(a) Accuracy over 100 datasets from SIM sub-benchmark.
10
1.0
1.0
10
Accuracy
0.5
0.5
0.5
0.5
0.5
500
1000
2000
0.0001 0.001 L2-Penalty
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-Flows
Number of Epochs
(h) Accuracy over 100 datasets from SIM-e nah-benchmark.
1.
10
10
10
Rccuracy
0.5
0.5
0.5
0.5
0.5
1000 Number of Epochs
0.00010.001 L2-Penalty
0.1
laplace gaussian Priors
Number of Hidden Neurons
Number of Sub-tlows
(c) Accuracy 100 datasets from SIM-In wub-benchmark.
1.0
10
1.0
10
10
Accuracy
0.5
0.5
0.5
0.5
0.5
laplace gaussian
Number of Hidden Neurons
Number of Sub-flows
Number of Epochs
L2-Penalty
Priors
(d) Accuracy 100 datasets from SIM-G sub benchmark.
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-M (0.8)
CAREFL-M (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
CAREFL-H (0.8)
CAREFL-H (1.0)
Figure 24: Results with SIM benchmarks.
37


Reviews:
[{"review_id": "tK9v0TrZcX", "reviewer": "Reviewer_yfAc", "summary": "This paper compares using maximum likelihood vs using tests of independence with regression residuals to select between a causal model and an anti-causal model, in order to infer the causal direction between two random variables. An empirical observation is that assuming a location-scale noise model, the maximum likelihood approach tends to perform poorly when the noise distribution is misspecified and the conditional variance in the causal direction is greater than that in the anti-causal direction. By contrast, the independence testing approach or at least the particular affine flow model used in this paper appears to maintain good performance even in such unfavourable circumstances. In addition to extensive empirical results to establish this phenomenon, this paper also provides some theoretical explanations of the phenomenon.", "questions": "Does the specification of noise distribution play any role in the flow-based independence tests? If not, I wonder if that is the main reason why it is more robust than the maximum likelihood methods. How much, if at all, do misleading CVs by themselves (i.e., without noise misspecification) affect the performance of the ML methods?", "limitations": "OK. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper is very well written and readable. The empirical results are convincing and illuminating. The theoretical results are interesting and provide a compelling and useful explanation of the empirical observation. Given the importance and difficulty of the problem of inferring causal asymmetry from observational data, the empirical results and theoretical analysis presented in this paper strike me as sufficiently significant. ", "weaknesses": "As far as I am concerned, this paper does not have notable weaknesses. I would appreciate some explanation or even speculation on why the flow estimator remains empirically suitable under noise misspecification and misleading conditional variances. Does noise specification play any role in the flow estimator? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re: Weakness. Questions. Why Flows Remain Suitable Under Noise Misspecification. Role of Noise Specification.\nPlease refer to the first global response (*Re: Robustness of Independence-Based Methods Under Noise Misspecification in LSNMs*) for our answer to these comments.\n\n### Re: Question, How Much do Misleading CVs Alone Affect the Performance of Maximum-Likelihood (ML) Methods\nThis question highlights another contribution of the work, demonstrating the differential impact of misleading CVs on the performance of ML methods in the context of LSNMs and ANMs. The details can be found in Figure 1, line 159-161 and Appendix O (line 567-571).\n\nIn summary, \n- In LSNMs with correctly specified noise priors (i.e. without noise misspecification), the ML model selection score decreases for the true causal structure as CVs become more misleading, but **does not flip**, as implied by the theoretical identifiability results. \n- In LSNMs, adding noise misspecification to misleading CVs pushes ML scores over the edge of a wrong decision. \n- In ANMs,  misleading CVs **lead towards a wrong causal structure** even with correctly specified noise priors.  \n\nThis illustrates a difference between LSNMs and ANMs regarding their sensitivity to misleading CVs. "}, {"review_id": "9hOyuFYCoP", "reviewer": "Reviewer_bEFo", "summary": "The authors investigate bi-variate causal inference with non iid, heteroscedastic noise. They specifically investigate a group of causal inference methods based on max-likelihood and present two main drawbacks: max-likelihood approaches with parametric assumptions on the noise distribution, such as normality, suffer when these distributions are misspecified. Further, the authors show that max-likelihood is inversely proportional to conditional variance. Hence, if the conditional variances are misleading the max-likelihood points in the anti-causal direction, which is the case for many pairs in the Tübingen real world cause-effect benchmark. The authors propose an independence test based approach fitting an affine flow to model the conditional density and HSIC test the residuals.\n", "questions": "- Table 1: If conditional variance is misspecified, i.e. higher in the causal direction, how do the M methods still get 100% right for all gaussian high noise pairs? And why does it not work for uniform noise?\n- What could be some benefits of using max likelihood, for example, if learned function \\hat{f} or scaling \\hat{g} is misspecified, my intuition is that max likelihood has more slack than independence tests.\n- What is the sample complexity of CAREFL-H, how much is it increased as both a train/test set is needed, where the test set needs to be large enough for an HSIC test with sufficient statistical power.\n- Experiments: you test noise independence after a regression with CAREFL-H same as RESIT, why is RESIT better on SIM across the board?\n\n\n", "limitations": "The downsides of independence testing approaches are not addressed. Hence, the comparison between independence testing and max-likelihood methods is not complete. For example, in the homoscedastic case max-likelihood methods (CAM) outperform independence test methods (RESIT). This hints towards max-likelihood methods also having an edge against independence tests in some cases.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Investigating the assumptions and failure modes of bi-variate causal inference methods is an important to ensure that these methods are accurate under real-world conditions.\n- The authors provide a thorough argument against max-likelihood based methods based on theory and empirical evidence. The empirical experiments demonstrate how misspecified noise and noise level affect the accuracy of max-likelihood methods.\n- The linkage of max-likelihoods failure modes to misleading conditional variance is plausible and is backed by theory.\n- The proposed method CAREFL-H performs well in the benchmarks covering synthetic and real world data. It outperforms both independence test and maximum likelihood based approaches alike.", "weaknesses": "- The contribution of this work is limited in light of recent work. It partially overlap with the work on noise misspecification by Strobl & Lasko 2022, and the critic on likelihood-based causal inference by Schultheiss & Bühlmann, 2023.\n- The pitfalls of independence test methods is not discussed, leading to an incomplete picture in regards to comparing both families of approaches.\n- The proposed amendment to the CAREFL method introduces only an independence test afterwards and applies the same scheme (RESIT, GRCI) with different function modelling tools.\n- The claim, that affine flows are \"empirically suitable under noise misspecification and misleading CVs\" is not sufficiently supported by the experiments which cover only limited function classes (sine, sigmoids).\n- For instance, CAREFL-H performs worse than RESIT on SIM, which also uses HSIC with a different function estimator that works than the affine flows.\n- Finally, the presentation including writing and figures is unclear at times (for details see minor remarks).\n\nReferences\nSchultheiss, C., & Bühlmann, P. (2023). On the pitfalls of Gaussian likelihood scoring for causal discovery. Journal of Causal Inference, 11(1), 20220068.\nStrobl, E. V., & Lasko, T. A. (2022). Identifying patient-specific root causes with the heteroscedastic noise model. arXiv preprint arXiv:2205.13085.\n\nMinor Remarks:\n- Figure 2 plots are squished.\n- Table 2a) is very cluttered and lacks a caption, maybe better as line chart to see trend.\n- Theorem 4.2 is a main result of the paper but not explained sufficiently, i.e. only half a line long and the following paragraph is also very sparse.\n- The acronyms ML and IT are associated with machine learning and information technology and hence easy to confuse, consider using different acronyms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re: Contribution of the Work & Distinction to Previous Works.\nPlease refer to the second global response.\n\n### Re: Pitfalls / Downsides of Independence Test\nOur answer depends on the pitfalls you have in mind. If your concern relates to the sample inefficiency of independence-based methods, we addressed this matter in Section 6.1.2, line 255-260, as well as in our future work discussion on line 296. If your concern relates to the computational cost of independence testing, it is not a concern in our setting since we are targeting bivariate settings as mentioned on lines 2, 34 and 108. \n\n### Re: Suitability\nSuitabilty basically means that an LSNM regression model has the capacity to predict the correct Y-value given input X (in expectation). It is well-known that since neural network models are universal function approximators, we can expect them to fit many regression functions, although it is difficult to prove suitability theoretically (line 203). Indeed our experiments show that suitability requires choosing hyperparameters to ensure sufficient model capacity (Section 6.1.2, lines 253 and 262)). \n\nFor these reasons, we do not claim a priori that flow models are suitable (see lines 202-205), but we have provided empirical evidence that with careful training and hyperparameter selection, we can expect them to be suitable in practice. Table 2 summarizes our main experiments, including several non-linear functions (tanh, cosine, sine, and sigmoid) as well as difference ground-truth noise distribution (continuous Bernoulli, uniform, and exponential).\n\n### Re: Questions 1.\nThe answer to the question is directly provided by lines 147-150 and Theorem C.2. \n- Identifiability theorems of LSNMs establish that the causal direction consistently has a higher likelihood when the model noise prior is correctly specified. This conclusion remains unaffected by whether the conditional variance in the data is higher in the causal direction or not.\n- Regarding your example comparing Gaussian noise to uniform noise, as explained in the paper (e.g. lines 42-43, 127-134, and Table 1), the discrepancy in accuracy arises from noise misspecification, not the noise distribution itself. For example, specifying a uniform model prior when the true noise is uniform also yields high accuracy, which is proved by Theorem C.2.\n\n### Re: Question 2. Benefits of using maximum likelihood.\nWhile our primary focus is on addressing the problem of noise misspecification and misleading CVs in LSNMs, where independence-based methods demonstrate superior performance, we have mentioned in the paper a few cases where likelihood-based methods may perform better:\n\n1. Lines 261-262, when neural networks have low capacity to fit the functions $f$ and $g$, results the learned functions $\\hat{f}$ or $\\hat{g}$ to be misspecified. In this case, independence-based methods cannot utilize a good reconstruction of the noise.\n2. Lines 155-156, when the noise distribution is known, the performance of likelihood-based methods is guaranteed by identifiability theorems.\n3. Lines 255-260, with a small sample size and correctly specified noise distribution, likelihood-based methods may be more accurate. Independence-based methods require a larger number of samples to achieve accurate results. Intuitively, when the user managed to correctly specify the model, the likelihood-based methods can take advantage of this knowledge to be more sample efficient. \n\n### Re: Question 3. Sample Complexity of CAREFL-H.\nThe specific answer to this question depends on the architecture of the neural networks employed in CAREFL-H, but it is challenging to provide exact answer due to the lack of theoretical results for neural networks. However, we mentioned on lines 255 that CAREFL-M can have greater sample efficiency compared to CAREFL-H. For sample complexity of different HSIC estimators, please refer to relevant papers such as [5] and [14].\n\n### Re: CAREFL-H VS. RESIT on SIM\nDid you mistakenly look at CAREFL-M instead of CAREFL-H? As indicated in Table 3(a), CAREFL-H demonstrates superior performance over RESIT on SIM benchmark in 3 out of 4 settings.\n\n### Re: Limitations\nWe have addressed the downsides of independence testing and cases when likelihood-based methods have an edge against independence-based methods in several places of the paper, as stated in earlier responses.\n\nFor your example of CAM outperforming RESIT, it happens in the homoscedasitc settings of ANMs, which have been extensively discussed, and are not the main focus of our paper. Our paper primarily focuses on heteroscedastic settings of LSNMs, specifically under noise misspecification and misleading CVs. In these contexts, independence-based methods demonstrate superior performance.\n\n### Re: Theorem 4.2 is Short\nPlease note that some of the informal explanation and motivation for Theorem 4.2 and Lemma 4.1 appears **before** the mathematical statements since they flow from our empirical results in Table 1. All terms in the mathematical expression in Theorem 4.2 are defined. As mentioned on line 142, Theorem 4.2 demonstrates that the likelihood and conditional variance are negatively related. We utilize Theorem 4.2 as a foundation to derive Corollary 4.3 and the three relationships discussed on lines 153-158. In essence, the key insights and highlights of Section 4 (lines 142-158) stem from Theorem 4.2.\n\n### Re: Acronyms\nWe would like to clarify that we have explicitly defined these acronyms, ML (Maximum Likelihood) and IT (Independence Testing), in multiple places throughout the paper (lines 37, 78, 289, and 292) to ensure good readability and comprehension. \"Machine Learning\" or \"Information Technology\" terminologies are rarely mentioned in the paper, if at all. Therefore, we believe that the acronyms do not lead to confusion for the readers.\n\n### Re: Captions\nWe want to clarify that all (sub)tables and figures, including Table 2(a), do have captions."}, {"review_id": "rh0nggDqak", "reviewer": "Reviewer_GXXz", "summary": "This paper explores causal discovery using heteroscedastic location-scale noise functional models (LSNMs) (which generalizes the additive noise models) and compares maximum likelihood model selection to causal model selection through residual independence testing. ", "questions": "- Q1. What is the implication and usage of Lemma 4.1 in the paper?\n- Q2. What is the definition of noise misspecification and misleading CV? Does it mean that the estimation model for the residual (noise) is wrongly specified?\n- Q3. “(Identifiability) When the form of the noise distribution is correctly specified, the causal model always has a higher likelihood (Figure 2a).” can be proved by Theorem 4.2 or Corollary 4.3? If so, could you please provide easy and intuitive explanations?\n- Q4. I think the proof for Proposition 5.1 should be provided.", "limitations": "I believe that more theoretical analysis is necessary to claim the robustness of IT. For instance, does Proposition 5.1 imply that the IT-based method is robust against noise misspecification and misleading cross-validation? If we assume regression suitability in the machine learning method, will the result change for ML?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I think this paper is significant in that it provides empirical and theoretical evidence of using the IT method for causal discovery. \n2. This paper is well-written. The contribution is very clearly written from the introduction, so little chances to miss the contribution of the paper. \n3. The simulations are extensive.", "weaknesses": "1. I think the robustness of the IT method should be more theoretically advocated. \n2. Some results (e.g., Proposition 5.1.) don’t have proofs, and I am not sure they are trivial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re: Weakness, 1 & Limitations, Robustness of IT Methods\n\nPlease refer to the first global response (Re: Robustness of Independence-Based Methods Under Noise Misspecification in LSNMs) for the answer regarding the robustness of IT methods.\n\n### Re: Questions, Q1\n\nThe implication and usage of Lemma 4.1 is 3-fold:\n1. (Appendix F) It is used to prove Theorem 4.2.\n2. (Lines 147-150) It indicates that the performance of likelihood-based methods for LSNMs remains unaffected by misspecifying the parameters of the model prior, such as the mean and variance.\n3. (Lines 159-161) It demonstrates that the estimated direction using likelihood in LSNMs does not flip when scaling the variances (under correct noise misspecification), which is a significant distinction from ANMs.\n\n### Re: Questions, Q2\n\nThe definitions of noise misspecification and misleading CVs are provided on lines 42-44 of the paper\n\nNoise misspecification refers to the situation where the noise of the estimation model is incorrectly specified by the user.\n\nMisleading CVs indicates the conditional variance in the ground-truth data being higher in the ground-truth causal direction and lower in the ground-truth anti-causal direction.\n\n### Re: Questions, Q3\n\nThe reference to the identifiability theorems from previous work can be found on line 147. These theorems establish that if the noise distribution is accurately specified in the model prior, and under the assumption that the data is generated by LSNM, the causal direction always has a higher likelihood than the anti-causal direction (except in some rare cases). These identifiability theorems are the foundation of likelihood-based methods for LSNMs.\n\n### Re: Question Q4, Proof of Proposition 5.1 & Weakness, 2\n\nWe did not give an independent proof of Proposition 5.1, because we felt it is too close to the analogous results for ANMs in Appendix A of [14] to present as a novel contribution. However, in Section 5.2 of our paper we present the underlying idea:\n1. (Line 183-187) Under suitability, the reconstructed noise $\\hat{N}$ approaches the ground-truth noise $N$ .\n2. (Line 188-189) By employing a consistent HSIC estimator, the estimated HSIC value $\\widehat{\\text{HSIC}}$ converges to the true HSIC value $\\text{HSIC}$.\n3. (Line 192) In the ground-truth LSNM, $\\text{HSIC}(\\text{Cause}, \\hat{N}\\_{\\text{Effect}}) = 0$ and $\\text{HSIC}(\\text{Effect}, \\hat{N}\\_{\\text{Cause}}) \\neq 0$ (i.e. identifiability of LSNMs).\n\nTogether, we have $\\widehat{\\text{HSIC}}(\\text{Cause}, \\hat{N}\\_{\\text{Effect}})=0$ and $\\widehat{\\text{HSIC}}(\\text{Effect}, \\hat{N}\\_{\\text{Cause}}) \\neq 0$ for LSNMs.\n\n### Re: Limitation, Suitability for Maximum-Likelihood Methods\n\nPlease refer to the first global response (Re: Robustness of Independence-Based Methods Under Noise Misspecification in LSNMs) for the robustness of IT methods under noise misspecification and misleading CVs.\n\nThe suitability of regression methods is a valuable property for independence-based methods since they facilitate the reconstruction of the ground-truth noise, aiding in the determination of the causal direction using independence testing. However, the suitability of regression methods does not necessarily guarantee a higher likelihood for the causal direction compared to the anti-causal direction. Therefore, it does not offer the same value for maximum-likelihood based methods.\n"}, {"review_id": "tUk7JM9bzH", "reviewer": "Reviewer_ttx5", "summary": "The paper discusses the sensitivity of the recently introduced heteroscedastic noise model to model specification for causal discovery in a bivariate scenario. Specifically, it focuses on the impact of misspecifying the assumed noise distribution, which leads to poor performance when choosing the direction based on maximizing the likelihood. Instead, the paper proposes the use of independence testing of the residuals and presents an improved model. The newly proposed model demonstrates promising results in both real-world and synthetic datasets.", "questions": "The paper is generally well-written and includes an in-depth comparison with related work. Some general remarks and questions:\n\n* You mention LiNGAM as an independence testing approach. However, strictly speaking, it only relies on the ICA components (assuming non-Gaussian noise), which is not utilizing independence tests. \n* The reference for GRCI appears to be incorrect.\n* There needs to be a more comprehensive comparison with the work by Strobl et al., as they do not rely on prior knowledge about the noise or exploitation of the conditional variance, which is the main critique of this paper.\n* The meaning of Figure 1 could be clearer. For instance, what does “high” and “low” exactly mean for accuracy?\n* When considering the conditional variance, it seems evident that this is primarily a matter of scaling. One can simply scale X and Y differently to arbitrarily flip edges for approaches that do not carefully scale the variables as part of their methodology.\n* Table 1 seems slightly out of place and appears to be more appropriate for the experiment section.\n* In the case of CAREFL-M, isn't the prior distribution merely a convention, since they learn a transformation function of the noise, i.e., they use g(N) and not N directly. Therefore, the Gaussian noise can be transformed into 'arbitrary' noise distributions. Perhaps, I'm missing something here?\n\n--Update after rebuttal-- I have read the rebuttal and further discussed with the authors. Many of my concerns have been addressed and I raised the score by 1.", "limitations": "No concerns regarding societal impact. Regarding technical limitations, this paper focuses primarily on improving an existing methodology and, thus, shares the same limitations as the original work. However, it aims to make the methodology less sensitive to model misspecifications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Excellent and clear introduction.\n* Comprehensive literature review with useful categorization of different approaches.\n* Highlights flaws in the existing prior approach.\n* Extensive experiments using popular causal discovery datasets.\n\nAlso see questions section.", "weaknesses": "* The novelty appears to be limited: The result seems rather incremental, considering that the implications of model misspecification aren't surprising, and the use of independence tests is a relatively common way to utilize functional causal models for causal discovery.\n* There is a limited discussion about the work \"Identifying Patient-Specific Root Causes with the Heteroscedastic Noise Model\" by Strobl et al., given that they don't rely on maximizing the likelihood for fitting the model, do not seem to depend on prior knowledge about the noise distribution and also focus on causal discovery.\n\nAlso see questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re: Weakness, Novelty & Limitations\nPlease refer to the 2nd global response (*Re: Contribution*) regarding the novelty / contribution of the work.\n\n### Re: Weakness, Little Discussion of GRCI\nWe introduced GRCI in our paper on line 90 and conducted empirical comparisons with it in the experiment section. The primary focus of our work is addressing the challenges of noise misspecification and misleading CVs in LSNMs. It's important to note that the authors of GRCI do not discuss this problem in their work.\n\nWe used GRCI as a comparison baseline for our proposed algorithm CAREFL-H. While both GRCI and CAREFL-H achieve similar accuracy against the Tübingen Cause-Effect Pairs benchmark, CAREFL-H demonstrates superior performance over GRCI in all settings when evaluated against the popular SIM benchmark (Table 3a). \n\n### Re: Questions, LiNGAM\nWe would like to clarify that in our paper, we are referring to the Direct-LiNGAM algorithm [25], not the ICA-LiNGAM [24]. Direct-LiNGAM is an independence-based algorithm.\n\nAs an additional clarification, the ICA-LiNGAM algorithm is also related to independence, specifically focusing on the mutual independence among noise terms in the context of SCMs. This concept is equivalent to mutual independence among the sources in the context of ICA.\n\n### Re: Questions, More Comparison with GRCI\nOur paper is primarily centered around identifying the challenges faced by likelihood-based methods in LSNMs, which is the main focus of our research. On the other hand, GRCI is not a likelihood-based method, and it lies outside the primary scope of our paper. However, for comparison purposes, we empirically evaluate and compare with GRCI since both GRCI and our proposed CAREFL-H are independence-based approaches.\n\nAlthough both our work and GRCI involve causal discovery in LSNMs, the main focus of the two papers differs significantly. Our primary emphasis is addressing the specific issue of likelihood-based methods in LSNMs. The authors of GRCI do not discuss this issue in their work. In contrast, GRCI's primary focus is on detecting the root causes of events, such as causes of diseases. These distinctions underscore the unique contributions and research objectives of the two papers.\n\n### Re: Questions, What does \"High\" and \"Low\" mean in Figure 1.\nThe primary purpose of Figure 1 is to provide a qualitative summary of the quantitative results from Tables 1 & 5 and Figures 4 & 5. The labels \"high\" and \"low\" are used to convey specific meanings:\n\n- When we refer to \"high\" accuracy, we imply superior accuracy that is supported by a robust theoretical explanation, such as Corollary 4.3 or the identifiability results.\n- On the other hand, for \"low\" accuracy, we are indicating a decrease in accuracy after switching settings, for example, transitioning from LSNM to ANM, or from having no misleading CVs to having misleading CVs. This inferior accuracy is also well-grounded by theoretical explanations, such as Corollary 4.3 and Proposition H.1.\n\nIt's important to note that these \"high\" and \"low\" accuracy assessments are demonstrated quantitatively in Tables 1 & 5 and Figures 4 & 5. For instance, the \"high\" accuracy in the path of (Noise misspecification \"No\" -> Misleading CVs \"Yes\" -> LSNM) in Figure 1 corresponds to a value of 1.0 in Table 1, while the \"low\" accuracy in the path of (Noise misspecification \"Yes\" -> Misleading CVs \"Yes\" -> LSNM) in Figure 1 corresponds to a value of 0.1 in Table 1.\n\n### Re: Questions, Scaling\nWe would like to answer the question both theoretically and empirically:\n\n- In **ANMs**, scaling may **lead to a flip in likelihoods**. However, our Lemma 4.1 provides a crucial finding that scaling **does not result in flipped likelihoods** in **LSNMs** (under correct noise specification). This discovery highlights a substantial difference between LSNMs and ANMs, as elaborated on lines 159-161 and Appendix O.\n- To address the issue of scaling, a common approach is to standardize variables to have mean 0 and variance 1. We have adopted this standardization for all our experiments, as indicated on lines 48-49, 223, and Table 1. Consequently, the identified problem persists in LSNMs even when X and Y are standardized to the same scale.\n\n### Re: Questions, Isn't Gaussian Distribution Merely a Convention\nIn many machine learning tasks, using a Gaussian distribution as the model prior is a common convention, and this Gaussian prior can be transformed into arbitrary distributions (e.g. in VAEs or Normalizing Flows). \n\nHowever, causal discovery relies on specific assumptions to determine the correct causal direction. Without these assumptions, a model becomes too flexible, making it impossible to distinguish between X->Y and X<-Y. The identifiability theorems of LSNMs for ML model selection necessitate the correct prior distribution $N$ to distinguish the directions.\n\nPlease note that the LSNM model has $g(X)$ not $g(N)$ as you write. This may just be a typo, but it is important because the $g(X)$ function does not directly transform the noise $N$. It simply rescales it through the transformation $g(X) \\cdot N$. The $g(X) \\cdot N$ transformation is not powerful enough to \"correct\" a misspecified prior on $N$. If we consider $g(X) \\cdot N$ as the \"noise\", the LSNM SCM becomes $Y= f(X) + N'$, where $N' = g(X) \\cdot N$. This new SCM may appear similar to an ANM, but it is not, as $N'$ and $X$ are not independent. As of now, we are not aware of any identifiable SCMs in this particular form.\n\n### Re: Questions, Reference For GRCI is Incorrect.\nThe GRCI paper was initially available on arXiv and accepted by the Journal of Computational Science on 14 June 2023, which is after the NeurIPS submission deadline. We have updated the reference in the paper to reflect this change.\n\n### Re: Questions, Table 1 is More Appropriate for the Experiment Section.\nWe utilize Table 1 as a preview to captivate readers' interest and motivate the problem we are addressing in our work."}]

Author rebuttal:
# Global Response to All Reviewers

Thank you very much for taking your time reviewing our work! We would like to answer some common questions here. We are sorry for the inconvenience. 

Please refer to the reference section of the paper for the citation numbers from the response. 

### Re: Robustness of Independence-Based Methods Under Noise Misspecification in LSNMs

Formal theoretical results for CAREFL-H are challenging to provide due to the involvement of neural networks in the flow estimators (lines 202-203). Informally, our intuition is that 
independence-based methods are based on the **posterior** latent variable distribution $P(Z|X,Y)$ not the prior distribution $P(Z)$. A suitable regression method essentially learns to predict the correct $\hat{Y}$ value from $X$ (in expectation), and given $X$, $\hat{Y}$ implies a definite value for $\hat{Z}$, so the posterior is sharply peaked around the true value. We give a brief formal discussion to make these intuitions more precise. 

In the context of the LSNMs, the ground-truth is $Y=f(X) + g(X) \cdot Z$ according to Equation 1. Let $\hat{f}\_{\theta}(X)$ be the conditional mean estimator, i.e. $\hat{f}\_{\theta}(X) = E[Y|X]$, and let $\hat{g}\_{\phi}(X)$ be the conditional standard deviation estimator, i.e. $\hat{g}\_{\phi}(X) = \text{STD}[Y|X]$. Here, $\theta$ and $\phi$ represent the parameterizations of the neural networks and may be omitted for clarity. We have:
- $\hat{f}(X) = E[Y|X] = E[f(X) + g(X) \cdot Z | X] = f(X) + g(X) \cdot E[Z | X] = f(X) + g(X) \cdot E[Z] $
- $\hat{g}(X) = \text{STD}[Y|X] = \text{STD}[f(X) + g(X) \cdot Z | X] = |g(X)| \cdot \text{STD}[Z | X]  = g(X) \cdot \text{STD}[Z] $

So, $\hat{Z} = \frac{Y-\hat{f}(X)}{\hat{g}(X)} = \frac{f(X) + g(X) \cdot Z - f(X) - g(X) \cdot E[Z]}{g(X) \cdot \text{STD}[Z]} = \frac{Z - E[Z]}{\text{STD}[Z]}$, where $E[Z]$ and $\text{STD}[Z] > 0$ are constants. Therefore, $\hat{Z}$ and $Z$ are identical up to shift and scale. Since $X$ and $Z$ are independent, so $X$ and $\hat{Z}$ are also independent.

To compare independence-based methods and likelihood-based methods under noise misspecification:
- As you can see from the math above, the specification of the prior distribution is **not involved**, indicating that the same independence result can be achieved with or without noise misspecification. 
- However, as the model prior distribution is **involved** in likelihood computation (Equation 2 in the paper), when the model prior distribution is misspecified, the estimated likelihood may be incorrect. 

This is the intuition why independence-based methods work better than likelihood-based methods under noise misspecification in LSNMs.

### Re: Contribution

Our main contribution is the study of **LSNM** models under **noise prior misspecification**, both through theoretical analysis and empirical evaluation (Line 55-59).  We provide comprehensive theoretical results for LSNMs under the challenging conditions of noise misspecification and misleading CVs. An extensive set of experiments evaluates the empirical performance of causal discovery methods in LSNMs. For all machine learning methods, it is important to understand how sensitive their performance is to their assumptions (such as knowing the noise prior distribution). Since the noise prior specifies a distribution over the unobserved factors, arguably it is especially difficult to specify it correctly based on domain knowledge.

While our proposed CAREFL-H algorithm is a member of the RESIT family of algorithms based on independence testing, the particular combination of fitting a neural net flow model with subsequent independence testing has to our knowledge not been previously evaluated by LSNMs. This combination is especially robust according to both our theory and experiments.  

We would like to further clarify that our contributions are distinct and do not overlap with Strobl & Lasko 2022 (GRCI) [28] or Schultheiss & Bühlmann 2023 [23] for the following reasons:

- Unlike [28], which does not address the noise misspecification or misleading CVs, we identify and thoroughly analyze these challenges. Therefore, our theoretical contributions are novel and distinct from [28].

- While [23] discusses ANMs (as we mentioned on lines 96-97), our work primarily focuses on LSNMs. As highlighted on lines 159-161, our theoretical results for LSNMs and ANMs significantly differ, as supported by Lemma 4.1 and Proposition H.1. These distinctions for ANMs are further validated through empirical evidence, as presented in Table 5, Appendix O and Figure 5 (most materials about ANMs are in the appendix, as ANMs are not the focus of our work). Therefore, our findings do not overlap with [23].


Meta-review: This paper is concerned with causal discovery based on heteroscedastic location-scale noise functional models, which are generalizations of the additive noise models, comparing likelihood-based model selection to residual independence testing-based method for finding causal discovery. The basic idea of the paper is interesting, and the empirical results are interesting and convincing. The theoretical results provide useful explanations of the empirical observations.  At the same time,  as pointed out by one of the reviewers, it has much overlap with Immer et al. (2022) and Schultheiss (2022), and the presentation of the paper is to be improved.  I suggest the authors try to make the paper more self-contained and make the unique contribution of the paper clearer.
Decision: Accept (poster)