Title: Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing
Venue: NeurIPS 2023
Accepted: yes
Keywords: Causal Discovery

Cause-Effect Inference

Location-Scale Noise Models
Abstract: A fundamental problem of causal discovery is cause-effect inference, to learn the correct causal direction between two random variables. Significant progress has been made through modelling the effect as a function of its cause and a noise term, which allows us to leverage assumptions about the generating function class. The recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified. However, through an extensive empirical evaluation, we demonstrate that the accuracy deteriorates sharply when the form of the noise distribution is misspecified by the user. Our analysis shows that the failure occurs mainly when the conditional variance in the anti-causal direction is smaller than that in the causal direction. As an alternative, we find that causal model selection through residual independence testing is much more robust to noise misspecification and misleading conditional variance.

Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing

Abstract

A fundamental problem of causal discovery is cause-effect inference, learning the correct causal direction between two random variables. Significant progress has been made through modelling the effect as a function of its cause and a noise term, which allows us to leverage assumptions about the generating function class. The recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified. However, through an extensive empirical evaluation, we demonstrate that the accuracy deteriorates sharply when the form of the noise distribution is misspecified by the user. Our analysis shows that the failure occurs mainly when the conditional variance in the anti-causal direction is smaller than that in the causal direction. As an alternative, we find that causal model selection through residual independence testing is much more robust to noise misspecification and misleading conditional variance.

1 Introduction

Distinguishing cause and effect is a fundamental problem in many disciplines, such as biology, healthcare and finance [31, 8, 13]. Randomized controlled trials (RCTs) are the gold standard for finding causal relationships. However, it may be unethical, expensive or even impossible to perform RCTs in real-world domains [20, 18]. Causal discovery algorithms aim to find causal relationships given observational data alone. Traditional causal discovery algorithms can only identify causal relationships up to Markov equivalence classes (MECs) [26, 10, 3]. To break the symmetry in a MEC, additional assumptions are needed [20, 22], such as the type of functional dependence of effect on cause. Structural causal models (SCMs) specify a functional class for the causal relations in the data [17, 20]. In this work, we focus on one particular type of SCM called location-scale noise models (LSNMs) [29, 11, 30, 28, 9]:

where X is the cause, Y is the effect, written X → Y , and Z Y is a latent noise variable independent of X (i.e., X ⊥ ⊥ Z Y ). The functions f and g are twice-differentiable on the domain of X , and g is strictly positive on the domain of X . LSNMs generalize the widely studied additive noise models (ANMs), where g ( x ) = 1 for all x . ANMs assume a constant conditional variance V [ Y | X ] for different values of X , which can limit their usefulness when dealing with heteroscedastic noise in real-world data. LSNMs allow for heteroscedasticity, meaning that V [ Y | X ] can vary depending on the value of X . In our experiments on real-world data, methods assuming an LSNM outperform those assuming ANM.

We follow previous work on cause-effect inference and focus on the bivariate setting [29, 11, 30, 9]. There are known methods for leveraging bivariate cause-effect inference in the multivariate setting (see Appendix A). Two major methods for selecting a causal SCM direction in cause-effect inference are maximum likelihood (ML) and independence testing (IT) of residuals vs. the (putative) cause [14]. Both have been recently shown good accuracy with LSNMs [11, 9], especially when the f and g functions are estimated by neural networks. Immer et al. [9] note, however, that ML can be less robust than IT, in the sense that accuracy deteriorates when the noise distribution is not Gaussian.

In this paper we investigate the robustness of ML vs. IT for LSNMs. Our analysis shows that ML cause-effect inference performs poorly when two factors coincide: (1) Noise Misspecification : the ML method assumes a different form of the noise distribution from the true one. (2) Misleading Conditional Variances (CVs) : V [ Y | X ] > V [ X | Y ] in the data generated by causal direction X → Y . For example, in the experiment on synthetic datasets shown in Table 1 below, (i) changing the true noise distribution from Gaussian to uniform and (ii) manipulating V [ Y | X ] , while keeping other settings equal, can decrease the rate of identifying the true causal direction from 100% to 10%. In contrast, IT methods maintain a perfect 100% accuracy. The difference occurs with and without data standardization, and with increasing sample sizes.

Both conditions (1) and (2) often hold in practice. For real-world domains, assumptions about the noise distribution can be hard to determine or verify. It is also common to have misleading CVs in real-world datasets. For example, in the Tübingen Cause-Effect Pairs benchmark [14], about 40% of the real-world datasets exhibit a misleading CV (see Table 7 in the appendix).

We make the following contributions to understanding location-scale noise models (LSNMs):

Describe experiments and theoretical analysis to show that ML methods succeed when the form of the noise distribution is known. In particular, ML methods are then robust to incorrectly specifying a noise variance parameter.

Demonstrate empirically that ML methods often fail when the form of the noise distribution is misspecified and CV is misleading, and analyze why.

Introduce a new IT method based on an affine flow LSNM model.

Demonstrate, both theoretically and empirically, that our IT method is robust to noise misspecification and misleading CVs.

The paper is structured as follows. We discuss related works and preliminaries in Section 2 and Section 3, respectively. Section 4 examines when and why ML methods fail. Section 5 demonstrates the robustness of the IT method. Experiments with 580 synthetic and 99 real-world datasets are given in Section 7. The code and scripts to reproduce all the results are given online 1 .

2 Related works

Causal Discovery. Causal discovery methods have been widely studied in machine learning [26, 10, 3]. Assuming causal sufficiency and faithfulness, they find causal structure up to a MEC. To identify causal relations within a MEC, additional assumptions are needed [20, 22]. SCMs exploit constraints that result from assumptions about the functional dependency of effects on causes. Functional dependencies are often studied in the fundamental case of two variables X and Y , the simplest MEC. Mooij et al. [14] provide an extensive overview and evaluation of different cause-effect methods in the ANM context. We follow this line of work for LSNMs with possibly misspecified models.

Structural Causal Models. Assuming a linear non-Gaussian acyclic model (LiNGAM) [24, 25], the causal direction was proved to be identifiable. The key idea in DirectLiNGAM [25] is that in the true causal direction X → Y , the model residuals are independent of X . We refer to methods derived from the DirectLiNGAM approach as independence testing (IT) methods. The more general ANMs [7, 19] allow for nonlinear cause-effect relationships and are generally identifiable, except for some special cases. There are other identifiable SCMs, such as causal additive models (CAM) [2], post-nonlinear models [32] and Poisson generalized linear models [16].

LSNM Identifiability. There are several identifiability results for the causal direction in LSNMs. Xu et al. [30] prove identifiability for LSNMs with linear causal dependencies. Khemakhem et al. [11]

Misleading CVs?

Low

Low

ANM

Yes

High

High

ANM

No

Yes

Misleading CVs?

High

Low

ANM

Yes

High

High

ANM

No

No

show that nonlinear LSNMs are identifiable with Gaussian noise. Strobl and Lasko [28], Immer et al. [9] prove LSNMs are identifiable except in some pathological cases (see Appendix C).

Cause-Effect Inference in LSNMs. The generic model selection blueprint is to fit two LSNM models for each direction X → Y and X ← Y and select the direction with a higher model score. HECI [30] bins putative cause values and selects a direction based on the Bayesian information criterion. BQCD [29] uses nonparamatric quantile regression to approximate minimum description length to select the direction. GRCI [28] finds the direction based on mutual information between the putative cause and the model residuals. LOCI [9] models the conditional distribution of effect given cause with Gaussian natural parameters. Then, it chooses the direction based on either likelihood (LOCI-M) or independence (LOCI-H). CAREFL-M [11] fits affine flow models and scores them by likelihood. CAREFL-M is more general than LOCI since LOCI uses a fixed Gaussian prior, whereas CAREFL-M can utilize different prior distributions. DECI [4] generalizes CAREFL-M to multivariate cases for ANMs. Unlike our work, none of these papers provide a theoretical analysis of noise misspecification and misleading CVs in LSNMs.

Noise Misspecification. Schultheiss and Bühlmann [23] present a theoretical analysis of model misspecification pitfalls when the data are generated by an ANM. Their results focus on model selection with Gaussian likelihood scores, which use Gaussian noise distributions. They suggest that IT-based methods may be more robust to model misspecification in ANMs. The authors also briefly discuss the complication of fitting data generated by a ground-truth ANM model with an LSNM model. Since the backward model of an ANM often takes the form of an LSNM, fitting data generated by a ground-truth ANM with an LSNM model may result in an incorrect causal direction. While their analysis examines ANM data, the ground-truth model in our work is an LSNM. The g () scale term in Equation (1), which is absent in ANMs, plays a critical role in LSNMs.

Conditional Variance and Causality. To the best of our knowledge, the problem of misleading CVs has not been identified nor analyzed in previous work for LSNMs. Prior studies that have focused on CVs were based on ANMs, and stated assumptions about how CVs relate to causal structure, to be leveraged in cause-effect inference [1, 15]. We do not make assumptions about CVs, but analyze them to understand why noise misspecification impairs cause-effect inference in LSNMs.

3 Preliminaries

In this section, we define the cause-effect inference problem and review the LSNM data likelihood.

3.1 Problem definition: cause-effect inference

Cause-effect inference takes as input a dataset D over two random variables X,Y with N observational data pairs ( X,Y ) = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . . , ( x N , y N ) } generated by a ground-truth LSNM in Equation (1). The binary output decision indicates whether X causes Y ( X → Y ) or Y causes X ( X ← Y ). We assume no latent confounders, selection bias or feedback cycles ([14, 29, 11, 9, 30]).

3.2 Definition of maximum likelihood for LSNMs

An LSNM model for two variables ( X,Y ) is a pair ( → , P Z ) . The → represents the direction X → Y with parameter space f ≡ f θ , g ≡ g ψ , P X ≡ P X,ζ . The ← represents the direction X ← Y with parameter space h ≡ h θ ′ , k ≡ k ψ ′ , P Y ≡ P Y,ζ ′ . For notational simplicity, we treat the model functions directly as model parameters and omit reference to their parameterizations. For example, we write f ≡ f θ for a function f that is implemented by a neural network with weights θ . We refer to P Z as the model prior noise distribution.

A parameterized LSNM model defines a data distribution as follows [9] (Appendix B):

For conciseness, we use shorter notation P → ,P Z ( X,Y ) and P ← ,P Z ( X,Y ) in later sections. The ML method computes the ML parameter estimates and uses them to score the → (or ← ) model:

4 Non-identifiability of LSNMs with noise misspecification

In this section, we show that ML methods fail under noise misspecification and misleading CVs, and analyse why. Figure 1 summarizes the diverse settings considered below.

Existing identifiability results for LSNM ML methods (Appendix C) require knowing the ground-truth noise distribution. We conducted a simple experiment to show that with noise misspecification, ML model selection can fail badly even on large sample sizes. Table 1 shows results for CAREFL-M [11] and LOCI-M [9] with model prior distribution Gaussian (0 , 1) . For the left half of the table, the data

was generated with correctly specified Gaussian (0 , 1) noise. In this case, both CAREFL-M and LOCI-M work well, and increasing CV in the causal direction does not affect their accuracy. For the right half of the table, the data was generated with misspecified uniform ( -1 , 1) noise. With noise misspecification, the accuracy of both CAREFL-M and LOCI-M decreases to 70% . With both noise misspecification and misleading CVs, their accuracy becomes even lower. For example, in the last column when V [ Y | X ] = 0 . 994 and V [ X | Y ] = 0 . 677 , they give an accuracy of just 10% .

The following results help explain why ML methods fail under noise misspecification and misleading CVs.

̸

Lemma 4.1. For an LSNM model Y := f ( X ) + g ( X ) · Z Y where Z Y is the noise with mean µ and variance σ 2 = 0 , there exists a likelihood-equivalent LSNM Model Y := f ′ ( X ) + g ′ ( X ) · Z ′ Y such that Z ′ Y = Z Y -µ σ , P → ,P Z ( Y | X ; f, g ) = P → ,P Z ′ ( Y | X ; f ′ , g ′ ) , and V Z Y [ Y | X ] = V Z ′ Y [ Y | X ] .

The proof is in Appendix E. Lemma 4.1 shows that for any (non-deterministic) LSNM model there is an equivalent standardized LSNM model with V [ Z Y ] = 1 .

The proof is in Appendix F. Theorem 4.2 shows that the model likelihood P → ,P Z ( Y | X ) and the conditional variance V [ Y | X ] are negatively related (similarly for P ← ,P Z ( X | Y ) and V [ X | Y ] ).

Corollary 4.3. Let → denote the causal model and ← denote the anti-causal model. When V [ Y | X ] increases and V [ X | Y ] decreases in the data, the log-likelihood model difference log P → ,P Z ( X,Y ) -log P ← ,P Z ( X,Y ) decreases.

The proof is in Appendix G. Under the identifiability assumption [11, 9] that ML is consistent under correct model specification, Lemma 4.1 also implies that ML remains consistent under misspecifying the variance of the noise distribution, as long as the form of the prior distribution matches that of the noise distribution. Table 4a in the appendix demonstrates this relationship empirically.

The overall relationship between likelihood and CV with respect to noise specification in LSNMs is as follows, illustrated in Figure 2 with actual datasets:

(Theorem 4.2) CV and likelihood are negatively related (Figures 2a and 2b), with either correctly or incorrectly specified noise distribution form.

(Identifiability) When the form of the noise distribution is correctly specified, the causal model always has a higher likelihood (Figure 2a).

(Corollary 4.3) When the form of the noise distribution is misspecified, the anti-causal model can have a higher likelihood under misleading CVs (Figure 2b).

ANM vs. LSNM. The relationship (2) is in general different for ANMs, when the noise variance is misleading (see Figure 1 and Appendix P). Proposition H.1 shows that the standardization lemma Lemma 4.1 does not apply to ANMs (confirmed by empirical results in Table 4b in the appendix).

5 Robustness of independence testing

This section describes IT methods for cause-effect learning in LSNMs, including a new IT method based on affine flows. Theoretical results below explain why IT methods are robust to noise misspecification and misleading CVs in LSNMs.

5.1 The independence testing method

Inspired by the breakthrough DirectLiNGAM approach [25], independence testing has been used in existing methods for various SCMs [7, 19, 28, 9]. Like ML methods (Equation (4)), IT methods fit the model parameters in both directions, typically maximizing the data likelihood (Equation (3)). The difference is in the model selection step, where IT methods select the direction with the highest degree of independence between the fitted model residuals and the putative cause.

Algorithm 1 is the pseudo-code for our IT method CAREFL-H . We fit the functions f and g in Equation (1), with the affine flow estimator T from CAREFL-M [11], implemented using neural

2.00

1.75

1.50

1.25

1.00

g-Likelihood:

0.75

0.50

0.25

0.00

0.2

0.4

0.6

0.8

1.0

1.2

1.4

MeanCV:Causal/Anti-Causal

(a) Log-likelihood difference under correct noise specification: Gaussian (0 , 1) noise

i-Causal

0.000

Anti-

-0.005

1

-0.010

Causal

-0.015

Value:

-0.020

-0.025

S

0.2

0.4

0.6

0.8

1.0

1.2

1.4

MeanCV:Causal/Anti-Causal

(c) HSIC difference under correct noise specification: Gaussian (0 , 1) noise

0.6

0.5

0.4

0.3

0.2

0.1

0.0

-0.1

-0.2

0.4

0.6

0.8

1.01.2

1.41.6

MeanCV:Causal/Anti-Causal

(b) Log-likelihood difference under noise misspecification: uniform ( -1 , 1) noise

Anti-Causal

0.0000

-0.0005

1

-0.0010

lesne

-0.0015

Value:

-0.0020

-0.0025

S

-0.0030

0.40.60.81.01.21.41.6

MeanCV:Causal/Anti-Causal

(d) HSIC difference under noise misspecification: uniform ( -1 , 1) noise

networks. For details on CAREFL-M and learning the flow transformation T see Appendix I. This combination of affine flow model with IT appears to be new. Another IT method is to test the independence of both residuals [6] (Appendix J). We found that this performs similarly to CAREFL-H and therefore report results only for the more common DirectLiNGAM-style method.

As in previous work [14], we use the Hilbert-Schmidt independence criterion (HSIC) [5] to measure (in)dependence throughout the paper. HSIC measures the squared distance between the joint probability of the two variables and the product of their marginals embedded in the reproducible kernel Hilbert space. We have HSIC ( U, V ) = 0 if and only if U ⊥ ⊥ V .

5.2 Theoretical comparison between IT and ML under noise misspecification in LSNMs

The intuition for why independence testing works is that if LSNM training is consistent, testing the independence of the putative cause and the residual will indicate the correct causal model. The next theorem provides a formal statement of this intuition.

Theorem 5.1. For data pairs ( X,Y ) generated by an LSNM model X → Y , let ̂ f ( X ) denote the conditional mean estimator and ̂ g ( X ) denote the conditional standard deviation estimator. Under the consistency conditions that ̂ f ( X ) = E [ Y | X ] and ̂ g ( X ) = STD [ Y | X ] , the reconstructed noise is independent of the cause, i.e. ̂ Z Y ⊥ ⊥ X , even under noise misspecification.

Proof. According to Equation (1),

Note that the proof does not involve the specification of the noise prior distribution P Z Y , which shows that under the consistency condition, the conclusion ̂ Z Y ⊥ ⊥ X holds whether the noise prior distribution is misspecified or not.

Algorithm 1 CAREFL-H

1: Input: data pairs D := ( X,Y ) , the flow estimator T of CAREFL-M with prior P Z , and an HSIC estimator 2: Output: estimated causal direction dir 3: Split D into training set D train := ( X train , Y train ) and testing set D test := ( X test , Y test ) 4: Optimize T ̂ θ, ̂ ψ, ̂ ζ ( D train ; P Z ) in X → Y direction via ML to estimate ̂ f and ̂ g 5: Compute the residual ̂ Z Y := Y test -̂ f ( X test ) ̂ g ( X test ) 6: Optimize T ̂ θ ′ , ̂ ψ ′ , ̂ ζ ′ ( D train ; P Z ) in X ← Y direction via ML to estimate ̂ h and ̂ k 7: Compute the residual ̂ Z X := X test -̂ h ( Y test ) ̂ k ( Y test ) 8: if HSIC ( X test , ̂ Z Y ) < HSIC ( Y test , ̂ Z X ) then 9: dir := X → Y 10: else if HSIC ( X test , ̂ Z Y ) > HSIC ( Y test , ̂ Z X ) then 11: dir := X ← Y 12: else 13: dir := no conclusion 14: end if

In contrast, the likelihood of a causal model depends on the specification of the noise prior distribution P Z Y (Equation (2)). This explains why IT methods are more robust than ML methods under noise misspecification in LSNMs.

In the context of ANMs, Mooij et al. [14] provide another sufficient condition for ̂ Z Y ⊥ ⊥ X called suitability. In Appendix K, we show how the suitability concept can be adapted for LSNMs to provide another argument for the robustness of IT methods.

6 Limitation of IT-based methods

Limitations of IT-based methods include the following: 1) When the noise prior distribution is correctly specified, ML-based methods can utilize this information to be more sample efficient. Conversely, since IT-based methods are less sensitive to the noise prior distribution, they require a larger sample size to infer causal direction accurately. 2) IT-based methods have higher computational cost, especially with nonlinear independence testing. 3) IT-based methods require high-capacity function estimators to model the unknown functions f and g for a good noise reconstruction.

Despite these complications, we advocate the use of IT-based methods over ML-based methods under noise misspecification and misleading CVs in LSNMs, because they are more reliable in such settings.

7 Experiments

On synthetic datasets, we find that across different hyperparameter choices the IT method (CAREFLH) produces much higher accuracy than the ML method (CAREFL-M) in the difficult settings with noise misspecification and misleading CVs, and produces comparable accuracy in the easier settings without noise misspecification or misleading CVs. On real-world data where the ground-truth noise distribution is unknown, the IT method is also more robust across different hyperparameter choices.

For all experiments, we start with the same default hyperparameter values for both CAREFL-M and CAREFL-H and alter one value at a time. The default hyperparameter values are those specified in CAREFL-M [11] for the Tübingen Cause-Effect Pairs benchmark [14]. Please see Appendix L for more details on default and alternative hyperparameter values. Previous work [14, 9] reported that MLmethods perform better with data splitting (split data into training set for model fitting and testing set for model selection) and IT methods perform better with data recycling (the same data is used for both model fitting and selection). Therefore, we use both splitting methods: (i) CAREFL(0.8): 80% as training and 20% as testing. (ii) CAREFL(1.0): training = testing = 100% . The training procedure is not supervised: no method accesses the ground-truth direction.

0.8

0.8

0.8

0.8

0.6

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.4

2

5

10

20

1

4

7

10

500

750

10002000

0.0

0.0001 0.001

0.1

laplace

gaussian

NumberofHiddenNeurons

Number ofSub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (O.8)

CAREFL-M (1.0)

CAREFL-H (O.8)

CAREFL-H (1.0)

CAREFL-H (1.0)

(a) Accuracy with SIM benchmarks

(b) Weighted accuracy with Tübingen Cause-Effect Pairs benchmark

We use a consistent HSIC estimator with Gaussian kernels [21]. A summary of experimental datasets is provided in Appendix Table 7. All the datasets are standardized to have mean 0 and variance 1. The datasets in Section 7.1 are generated by LSNMs and the datasets in Sections 7.2 and 7.3 are not. Consequently, the results demonstrate that the proposed method performs well not only when the ground-truth SCMs are strictly LSNMs, but also when the ground-truth SCMs are unknown or not strictly LSNMs.

7.1 Results on synthetic LSNM data

See Appendix M for the definition of the ground-truth LSNM SCMs and details on how synthetic datasets are generated from them. The sample sizes in each synthetic dataset are 500 or 5,000. As shown in Appendix Table 7, most synthetic datasets generated by such SCMs have misleading CVs. Based on the analysis of Section 4 we formulate the following hypotheses. (i) CAREFL-M should be accurate given a correct specification on the form of noise distribution, with or without misleading CVs. (ii) With noise misspecification and mild misleading CVs, the accuracy of CAREFL-M should be reduced. (iii) With noise misspecification and severe misleading CVs, the accuracy should be very low, often below 50%. Overall, the results from the experiments confirm our hypotheses.

7.1.1 Noise misspecification

We evaluate CAREFL-M and CAREFL-H against data generated with uniform ( -1 , 1) , exponential (1) , continuousBernoulli (0 . 9) or beta (0 . 5 , 0 . 5) noise, covered by our identifiability Theorem C.2 (except beta (0 . 5 , 0 . 5) ). Khemakhem et al. [11] empirically find that CAREFL-M with a Laplace model prior is robust with similar ground-truth noise distributions such as Gaussian and Student's t. We show that it may fail remarkably with a dissimilar distribution.

Wesummarize findings from the 336 settings here; the detailed results are given in Appendix Figures 6 to 17. In 289 settings (86.01%), both CAREFL-M(0.8) and CAREFL-M(1.0) select the correct causal

direction with less than 50% random accuracy. Furthermore, in 110 settings (32.74%) both CAREFLM(0.8) and CAREFL-M(1.0) fail catastrophically with an accuracy of 0%. These experiments also show that the accuracy of CAREFL-M often decreases as N increases. In contrast, CAREFL-H(1.0) achieves better accuracy than CAREFL-M in 333 settings (99.11%). The accuracy of CAREFLH(1.0) goes below 50% in only 6 settings (1.79%). The results demonstrate the robustness of the IT method under noise misspecification and misleading CVs, across different hyperparameter choices.

Appendix Table 7 and Appendix Figure 17 show that with severe misleading CVs, the accuracy of CAREFL-M is close to 0%. This is much lower than the corresponding cases with mild misleading CVs in Appendix Figures 9 and 13.

7.1.2 Correct specification on the form of the noise distribution

These experiments show that CAREFL-H is comparable with CAREFL-M under correct form of noise specification, with or without misleading CVs, especially on larger datasets, as long as the affine model capacity is sufficient. We evaluate CAREFL-M and CAREFL-H against data generated with Gaussian (0 , 1) and Laplace (0 , 1) noise. The detailed results are in Appendix Figures 18 to 23. We find that CAREFL-M is more sample efficient than CAREFL-H when the model prior matches the data, which is a general pattern for ML vs. IT methods [23]. Consistent with the suitability results in Appendix K.1, the accuracy of CAREFL-H improves with more data. For example, with N = 500 , there are 49 out of 84 settings (58.33%) where CAREFL-M outperforms CAREFL-H(1.0). However, with N = 5 , 000 , CAREFL-H(1.0) achieves similar accuracy as CAREFL-M on all datasets (except LSNM-sigmoid-sigmoid with Laplace (0 , 1) noise.) In addition, CAREFL-H(1.0) may underperform CAREFL-M when the number of hidden neurons, sub-flows or training epochs is low. An IT method requires more model capacity to fit the LSNM functions and produce a good reconstruction of the noise.

7.2 Results on synthetic benchmarks

Similar to Tagasovska et al. [29], Immer et al. [9], we compare CAREFL-M and CAREFL-H against the SIM benchmark suite [14]. SIM comprises 4 sub-benchmarks: default (SIM), with one confounder (SIM-c), low noise levels (SIM-ln) and Gaussian noise (SIM-G). In this benchmark, most datasets do not have misleading CVs (Appendix Table 7), which favors ML methods. Each sub-benchmark contains 100 datasets and each dataset has N = 1000 data pairs. As shown in Appendix Figure 24, CAREFL-M and CAREFL-H(1.0) achieve similar accuracy on SIM-ln and SIM-G across different hyperparameter choices. For SIM and SIM-c, CAREFL-H, especially CAREFL-H(1.0), outperforms CAREFL-M by 20%-30% in all settings. The accuracy of CAREFL-M is only about random guess (40%-60%) on SIM and SIM-c.

Table 2a compares CAREFL-H with SOTA methods. For each CAREFL method, we report the best accuracy obtained in Appendix Figure 24 without further tuning. CAREFL-H achieves the best accuracy on SIM and SIM-c, and achieves competitive accuracy on SIM-ln and SIM-G.

7.3 Results on real-world benchmarks: Tübingen Cause-Effect Pairs

The Tübingen Cause-Effect Pairs benchmark [14] is commonly used to evaluate cause-effect inference algorithms [11, 30, 9]. To be consistent with previous work [29, 28, 9], we exclude 6 multivariate and 3 discrete datasets (#47, #52-#55, #70, #71, #105, #107) and utilize the remaining 99 bivariate datasets. As recommended by Mooij et al. [14], we report weighted accuracy. 40% of datasets in the benchmark feature misleading CVs. CAREFL-H(1.0) outperforms CAREFL-M in all configurations by large margins (7%-30%) ; see Figure 3. We also compare CAREFL-H with SOTA methods (see Appendix N for hyperparameters). Table 2b shows that CAREFL-H achieves the SOTA accuracy (82%) and is 9% more accurate than CAREFL-M [11].

Furthermore, LSNM methods (i.e. LOCI, GRCI, BQCD, HECI, CAREFL) perform better than ANM methods (i.e. CAM, RESIT). The reason is that LSNM is a weaker assumption than ANM and allows for heteroscedastic noise.

8 Conclusion and future work

We identified a failure mode of maximum-likelihood (ML) methods for cause-effect inference in location-scale noise models (LSNMs). Our analysis shows that the failure occurs when the form of the noise distribution is misspecified and conditional variances are misleading (i.e., higher in the causal direction). Selecting causal models by independence tests (IT) is robust even in this difficult setting. Extensive empirical evaluation compared the ML method and a new IT method based on affine flows, on both synthetic and real-world datasets. The IT flow method achieves better accuracy under noise misspecification and misleading CVs, with robust performance across different hyperparameter choices. Future directions include improving the sample efficiency of IT methods, and improving the robustness of ML methods by learning the noise distribution instead of using a fixed prior.

9 Acknowledgments

This research was supported by a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC).

References

Patrick Blöbaum, Dominik Janzing, Takashi Washio, Shohei Shimizu, and Bernhard Schölkopf. Cause-effect inference by comparing regression errors. In International Conference on Artificial Intelligence and Statistics , pages 900-909. PMLR, 2018.

Peter Bühlmann, Jonas Peters, and Jan Ernest. Cam: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics , 42(6):2526, 2014.

Diego Colombo, Marloes H Maathuis, Markus Kalisch, and Thomas S Richardson. Learning high-dimensional directed acyclic graphs with latent and selection variables. The Annals of Statistics , pages 294-321, 2012.

Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.02195 , 2022.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pages 63-77. Springer, 2005.

Yue He, Peng Cui, Zheyan Shen, Renzhe Xu, Furui Liu, and Yong Jiang. Daring: Differentiable causal discovery with residual independence. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 596-605, 2021.

Patrik Hoyer, Dominik Janzing, Joris M Mooij, Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems , 21, 2008.

Biwei Huang. Diagnosis of autism spectrum disorder by causal influence strength learned from resting-state fmri data. In Neural Engineering Techniques for Autism Spectrum Disorder , pages 237-267. Elsevier, 2021.

Alexander Immer, Christoph Schultheiss, Julia E Vogt, Bernhard Schölkopf, Peter Bühlmann, and Alexander Marx. On the identifiability and estimation of causal location-scale noise models. In International Conference on Machine Learning , pages 14316-14332. PMLR, 2023.

Markus Kalisch and Peter Bühlman. Estimating high-dimensional directed acyclic graphs with the pc-algorithm. Journal of Machine Learning Research , 8(3), 2007.

Ilyes Khemakhem, Ricardo Monti, Robert Leech, and Aapo Hyvarinen. Causal autoregressive flows. In International conference on artificial intelligence and statistics , pages 3520-3528. PMLR, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Mehrdad Mansouri, Sahand Khakabimamaghani, Leonid Chindelevitch, and Martin Ester. Aristotle: stratified causal discovery for omics data. BMC bioinformatics , 23(1):1-18, 2022.

Joris M Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research , 17(1):1103-1204, 2016.

Gunwoong Park. Identifiability of additive noise models using conditional variances. The Journal of Machine Learning Research , 21(1):2896-2929, 2020.

Gunwoong Park and Sion Park. High-dimensional poisson structural equation model learning via \ ell_1-regularized regression. J. Mach. Learn. Res. , 20:95-1, 2019.

Judea Pearl. Causality . Cambridge university press, 2009.

Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect . Basic books, 2018.

Jonas Peters, Joris M Mooij, Dominik Janzing, and Bernhard Schölkopf. Causal discovery with continuous additive noise models. Journal of Machine Learning Research , 15:2009-2053, 2014.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Niklas Pfister, Peter Bühlmann, Bernhard Schölkopf, and Jonas Peters. Kernel-based tests for joint independence. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 80(1):5-31, 2018.

Bernhard Schölkopf. Causality for machine learning. In Probabilistic and Causal Inference: The Works of Judea Pearl , pages 765-804. 2022.

Christoph Schultheiss and Peter Bühlmann. On the pitfalls of gaussian likelihood scoring for causal discovery. Journal of Causal Inference , 11(1):20220068, 2023.

Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-Gaussian acyclic model for causal discovery. Journal of Machine Learning Research , 7(10), 2006.

Shohei Shimizu, Takanori Inazumi, Yasuhiro Sogawa, Aapo Hyvarinen, Yoshinobu Kawahara, Takashi Washio, Patrik O Hoyer, Kenneth Bollen, and Patrik Hoyer. DirectLiNGAM: A direct method for learning a linear non-Gaussian structural equation model. Journal of Machine Learning Research-JMLR , 12(Apr):1225-1248, 2011.

Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review , 9(1):62-72, 1991.

Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search . MIT press, 2000.

Eric V Strobl and Thomas A Lasko. Identifying patient-specific root causes with the heteroscedastic noise model. Journal of Computational Science , 72:102099, 2023.

Natasa Tagasovska, Valérie Chavez-Demoulin, and Thibault Vatter. Distinguishing cause from effect using quantiles: Bivariate quantile causal discovery. In International Conference on Machine Learning , pages 9311-9323. PMLR, 2020.

Sascha Xu, Osman A Mian, Alexander Marx, and Jilles Vreeken. Inferring cause and effect in the presence of heteroscedastic noise. In International Conference on Machine Learning , pages 24615-24630. PMLR, 2022.

Kun Zhang and Lai-Wan Chan. Extensions of ICA for causality discovery in the hong kong stock market. In International Conference on Neural Information Processing , pages 400-409. Springer, 2006.

Kun Zhang and Aapo Hyvarinen. On the identifiability of the post-nonlinear causal model. arXiv preprint arXiv:1205.2599 , 2012.

Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning. Advances in Neural Information Processing Systems , 31, 2018.

Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics , pages 3414-3425. PMLR, 2020.

A Extension to multivariate settings

To extend a bivariate cause-effect inference method to multivariate, the most common approach is to first use the PC algorithm [27] (or other algorithms) to learn a completed partially directed acyclic graph (CPDAG) that may contain undirected edges. Then, orient the undirected edges using the cause-effect inference method. This approach can be found in Tagasovska et al. [29], Khemakhem et al. [11], Xu et al. [30], Strobl and Lasko [28].

B Derivation of Equation (2)

Lemma B.1. For an LSNM model X → Y defined in Equation (1) , we have P Y | X ( Y | X ) = P Z Y ( Y -f ( X ) g ( X ) ) · 1 g ( X ) , where P Z Y is the noise distribution.

For the data distribution in the X → Y direction:

Similarly, in the X ← Y direction:

Proof for Lemma B.1. For an LSNM model defined in Equation (1), we have Z Y = Y -f ( X ) g ( X ) . Hence, ∂Z Y ∂Y = 1 g ( X ) . Since g ( X ) > 0 , so | ∂Z Y ∂Y | = | 1 g ( X ) | = 1 g ( X ) .

C Identifiability of LSNMs with correct noise distribution

Strobl and Lasko [28], Immer et al. [9] prove the identifiability of LSNMs, assuming a correctly specified noise distribution. That is, given that the data generating distribution ( X,Y ) follows an LSNM in the direction X → Y , the same distribution with equal likelihood cannot be induced by an LSNM in the backward direction X ← Y , except in some pathological cases. In terms of our notation, direction identifiability means that if ( → , P Z ) is the data generating model, then

Immer et al. [9] prove the following identifiability result:

Theorem C.1 (Theorem 1 from [9]) . For data ( X,Y ) that follows an LSNM in both direction X → Y and X ← Y , i.e.,

The following condition must be true:

̸

where G ( x, y ) = g ( x ) · f ′ ( x )+ g ′ ( x ) · [ y -f ( x )] = 0 and ν X | Y ( x | y ) = log p Z X ( x -h ( y ) k ( y ) ) -log k ( y ) .

They state that Equation (5) will be false except for 'pathological cases". In addition, Khemakhem et al. [11] provide sufficient conditions for LSNMs with Gaussian noise to be identifiable. Our next theorem provides identifiability results for some non-Gaussian noise distributions:

Theorem C.2. Suppose that the true data-generating distribution follows an LSNM model in both X → Y and X ← Y directions:

If the noise distribution is uniform ( a, b ) , then both g ( X ) and k ( Y ) are constant functions.

̸

If the noise distribution is continuousBernoulli ( λ = 0 . 5) 2 or exponential ( λ ) , then one of the following conditions holds:

1 g ( X ) and 1 k ( Y ) are constant functions.

1 g ( X ) and 1 k ( Y ) are linear functions with the same coefficients on X and Y , respectively.

The proof is in Appendix D. Essentially, the theorem shows that for uniform, exponential, and continuousBernoulli noise distributions, the true LSNM model can be identified unless it degenerates to (i) a homoscedastic additive noise model or (ii) a heteroscedastic model with the same linear scale in both directions.

D Identifiability proofs

In this section, we prove Theorem C.2.

If the data ( x, y ) follows an LSNM in the forward (i.e. causal) model:

where z Y is the noise term, x ⊥ ⊥ z Y , g ( x ) > 0 for all x on its domain. We assume f ( · ) and g ( · ) are twice-differentiable on the domain of x .

If the data ( x, y ) follows an LSNM in the backward (i.e. anti-causal) model:

where m X is the noise term, y ⊥ ⊥ m X , k ( y ) > 0 for all y on its domain. We assume h ( · ) and k ( · ) are twice-differentiable on the domain of y .

z Y and m X follow one of uniform ( a, b ) , exponential ( λ ) or continuousBernoulli ( λ ) distribution accordingly.

Proof for uniform ( a, b ) noise. For the causal model, according to Lemma B.1, we have P Y | X ( y | x ) = P Z Y ( y -f ( x ) g ( x ) ) · 1 g ( x ) = P Z Y ( z Y ) · 1 g ( x ) = 1 b -a · 1 g ( x ) = 1 ( b -a ) · g ( x ) . Similarly, for the backward model, we have P X | Y ( x | y ) = 1 ( b -a ) · k ( y ) .

The joint likelihood of the observation ( x, y ) in the causal model is:

The joint likelihood of the observation ( x, y ) in the backward model is:

If the data follows both models:

Take the derivative of both sides with respect to x :

Similarly, if we take the derivative of both sides with respect to y instead, we have:

These imply that both g ( x ) and k ( y ) are constant functions.

Proof for exponential ( λ ) noise. For the causal model, according to Lemma B.1, we have P Y | X ( y | x ) = P Z Y ( y -f ( x ) g ( x ) ) · 1 g ( x ) = λ · e -λ · y -f ( x ) g ( x ) · 1 g ( x ) = λ g ( x ) · e -λ g ( x ) · ( y -f ( x )) . Similarly, for the backward model, we have P X | Y ( x | y ) = λ k ( y ) · e -λ k ( y ) · ( x -h ( y )) .

The joint likelihood of the observation ( x, y ) in the causal model is:

The joint likelihood of the observation ( x, y ) in the backward model is:

If the data follows both models:

Take the derivative of both sides with respect to x :

Take the derivative of both sides with respect to y :

They can be equal only if both sides are constants. Therefore, 1 g ( x ) and 1 k ( y ) are both constants or both linear functions with the same coefficient on x and y , respectively.

̸

Proof for continuousBernoulli ( λ = 0 . 5) noise. Please refer to uniform for continuousBernoulli ( λ = 0 . 5) , which equals to uniform (0 , 1) . For the causal model, according to Lemma B.1, we have P Y | X ( y | x ) = P Z Y ( y -f ( x ) g ( x ) ) · 1 g ( x ) = C λ · λ y -f ( x ) g ( x ) · (1 -λ ) 1 -y -f ( x ) g ( x ) · 1 g ( x ) , where C λ is the normalizing constant of the continuous Bernoulli distribution. Similarly, for the backward model, we have P X | Y ( x | y ) = C λ · λ x -h ( y ) k ( y ) · (1 -λ ) 1 -x -h ( y ) k ( y ) · 1 k ( y ) .

The joint likelihood of the observation ( x, y ) in the causal model is:

The joint likelihood of the observation ( x, y ) in the backward model is:

If the data follows both models:

Take the derivative of both sides with respect to x :

Take the derivative of both sides with respect to y :

̸

̸

They can be equal only if both sides are constants. Therefore, 1 g ( x ) and 1 k ( y ) are both constants or both linear functions with the same coefficient on x and y , respectively.

E Proof for Lemma 4.1

Proof. According to Equation (2), we have P → ,P Z ( Y | X ; f, g ) = P Z Y ( Y -f ( X ) g ( X ) ) · 1 g ( X ) and P → ,P Z ′ ( Y | X ; f ′ , g ′ ) = P Z ′ Y ( Y -f ′ ( X ) g ′ ( X ) ) · 1 g ′ ( X ) . Let Z ′ Y be the standardized version of Z Y , i.e. Z ′ Y = Z Y -µ σ .

Let 1 g ( X ) = σ · 1 g ′ ( X ) and f ′ ( X ) = f ( X ) + µ · g ( X ) , then

Therefore, P → ,P Z ( Y | X ; f, g ) = P → ,P Z ′ ( Y | X ; f ′ , g ′ ) .

For conditional variances, we have:

V Z Y [ Y | X ] = g 2 ( X ) · V [ Z Y ] = g 2 ( X ) · σ 2 V Z ′ Y [ Y | X ] = g ′ 2 ( X ) · V [ Z ′ Y ] = g ′ 2 ( X ) · 1 = σ 2 · g 2 ( X ) Therefore, V Z Y [ Y | X ] = V Z ′ Y [ Y | X ] .

F Proof for Theorem 4.2

Proof. An LSNM model defined in Equation (1) entails the following relationships:

where Lemma 4.1 shows that we can always assume V [ Z Y ] = 1 . Also, since there is no causal relationship between X and Z Y in the ground-truth data, i.e., changing X does not change Z Y , we have ∂Z Y ∂X = 0 , ∂ log P Z Y ( Z Y ) ∂X = 0 and ∂ log P Z Y ( Z Y ) ∂g 2 ( X ) = 0 .

Since g ( X ) > 0 , i.e., strictly positive on the domain of X , therefore, ∂ log P → ,P Z ( Y | X ) ∂ V [ Y | X ] < 0 .

G Proof for Corollary 4.3

Proof. To compare the data log-likelihood under the causal model → and under the anti-causal model ← , we have:

Theorem 4.2 shows that conditional likelihood and conditional variance are negatively related. Therefore, by increasing V [ Y | X ] and decreasing V [ X | Y ] in the data, log P → ,P Z ( Y | X ) decreases and log P ← ,P Z ( X | Y ) increases. Therefore, log P → ,P Z ( X,Y ) decreases, log P ← ,P Z ( X,Y ) increases, and their difference decreases.

H The ANM version of Lemma 4.1

̸

Proposition H.1. Let M 1 be an ANM model Y := f ( X ) + Z Y , where Z Y is the noise with mean µ and variance σ 2 = 0 . Let M 2 be another ANM model Y := f ′ ( X ) + Z ′ Y , where Z ′ Y is the standardized version of Z Y , i.e. Z ′ Y = Z Y -µ σ . When both Z Y and Z ′ Y are Gaussian, P → ,P Z ( Y | X ; f ) = P → ,P Z ′ ( Y | X ; f ′ ) .

̸

Proof. ANM is a restricted case of LSNM with g ( X ) = 1 . Similar to Equation (2), we have P → ,P Z ( Y | X ; f ) = P Z Y ( Y -f ( X )) for M 1 and P → ,P Z ′ ( Y | X ; f ′ ) = P Z ′ Y ( Y -f ′ ( X )) for M 2 .

̸

We show that P Z Y ( Y -f ( X )) = P Z ′ Y ( Y -f ′ ( X )) when Z Y and Z ′ Y are Gaussian.

̸

Therefore, P → ,P Z ( Y | X ; f ) = P → ,P Z ′ ( Y | X ; f ′ ) in the case of Gaussian unless σ = 1 . Empirical results are provided in Table 4b and Appendix P.

I CAREFL-M

CAREFL-M [11] models an LSNM in Equation (1) via affine flows T . Each sub-flow T k ∈ T is defined as the following:

where X is the putative cause and Y is the putative effect in X → Y direction. t 1 and s 1 are constants. t 2 and s 2 are functions parameterized using neural networks. Without loss of generality, X is assumed to be a function of latent noise variable Z X . If t 1 = 0 and s 1 = 0 , then X = Z X . The exponential function e ensures the multipliers to Z are positive without expression loss. Similarly, for the backward direction X ← Y :

where Y is the putative cause and X is the putative effect in X ← Y direction. t ′ 1 and s ′ 1 are constants. t ′ 2 and s ′ 2 are functions parameterized using neural networks.

Given Equation (6), the joint log-likelihood of ( x, y ) in X → Y direction is:

Similarly for the X ← Y direction. Note that the priors P Z = { P Z X , P Z Y } in Equation (8) may mismatch the unknown ground-truth noise distribution P ∗ Z = { P ∗ Z X , P ∗ Z Y } . Both CAREFL-M and

Algorithm 2 CAREFL-M

1: Input: data pairs D := ( X,Y ) , and the flow estimator T with prior P Z

2: Output: estimated causal direction dir

3: Split D into training set D train := ( X train , Y train ) and testing set D test := ( X test , Y test )

4: Optimize T ̂ t 1 , s 1 , ̂ t 2 , s 2 ( D train ; P Z ) in X → Y direction via ML

̂

̂

5: Compute the likelihood ̂ L → ,P Z ( D test ; T ̂ t 1 , s 1 , ̂ t 2 , s 2 ) in X → Y direction

̂ ̂ 6: Optimize T ̂ t ′ 1 , ̂ s ′ 1 , ̂ t ′ 2 , ̂ s ′ 2 ( D train ; P Z ) in X ← Y direction via ML

7: Compute the likelihood ̂ L ← ,P Z ( D test ; T ̂ t ′ 1 , ̂ s ′ 1 , ̂ t ′ 2 , ̂ s ′ 2 ) in X ← Y direction

8: if ̂ L → ,P Z ( D test ; T ̂ t 1 , ̂ s 1 , ̂ t 2 , ̂ s 2 ) > ̂ L ← ,P Z ( D test ; T ̂ t ′ 1 , ̂ s ′ 1 , ̂ t ′ 2 , ̂ s ′ 2 ) then

9: dir := X → Y

10: else if ̂ L → ,P Z ( D test ; T ̂ t 1 , ̂ s 1 , ̂ t 2 , ̂ s 2 ) < ̂ L ← ,P Z ( D test ; T ̂ t ′ 1 , ̂ s ′ 1 , ̂ t ′ 2 , ̂ s ′ 2 ) then

11: dir := X ← Y

12: else

13: dir := no conclusion

14: end if

Algorithm 3 CAREFL-H (Between Residuals)

1: Input: data pairs D := ( X,Y ) , the flow estimator T of CAREFL-M with prior P Z , and an HSIC estimator

2: Output: estimated causal direction dir

3: Split D into training set D train := ( X train , Y train ) and testing set D test := ( X test , Y test )

4: Optimize T ̂ t 1 , s 1 , ̂ t 2 , s 2 ( D train ; P Z ) in X → Y direction via ML to estimate ̂ t 1 , ̂ s 1 , ̂ t 2 and ̂ s 2

̂ ̂ 5: Compute the residuals ̂ Z X, → := X test -̂ t 1 e ̂ s 1 and ̂ Z Y, → := Y test -̂ t 2 ( X ) e ̂ s 2 ( X )

6: Optimize T ̂ t ′ 1 , ̂ s ′ 1 , ̂ t ′ 2 , ̂ s ′ 2 ( D train ; P Z ) in X ← Y direction via ML to estimate ̂ t ′ 1 , ̂ s ′ 1 , ̂ t ′ 2 and ̂ s ′ 2

7: Compute the residuals ̂ Z Y, ← := Y test -̂ t ′ 1 e ̂ s ′ 1 and ̂ Z X, ← := X test -̂ t ′ 2 ( Y ) e ̂ s ′ 2 ( Y )

8: if HSIC ( ̂ Z X, → , ̂ Z Y, → ) < HSIC ( ̂ Z X, ← , ̂ Z Y, ← ) then

9: dir := X → Y

10: else if HSIC ( ̂ Z X, → , ̂ Z Y, → ) > HSIC ( ̂ Z X, ← , ̂ Z Y, ← ) then

11: dir := X ← Y

12: else

13: dir := no conclusion

14: end if

CAREFL-H optimize Equation (8) for each direction over the training set. For CAREFL-M, it chooses the direction with ML Score L over the testing set as the estimated causal direction. Detailed procedure for CAREFL-M is given in Algorithm 2. To map the parameters θ , ψ , ζ , θ ′ , ψ ′ and ζ ′ in LSNM (Equation (2)) to the flow estimator T of CAREFL (Equations (6) and (7)), we have f θ ≡ t 2 , g ψ ≡ e s 2 , P X,ζ ≡ { t 1 , e s 1 } , f θ ′ ≡ t ′ 2 , g ψ ′ ≡ e s ′ 2 and P Y,ζ ′ ≡ { t ′ 1 , e s ′ 1 } .

J CAREFL-H alternative independence testing

In Algorithm 1, CAREFL-H tests independence between the putative cause and the residual of the putative effect in each direction, i.e., between X and ̂ Z Y in X → Y direction, and between Y and ̂ Z X in X ← Y direction. An alternative way of testing independence is to test between the residual of the putative cause and the residual of the putative effect [6], i.e., between ̂ Z X and ̂ Z Y in both directions. Please see Algorithm 3 for complete steps.

Although in our experiments the two algorithms often produce the same estimation of causal direction, we prefer Algorithm 1, since it relies on few estimations of the residuals.

K Suitability theory

With a consistent HSIC estimator , Mooij et al. [14] show that an IT method consistently selects the causal direction for ANMs if the regression method is suitable . A regression method is suitable if the expected mean squared error between the predicted residuals ̂ E and the true residuals E approaches 0 in the limit of N →∞ :

where D and D ′ denote training set and testing set, respectively. Hence, with enough data a suitable regression method reconstructs the ground-truth noise.

If an HSIC estimator is consistent, the estimated HSIC value converges in probability to the population HSIC value.

With a consistent HSIC estimator and a suitable regression method, the consistency result for ANMs in Mooij et al. [14] extends naturally to LSNMs (see Appendix K.2 for a proof outline).

Proposition K.1. For identifiable LSNMs with an independent noise term in one causal direction only, if an IT method is used with a suitable regression method for LSNMs and a consistent HSIC estimator, then the IT method is consistent for inferring causal direction for LSNMs.

It is important to note that, while the suitability of regression methods is a valuable property for IT-based methods aiding in the determination of causal directions using independence testing, it does not guarantee a higher likelihood for the causal direction compared to the anti-causal direction under noise misspecification. Therefore, suitability does not offer the same robustness guarantee for ML-based methods.

K.1 Suitability: empirical results

Table 5: Suitability of the flow estimator T of CAREFL-M in the causal direction under noise misspecification and misleading CVs. T is trained with a Laplace prior. The original dataset with size 2 N is split into two: 50% as training set and 50% as testing set. V [ Y | X ] > V [ X | Y ] indicates misleading CVs in the dataset. LSNM definition is in Appendix M.

(a) LSNM-tanh-exp-cosine and continuousBernoulli ( 0 . 9 ) noise. V [ Y | X ] vs. V [ X | Y ] : 0.324 vs. 0.291.

Let suitability value S be the left-hand side of Equation (9). Table 5 shows an empirical evaluation of S , for the flow estimator T in the causal direction. We generate data from 3 synthetic LSNMs, and evaluate T under noise misspecification and misleading CVs. We find that as the sample size grows, S approaches 0. In other words, T is empirically suitable under noise misspecification and misleading CVs. Therefore, Proposition K.1 entails that CAREFL-H based on T and a consistent HSIC estimator is empirically consistent for inferring causal direction in LSNMs under these conditions.

Because T in CAREFL-M [11] uses neural networks to approximate the observed data distribution, it is difficult to provide a theoretical guarantee of suitability. Therefore, although our experiments indicate that T is often suitable in practice, we do not claim that it is suitable for all LSNMs. For

example, it may not be suitable in the low-noise regime when the LSNMs are close to deterministic. It is well-known that since neural network models are universal function approximators, we can expect them to fit many regression functions, although it is difficult to prove suitability theoretically. For these reasons, we do not claim a priori that flow models are suitable, but we provide empirical evidence that with careful training and hyperparameter selection, we can expect them to be suitable in practice.

K.2 Proof outline for Proposition K.1

For data pairs ( X,Y ) generated by an LSNM model X → Y , we have:

Under suitability, the reconstructed noise ̂ Z Y approaches the ground-truth noise Z Y .

By employing a consistent HSIC estimator, the estimated HSIC value ̂ HSIC converges to the true HSIC value HSIC .

̸

In the ground-truth LSNM, HSIC ( X, ̂ Z Y ) = 0 and HSIC ( Y, ̂ Z X ) = 0 (i.e. identifiability of LSNMs).

̸

Therefore, ̂ HSIC ( X, ̂ Z Y ) = 0 and ̂ HSIC ( Y, ̂ Z X ) = 0 for LSNMs. We only provide a proof outline here, since a formal proof is analogous to the proof for ANMs in Appendix A of Mooij et al. [14].

L Default and alternative hyperparameter values used in Section 7

We use the reported hyperparameter values in CAREFL-M [11] for the Tübingen Cause-Effect Pairs benchmark [14] as the default hyperparameter values in all our experiments:

The flow estimator T is parameterized with 4 sub-flows (alternatively: 1, 7 and 10).

For each sub-flow, f , g , h and k are modelled as four-layer MLPs with 5 hidden neurons in each layer (alternatively: 2, 10 and 20).

Prior distribution is Laplace (alternatively: Gaussian prior).

Adam optimizer [12] is used to train each model for 750 epochs (alternatively: 500, 1000 and 2000).

L2-penalty strength is 0 by default (alternatively: 0.0001, 0.001, 0.1).

Although we also observe that LOCI-H is more robust than LOCI-M under noise misspecification and misleading CVs, we omit their results because: (1) CAREFL often outperforms LOCI (see Tables 1 and 2). (2) LOCI is fixed as a Gaussian distribution, whereas CAREFL can specify different prior distributions. (3) LOCI uses different set of hyperparameters than CAREFL. So their results cannot be merged into the same figures.

M Synthetic SCMs used in Section 7.1

We use the following SCMs to generate synthetic datasets:

where Z Y is the ground-truth noise sampled from one of the following distributions: continuousBernoulli (0 . 9) , uniform ( -1 , 1) , exponential (1) , beta (0 . 5 , 0 . 5) , Gaussian (0 , 1) and Laplace (0 , 1) . σ is the sigmoid function. Although we did not prove identifiability with beta (0 . 5 , 0 . 5) noise in Theorem C.2, empirically we find it is identifiable. Following [33, 34], each θ and ψ are sampled uniformly from range [ -2 , -0 . 5] ∪ [0 . 5 , 2] . ϕ is sampled uniformly from range [1 , 2] to make the tanh function positive. The number of data pairs in each synthetic dataset is N ∈ { 500 , 5000 } .

N Hyperparameter values of CAREFL-H for Table 2b

Following Khemakhem et al. [11], we use a single set of hyperparameters for all 99 datasets, which is found by grid search. To acquire the result of CAREFL-H in Table 2b, the hyperparameter values used are as follows:

Number of hidden neurons in each layer of the MLPs: 2

Number of sub-flows: 10

Training dataset = testing dataset = 100%.

The rest of the hyperparameter values are identical to the default ones.

O Running time

The running time of CAREFL-H is slightly longer than CAREFL-M, due to the additional independence tests, i.e. HSIC ( X test , ̂ Z Y ) and HSIC ( Y test , ̂ Z X ) . Please see Table 6. The running time is measured on a computer running Ubuntu 20.04.5 LTS with Intel Core i7-6850K 3.60GHz CPU and 32 GB memory. No GPUs are used.

Table 6: Accuracy running time in seconds over 10 datasets generated by SCM LSNM-sine-tanh (definition in Appendix M) with N = 10 , 000 samples. α = 1 . Ground-truth noise distribution is uniform ( -1 , 1) . Model prior distribution is Gaussian (0 , 1) .

P Additive noise models under misleading conditional variances

We show the accuracy of ANM ML methods with ANM data under misleading CVs, and compare with LSNM ML methods. The ANM data is generated by the following SCMs:

where Z Y is the ground-truth noise sampled from one of the following distributions: uniform ( -1 , 1) and Gaussian (0 , 1) . σ is the sigmoid function. Following [33, 34], each θ is sampled uniformly from range [ -2 , -0 . 5] ∪ [0 . 5 , 2] .

CAREFL-ANM-M is the ANM counterpart of CAREFL-M (see Appendix I). The LSNM flows (see Equation (6)) in CAREFL-M are changed to model ANMs:

Similar to LSNM ML methods, the accuracy of ANM ML methods is also negatively related to misleading CVs, as shown in Figure 5. Unlike LSNM ML methods, which suffer from misleading CVs under noise misspecification (see Table 1 and Figures 2 and 4), an ANM ML method suffers from misleading CVs with either noise misspecification or correct specification, as illustrated in Figure 5. Figure 1 summarizes the differences between ANM and LSNM ML methods.

(a) LSNM-sine-tanh and uniform ( - 1 , 1) noise

(b) LSNM-sine-tanh and Gaussian (0 , 1) noise

(c)

and

uniform

(

-

1

,

1)

noise

(d) LSNM-tanh-exp-cosine and Gaussian (0 , 1) noise

(e)

and

uniform

(

-

1

,

1)

noise

(f)

and

(0

,

1)

noise

1.0

X

CAREFL-M

LOCI-M

0.8

0.6

0.4

0.2

0.0

30

40

50

60

70

8090100

Percentage ofDatasetsWithMisleading CVs(%)

1.0

0.8

0.6

0.4

0.2

0.0

×

CAREFL-M

LOCI-M

30

40

50

60

70

80

90100

Percentage ofDatasetsWithMisleading CVs(%)

1.0

CAREFL-M

LOCI-M

0.8

0.6

0.4

0.2

0.0

2030405060708090100

PercentageofDatasetsWithMisleadingCVs(%)

1.0

0.8

0.6

Accurac

0.4

0.2

0.0

×

CAREFL-M

LOCI-M

30

40

50

60

PercentageofDatasetsWithMisleadingCVs(%)

1.0

×

CAREFL-M

LOCI-M

0.8

0.6

ccurac

0.4

0.2

0.0

60

70

80

90

100

PercentageofDatasetsWithMisleadingCVs(%)

1.0

0.8

0.6

ccurac

0.4

0.2

0.0

×

CAREFL-M

LOCI-M

80

90

100

Percentage ofDatasetsWithMisleading CVs(%)

(a) ANM-sine and

uniform

(

-

1

,

1)

noise

(b) ANM-sine and

(0

,

1)

noise

(c) ANM-tanh and

uniform

(

-

1

,

1)

noise

(d) ANM-tanh and

(0

,

1)

noise

(e)

ANM-sigmoid

and

uniform

(

-

1

,

1)

(f) ANM-sigmoid and

(0

,

1)

noise

1.0

CAREFL-ANM-M

CAM

0.8

0.6

0.4

0.2

0.0

0102030405060708090100

PercentageofDatasetsWithMisleadingCVs(%)

1.0

CAREFL-ANM-M

CAM

0.8

0.6

0.4

0.2

0.0

0102030405060708090100

Percentage ofDatasetsWithMisleadingCVs(%)

1.0

X

CAREFL-ANM-M

CAM

0.8

0.6

0.4

0.2

0.0

0102030405060708090100

Percentage ofDatasets With Misleading CVs(%)

1.0

CAREFL-ANM-M

CAM

0.8

Accurac

0.6

0.4

0.2

0.0

0

10

20

30

40

50

60

PercentageofDatasetsWithMisleadingCVs(%)

1.0

×

×

CAREFL-ANM-M

CAM

0.8

0.6

Accurac

0.4

0.2

0.0

0

10

2030

40

50

60

70

Percentage of DatasetsWith Misleading CVs(%)

1.0

0.8

Accurac

0.6

X

0.4

0.2

0.0

CAREFL-ANM-M

CAM

102030405060708090

PercentageofDatasetsWithMisleadingCVs(%)

noise

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

10

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of HiddenNeurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0 -

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

4

10

5007501000 2000

0.0 0.00010.001

10.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplace

gaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

4

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of HiddenNeurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.00.00010.001(

0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplace

gaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (O.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001

0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

2

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

L2-Penalty

Priors

1.0

1.0

1.0 -

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (O.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0-

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0 -

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0-

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (O.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplace

gaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of HiddenNeurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.01

0.0

0.0

2

5

10

20

1

4

1

10

50075010002000

0.00.00010.0010.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

10

5007501000 2000

0.0 0.00010.001 0.1

laplace

gaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001

0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0-

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0-

0.0

2

5

10

20

1

4

>

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

1

4

7

10

50075010002000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001

0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0-

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0-

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001

0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5-

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001(

0.1

laplace

gaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0-

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

X

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

N

= 500

(b)

N

= 5000

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001(

0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0-

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (O.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

(a)

over 100 datasets from

SIM

sub-benchmark.

(b)

over 100 datasets from

SIM-c

sub-benchmark.

(c)

over 100 datasets from

SIM-ln

sub-benchmark.

(d)

over 100 datasets from

SIM-G

sub-benchmark.

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0-

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

NumberofHiddenNeurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

500750 1000 2000

0.0 0.00010.001 0.1

laplacegaussian

NumberofHiddenNeurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

10

5007501000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

1.0

1.0

1.0

1.0

1.0

0.5

0.5

0.5

0.5

0.5

0.0

0.0

0.0

0.0

0.0

2

5

10

20

1

4

7

500750 1000 2000

0.0 0.00010.001 0.1

laplacegaussian

Number of Hidden Neurons

Number of Sub-Flows

Number of Epochs

L2-Penalty

Priors

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-M (0.8)

CAREFL-M (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

CAREFL-H (0.8)

CAREFL-H (1.0)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the sensitivity of the recently introduced heteroscedastic noise model to model specification for causal discovery in a bivariate scenario. Specifically, it focuses on the impact of misspecifying the assumed noise distribution, which leads to poor performance when choosing the direction based on maximizing the likelihood. Instead, the paper proposes the use of independence testing of the residuals and presents an improved model. The newly proposed model demonstrates promising results in both real-world and synthetic datasets.\n\nSTRENGTHS:\n* Excellent and clear introduction.\n* Comprehensive literature review with useful categorization of different approaches.\n* Highlights flaws in the existing prior approach.\n* Extensive experiments using popular causal discovery datasets.\n\nAlso see questions section.\n\nWEAKNESSES:\n* The novelty appears to be limited: The result seems rather incremental, considering that the implications of model misspecification aren't surprising, and the use of independence tests is a relatively common way to utilize functional causal models for causal discovery.\n* There is a limited discussion about the work \"Identifying Patient-Specific Root Causes with the Heteroscedastic Noise Model\" by Strobl et al., given that they don't rely on maximizing the likelihood for fitting the model, do not seem to depend on prior knowledge about the noise distribution and also focus on causal discovery.\n\nAlso see questions section.\n\nQUESTIONS:\nThe paper is generally well-written and includes an in-depth comparison with related work. Some general remarks and questions:\n\n* You mention LiNGAM as an independence testing approach. However, strictly speaking, it only relies on the ICA components (assuming non-Gaussian noise), which is not utilizing independence tests. \n* The reference for GRCI appears to be incorrect.\n* There needs to be a more comprehensive comparison with the work by Strobl et al., as they do not rely on prior knowledge about the noise or exploitation of the conditional variance, which is the main critique of this paper.\n* The meaning of Figure 1 could be clearer. For instance, what does “high” and “low” exactly mean for accuracy?\n* When considering the conditional variance, it seems evident that this is primarily a matter of scaling. One can simply scale X and Y differently to arbitrarily flip edges for approaches that do not carefully scale the variables as part of their methodology.\n* Table 1 seems slightly out of place and appears to be more appropriate for the experiment section.\n* In the case of CAREFL-M, isn't the prior distribution merely a convention, since they learn a transformation function of the noise, i.e., they use g(N) and not N directly. Therefore, the Gaussian noise can be transformed into 'arbitrary' noise distributions. Perhaps, I'm missing something here?\n\n--Update after rebuttal-- I have read the rebuttal and further discussed with the authors. Many of my concerns have been addressed and I raised the score by 1.\n\nLIMITATIONS:\nNo concerns regarding societal impact. Regarding technical limitations, this paper focuses primarily on improving an existing methodology and, thus, shares the same limitations as the original work. However, it aims to make the methodology less sensitive to model misspecifications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper explores causal discovery using heteroscedastic location-scale noise functional models (LSNMs) (which generalizes the additive noise models) and compares maximum likelihood model selection to causal model selection through residual independence testing.

STRENGTHS:
1. I think this paper is significant in that it provides empirical and theoretical evidence of using the IT method for causal discovery. 
2. This paper is well-written. The contribution is very clearly written from the introduction, so little chances to miss the contribution of the paper. 
3. The simulations are extensive.

WEAKNESSES:
1. I think the robustness of the IT method should be more theoretically advocated. 
2. Some results (e.g., Proposition 5.1.) don’t have proofs, and I am not sure they are trivial.

QUESTIONS:
- Q1. What is the implication and usage of Lemma 4.1 in the paper?
- Q2. What is the definition of noise misspecification and misleading CV? Does it mean that the estimation model for the residual (noise) is wrongly specified?
- Q3. “(Identifiability) When the form of the noise distribution is correctly specified, the causal model always has a higher likelihood (Figure 2a).” can be proved by Theorem 4.2 or Corollary 4.3? If so, could you please provide easy and intuitive explanations?
- Q4. I think the proof for Proposition 5.1 should be provided.

LIMITATIONS:
I believe that more theoretical analysis is necessary to claim the robustness of IT. For instance, does Proposition 5.1 imply that the IT-based method is robust against noise misspecification and misleading cross-validation? If we assume regression suitability in the machine learning method, will the result change for ML?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate bi-variate causal inference with non iid, heteroscedastic noise. They specifically investigate a group of causal inference methods based on max-likelihood and present two main drawbacks: max-likelihood approaches with parametric assumptions on the noise distribution, such as normality, suffer when these distributions are misspecified. Further, the authors show that max-likelihood is inversely proportional to conditional variance. Hence, if the conditional variances are misleading the max-likelihood points in the anti-causal direction, which is the case for many pairs in the Tübingen real world cause-effect benchmark. The authors propose an independence test based approach fitting an affine flow to model the conditional density and HSIC test the residuals.\n\nSTRENGTHS:\n- Investigating the assumptions and failure modes of bi-variate causal inference methods is an important to ensure that these methods are accurate under real-world conditions.\n- The authors provide a thorough argument against max-likelihood based methods based on theory and empirical evidence. The empirical experiments demonstrate how misspecified noise and noise level affect the accuracy of max-likelihood methods.\n- The linkage of max-likelihoods failure modes to misleading conditional variance is plausible and is backed by theory.\n- The proposed method CAREFL-H performs well in the benchmarks covering synthetic and real world data. It outperforms both independence test and maximum likelihood based approaches alike.\n\nWEAKNESSES:\n- The contribution of this work is limited in light of recent work. It partially overlap with the work on noise misspecification by Strobl & Lasko 2022, and the critic on likelihood-based causal inference by Schultheiss & Bühlmann, 2023.\n- The pitfalls of independence test methods is not discussed, leading to an incomplete picture in regards to comparing both families of approaches.\n- The proposed amendment to the CAREFL method introduces only an independence test afterwards and applies the same scheme (RESIT, GRCI) with different function modelling tools.\n- The claim, that affine flows are \"empirically suitable under noise misspecification and misleading CVs\" is not sufficiently supported by the experiments which cover only limited function classes (sine, sigmoids).\n- For instance, CAREFL-H performs worse than RESIT on SIM, which also uses HSIC with a different function estimator that works than the affine flows.\n- Finally, the presentation including writing and figures is unclear at times (for details see minor remarks).\n\nReferences\nSchultheiss, C., & Bühlmann, P. (2023). On the pitfalls of Gaussian likelihood scoring for causal discovery. Journal of Causal Inference, 11(1), 20220068.\nStrobl, E. V., & Lasko, T. A. (2022). Identifying patient-specific root causes with the heteroscedastic noise model. arXiv preprint arXiv:2205.13085.\n\nMinor Remarks:\n- Figure 2 plots are squished.\n- Table 2a) is very cluttered and lacks a caption, maybe better as line chart to see trend.\n- Theorem 4.2 is a main result of the paper but not explained sufficiently, i.e. only half a line long and the following paragraph is also very sparse.\n- The acronyms ML and IT are associated with machine learning and information technology and hence easy to confuse, consider using different acronyms.\n\nQUESTIONS:\n- Table 1: If conditional variance is misspecified, i.e. higher in the causal direction, how do the M methods still get 100% right for all gaussian high noise pairs? And why does it not work for uniform noise?\n- What could be some benefits of using max likelihood, for example, if learned function \\hat{f} or scaling \\hat{g} is misspecified, my intuition is that max likelihood has more slack than independence tests.\n- What is the sample complexity of CAREFL-H, how much is it increased as both a train/test set is needed, where the test set needs to be large enough for an HSIC test with sufficient statistical power.\n- Experiments: you test noise independence after a regression with CAREFL-H same as RESIT, why is RESIT better on SIM across the board?\n\nLIMITATIONS:\nThe downsides of independence testing approaches are not addressed. Hence, the comparison between independence testing and max-likelihood methods is not complete. For example, in the homoscedastic case max-likelihood methods (CAM) outperform independence test methods (RESIT). This hints towards max-likelihood methods also having an edge against independence tests in some cases.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper compares using maximum likelihood vs using tests of independence with regression residuals to select between a causal model and an anti-causal model, in order to infer the causal direction between two random variables. An empirical observation is that assuming a location-scale noise model, the maximum likelihood approach tends to perform poorly when the noise distribution is misspecified and the conditional variance in the causal direction is greater than that in the anti-causal direction. By contrast, the independence testing approach or at least the particular affine flow model used in this paper appears to maintain good performance even in such unfavourable circumstances. In addition to extensive empirical results to establish this phenomenon, this paper also provides some theoretical explanations of the phenomenon.\n\nSTRENGTHS:\nThis paper is very well written and readable. The empirical results are convincing and illuminating. The theoretical results are interesting and provide a compelling and useful explanation of the empirical observation. Given the importance and difficulty of the problem of inferring causal asymmetry from observational data, the empirical results and theoretical analysis presented in this paper strike me as sufficiently significant.\n\nWEAKNESSES:\nAs far as I am concerned, this paper does not have notable weaknesses. I would appreciate some explanation or even speculation on why the flow estimator remains empirically suitable under noise misspecification and misleading conditional variances. Does noise specification play any role in the flow estimator?\n\nQUESTIONS:\nDoes the specification of noise distribution play any role in the flow-based independence tests? If not, I wonder if that is the main reason why it is more robust than the maximum likelihood methods. How much, if at all, do misleading CVs by themselves (i.e., without noise misspecification) affect the performance of the ML methods?\n\nLIMITATIONS:\nOK.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}