SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiangyu Sun
xiangyu_sun@sfu.ca
Oliver Schulte
oschulte@cs.sfu.ca
A fundamental problem of causal discovery is cause-effect inference, learning the correct causal direction between two random variables. Significant progress has been made through modelling the effect as a function of its cause and a noise term, which allows us to leverage assumptions about the generating function class. The recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified. However, through an extensive empirical evaluation, we demonstrate that the accuracy deteriorates sharply when the form of the noise distribution is misspecified by the user. Our analysis shows that the failure occurs mainly when the conditional variance in the anti-causal direction is smaller than that in the causal direction. As an alternative, we find that causal model selection through residual independence testing is much more robust to noise misspecification and misleading conditional variance.
Distinguishing cause and effect is a fundamental problem in many disciplines, such as biology, healthcare and finance [31, 8, 13]. Randomized controlled trials (RCTs) are the gold standard for finding causal relationships. However, it may be unethical, expensive or even impossible to perform RCTs in real-world domains [20, 18]. Causal discovery algorithms aim to find causal relationships given observational data alone. Traditional causal discovery algorithms can only identify causal relationships up to Markov equivalence classes (MECs) [26, 10, 3]. To break the symmetry in a MEC, additional assumptions are needed [20, 22], such as the type of functional dependence of effect on cause. Structural causal models (SCMs) specify a functional class for the causal relations in the data [17, 20]. In this work, we focus on one particular type of SCM called location-scale noise models (LSNMs) [29, 11, 30, 28, 9]:
$$
Y := f (X) + g (X) \cdot Z _ {Y} \tag {1}
$$
where  $X$  is the cause,  $Y$  is the effect, written  $X \to Y$ , and  $Z_{Y}$  is a latent noise variable independent of  $X$  (i.e.,  $X \perp Z_{Y}$ ). The functions  $f$  and  $g$  are twice-differentiable on the domain of  $X$ , and  $g$  is strictly positive on the domain of  $X$ . LSNMs generalize the widely studied additive noise models (ANMs), where  $g(x) = 1$  for all  $x$ . ANMs assume a constant conditional variance  $\mathbb{V}[Y|X]$  for different values of  $X$ , which can limit their usefulness when dealing with heteroscedastic noise in real-world data. LSNMs allow for heteroscedasticity, meaning that  $\mathbb{V}[Y|X]$  can vary depending on the value of  $X$ . In our experiments on real-world data, methods assuming an LSNM outperform those assuming ANM.
We follow previous work on cause-effect inference and focus on the bivariate setting [29, 11, 30, 9]. There are known methods for leveraging bivariate cause-effect inference in the multivariate setting (see Appendix A). Two major methods for selecting a causal SCM direction in cause-effect inference are maximum likelihood (ML) and independence testing (IT) of residuals vs. the (putative) cause [14]. Both have been recently shown good accuracy with LSNMs [11, 9], especially when the  $f$  and  $g$  functions are estimated by neural networks. Immer et al. [9] note, however, that ML can be less robust than IT, in the sense that accuracy deteriorates when the noise distribution is not Gaussian.
In this paper we investigate the robustness of ML vs. IT for LSNMs. Our analysis shows that ML cause-effect inference performs poorly when two factors coincide: (1) Noise Misspecification: the ML method assumes a different form of the noise distribution from the true one. (2) Misleading Conditional Variances (CVs):  $\mathbb{V}[Y|X] > \mathbb{V}[X|Y]$  in the data generated by causal direction  $X\rightarrow Y$ . For example, in the experiment on synthetic datasets shown in Table 1 below, (i) changing the true noise distribution from Gaussian to uniform and (ii) manipulating  $\mathbb{V}[Y|X]$ , while keeping other settings equal, can decrease the rate of identifying the true causal direction from  $100\%$  to  $10\%$ . In contrast, IT methods maintain a perfect  $100\%$  accuracy. The difference occurs with and without data standardization, and with increasing sample sizes.
Both conditions (1) and (2) often hold in practice. For real-world domains, assumptions about the noise distribution can be hard to determine or verify. It is also common to have misleading CVs in real-world datasets. For example, in the Tübingen Cause-Effect Pairs benchmark [14], about  $40\%$  of the real-world datasets exhibit a misleading CV (see Table 7 in the appendix).
We make the following contributions to understanding location-scale noise models (LSNMs):
- Describe experiments and theoretical analysis to show that ML methods succeed when the form of the noise distribution is known. In particular, ML methods are then robust to incorrectly specifying a noise variance parameter.
- Demonstrate empirically that ML methods often fail when the form of the noise distribution is misspecified and CV is misleading, and analyze why.
- Introduce a new IT method based on an affine flow LSNM model.
- Demonstrate, both theoretically and empirically, that our IT method is robust to noise misspecification and misleading CVs.
The paper is structured as follows. We discuss related works and preliminaries in Section 2 and Section 3, respectively. Section 4 examines when and why ML methods fail. Section 5 demonstrates the robustness of the IT method. Experiments with 580 synthetic and 99 real-world datasets are given in Section 7. The code and scripts to reproduce all the results are given online<sup>1</sup>.
Causal Discovery. Causal discovery methods have been widely studied in machine learning [26, 10, 3]. Assuming causal sufficiency and faithfulness, they find causal structure up to a MEC. To identify causal relations within a MEC, additional assumptions are needed [20, 22]. SCMs exploit constraints that result from assumptions about the functional dependency of effects on causes. Functional dependencies are often studied in the fundamental case of two variables  $X$  and  $Y$ , the simplest MEC. Mooij et al. [14] provide an extensive overview and evaluation of different cause-effect methods in the ANM context. We follow this line of work for LSNMs with possibly misspecified models.
Structural Causal Models. Assuming a linear non-Gaussian acyclic model (LiNGAM) [24, 25], the causal direction was proved to be identifiable. The key idea in DirectLiNGAM [25] is that in the true causal direction  $X \rightarrow Y$ , the model residuals are independent of  $X$ . We refer to methods derived from the DirectLiNGAM approach as independence testing (IT) methods. The more general ANMs [7, 19] allow for nonlinear cause-effect relationships and are generally identifiable, except for some special cases. There are other identifiable SCMs, such as causal additive models (CAM) [2], post-nonlinear models [32] and Poisson generalized linear models [16].
LSNM Identifiability. There are several identifiability results for the causal direction in LSNMs. Xu et al. [30] prove identifiability for LSNMs with linear causal dependencies. Khemakhem et al. [11]
$^{1}$ https://github.com/xiangyu-sun-789/CAREFL-H
Figure 1: Summary of the accuracy results for ML methods for LSNMs and ANMs with respect to noise misspecification and misleading CVs.
show that nonlinear LSNMs are identifiable with Gaussian noise. Strobl and Lasko [28], Immer et al. [9] prove LSNMs are identifiable except in some pathological cases (see Appendix C).
Cause-Effect Inference in LSNMs. The generic model selection blueprint is to fit two LSNM models for each direction  $X \rightarrow Y$  and  $X \leftarrow Y$  and select the direction with a higher model score. HECI [30] bins putative cause values and selects a direction based on the Bayesian information criterion. BQCD [29] uses nonparametric quantile regression to approximate minimum description length to select the direction. GRCI [28] finds the direction based on mutual information between the putative cause and the model residuals. LOCI [9] models the conditional distribution of effect given cause with Gaussian natural parameters. Then, it chooses the direction based on either likelihood (LOCI-M) or independence (LOCI-H). CAREFL-M [11] fits affine flow models and scores them by likelihood. CAREFL-M is more general than LOCI since LOCI uses a fixed Gaussian prior, whereas CAREFL-M can utilize different prior distributions. DECI [4] generalizes CAREFL-M to multivariate cases for ANMs. Unlike our work, none of these papers provide a theoretical analysis of noise misspecification and misleading CVs in LSNMs.
Noise Misspecification. Schultheiss and Buhlmann [23] present a theoretical analysis of model misspecification pitfalls when the data are generated by an ANM. Their results focus on model selection with Gaussian likelihood scores, which use Gaussian noise distributions. They suggest that IT-based methods may be more robust to model misspecification in ANMs. The authors also briefly discuss the complication of fitting data generated by a ground-truth ANM model with an LSNM model. Since the backward model of an ANM often takes the form of an LSNM, fitting data generated by a ground-truth ANM with an LSNM model may result in an incorrect causal direction. While their analysis examines ANM data, the ground-truth model in our work is an LSNM. The  $g(\cdot)$  scale term in Equation (1), which is absent in ANMs, plays a critical role in LSNMs.
Conditional Variance and Causality. To the best of our knowledge, the problem of misleading CVs has not been identified nor analyzed in previous work for LSNMs. Prior studies that have focused on CVs were based on ANMs, and stated assumptions about how CVs relate to causal structure, to be leveraged in cause-effect inference [1, 15]. We do not make assumptions about CVs, but analyze them to understand why noise misspecification impairs cause-effect inference in LSNMs.
In this section, we define the cause-effect inference problem and review the LSNM data likelihood.
3.1 Problem definition: cause-effect inference
Cause-effect inference takes as input a dataset  $D$  over two random variables  $X, Y$  with  $N$  observational data pairs  $(X, Y) = \{(x_1, y_1), (x_2, y_2), \ldots, (x_N, y_N)\}$  generated by a ground-truth LSNM in Equation (1). The binary output decision indicates whether  $X$  causes  $Y$  ( $X \to Y$ ) or  $Y$  causes  $X$  ( $X \gets Y$ ). We assume no latent confounders, selection bias or feedback cycles ([14, 29, 11, 9, 30]).
Table 1: Accuracy over 10 datasets generated by SCM LSNM-sine-tanh (definition in Appendix M) with  $N = 10,000$  samples. The task is a binary decision whether  $X$  causes  $Y$  or  $Y$  causes  $X$ .  $X$  denotes the ground-truth cause and  $Y$  denotes the ground-truth effect. We rewrite Equation (1) as  $Y = f(X) + \alpha \cdot g(X) \cdot Z_Y$ , where  $\alpha$  is a scale factor to alter the CVs. CVs are computed by binning the putative cause. We used Gaussian(0,1) as model prior for both CAREFL and LOCI. The suffix -M denotes a ML method. The suffix -H denotes the corresponding IT method (see Section 5). All the datasets are standardized to have mean 0 and variance 1.
True Noise	Gaussian(0,1)	Uniform(-1,1)
α	0.1	0.5	1	5	10	0.1	0.5	1	5	10
\(\overline{\mathbb{V}}[Y|X]\)	0.166	0.615	0.834	0.990	0.997	0.044	0.404	0.673	0.975	0.994
\(\overline{\mathbb{V}}[X|Y]\)	0.455	0.709	0.793	0.821	0.817	0.047	0.375	0.566	0.681	0.677
Percentage of Datasets With Misleading CVs	30%	50%	70%	100%	100%	30%	90%	100%	100%	100%
CAREFL-M	1.0	1.0	1.0	1.0	1.0	0.7	0.6	0.7	0.1	0.1
LOCI-M	1.0	1.0	1.0	1.0	1.0	0.7	0.6	0.7	0.1	0.1
CAREFL-H	1.0	1.0	1.0	1.0	1.0	0.9	1.0	1.0	1.0	1.0
LOCI-H	1.0	1.0	1.0	1.0	1.0	0.7	1.0	1.0	1.0	1.0
3.2 Definition of maximum likelihood for LSNMs
An LSNM model for two variables  $(X,Y)$  is a pair  $(\rightarrow, P_Z)$ . The  $\rightarrow$  represents the direction  $X \rightarrow Y$  with parameter space  $f \equiv f_{\theta}, g \equiv g_{\psi}, P_X \equiv P_{X,\zeta}$ . The  $\leftarrow$  represents the direction  $X \leftarrow Y$  with parameter space  $h \equiv h_{\theta'}, k \equiv k_{\psi'}, P_Y \equiv P_{Y,\zeta'}$ . For notational simplicity, we treat the model functions directly as model parameters and omit reference to their parameterizations. For example, we write  $f \equiv f_{\theta}$  for a function  $f$  that is implemented by a neural network with weights  $\theta$ . We refer to  $P_Z$  as the model prior noise distribution.
A parameterized LSNM model defines a data distribution as follows [9] (Appendix B):
$$
P _ {\rightarrow , P _ {Z}} (X, Y; f, g, P _ {X}) = P _ {X} (X) \cdot P _ {Z _ {Y}} \left(\frac {Y - f (X)}{g (X)}\right) \cdot \frac {1}{g (X)} \tag {2}
$$
$$
P _ {\leftarrow , P _ {Z}} (X, Y; h, k, P _ {Y}) = P _ {Y} (Y) \cdot P _ {Z X} \left(\frac {X - h (Y)}{k (Y)}\right) \cdot \frac {1}{k (Y)}
$$
For conciseness, we use shorter notation  $P_{\rightarrow ,P_Z}(X,Y)$  and  $P_{\leftarrow ,P_Z}(X,Y)$  in later sections. The ML method computes the ML parameter estimates and uses them to score the  $\rightarrow$  (or  $\leftarrow$ ) model:
$$
\widehat {f}, \widehat {g}, \widehat {P} _ {X} := \arg \max  _ {f, g, P _ {X}} \prod_ {i = 1} ^ {N} P _ {\rightarrow , P _ {Z}} \left(x _ {i}, y _ {i}; f, g, P _ {X}\right) \tag {3}
$$
$$
L _ {\rightarrow , P _ {Z}} (D) := \prod_ {i = 1} ^ {N} P _ {\rightarrow , P _ {Z}} \left(x _ {i}, y _ {i}; \widehat {f}, \widehat {g}, \widehat {P _ {X}}\right) \tag {4}
$$
4 Non-identifiability of LSNMs with noise misspecification
In this section, we show that ML methods fail under noise misspecification and misleading CVs, and analyse why. Figure 1 summarizes the diverse settings considered below.
Existing identifiability results for LSNM ML methods (Appendix C) require knowing the ground-truth noise distribution. We conducted a simple experiment to show that with noise misspecification, ML model selection can fail badly even on large sample sizes. Table 1 shows results for CAREFL-M [11] and LOCI-M [9] with model prior distribution Gaussian(0,1). For the left half of the table, the data
was generated with correctly specified Gaussian(0,1) noise. In this case, both CAREFL-M and LOCI-M work well, and increasing CV in the causal direction does not affect their accuracy. For the right half of the table, the data was generated with misspecified uniform  $(-1,1)$  noise. With noise misspecification, the accuracy of both CAREFL-M and LOCI-M decreases to  $70\%$ . With both noise misspecification and misleading CVs, their accuracy becomes even lower. For example, in the last column when  $\overline{\mathbb{V}}[Y|X] = 0.994$  and  $\overline{\mathbb{V}}[X|Y] = 0.677$ , they give an accuracy of just  $10\%$ .
The following results help explain why ML methods fail under noise misspecification and misleading CVs.
Lemma 4.1. For an LSNM model  $Y \coloneqq f(X) + g(X) \cdot Z_Y$  where  $Z_Y$  is the noise with mean  $\mu$  and variance  $\sigma^2 \neq 0$ , there exists a likelihood-equivalent LSNM Model  $Y \coloneqq f'(X) + g'(X) \cdot Z_Y'$  such that  $Z_Y' = \frac{Z_Y - \mu}{\sigma}$ ,  $P_{\rightarrow, P_Z}(Y|X; f, g) = P_{\rightarrow, P_{Z'}}(Y|X; f', g')$ , and  $\mathbb{V}_{Z_Y}[Y|X] = \mathbb{V}_{Z_Y'}[Y|X]$ .
The proof is in Appendix E. Lemma 4.1 shows that for any (non-deterministic) LSNM model there is an equivalent standardized LSNM model with  $\mathbb{V}[Z_Y] = 1$ .
Theorem 4.2.  $\frac{\partial\log P_{\rightarrow,P_Z}(Y|X)}{\partial\mathbb{V}[Y|X]} < 0$  for LSNM models.
The proof is in Appendix F. Theorem 4.2 shows that the model likelihood  $P_{\rightarrow, P_Z}(Y|X)$  and the conditional variance  $\mathbb{V}[Y|X]$  are negatively related (similarly for  $P_{\leftarrow, P_Z}(X|Y)$  and  $\mathbb{V}[X|Y]$ ).
Corollary 4.3. Let  $\rightarrow$  denote the causal model and  $\leftarrow$  denote the anti-causal model. When  $\mathbb{V}[Y|X]$  increases and  $\mathbb{V}[X|Y]$  decreases in the data, the log-likelihood model difference  $\log P_{\rightarrow ,P_Z}(X,Y) - \log P_{\leftarrow ,P_Z}(X,Y)$  decreases.
The proof is in Appendix G. Under the identifiability assumption [11, 9] that ML is consistent under correct model specification, Lemma 4.1 also implies that ML remains consistent under misspecifying the variance of the noise distribution, as long as the form of the prior distribution matches that of the noise distribution. Table 4a in the appendix demonstrates this relationship empirically.
The overall relationship between likelihood and CV with respect to noise specification in LSNMs is as follows, illustrated in Figure 2 with actual datasets:
1. (Theorem 4.2) CV and likelihood are negatively related (Figures 2a and 2b), with either correctly or incorrectly specified noise distribution form.
2. (Identifiability) When the form of the noise distribution is correctly specified, the causal model always has a higher likelihood (Figure 2a).
3. (Corollary 4.3) When the form of the noise distribution is misspecified, the anti-causal model can have a higher likelihood under misleading CVs (Figure 2b).
ANM vs. LSNM. The relationship (2) is in general different for ANMs, when the noise variance is misleading (see Figure 1 and Appendix P). Proposition H.1 shows that the standardization lemma Lemma 4.1 does not apply to ANMs (confirmed by empirical results in Table 4b in the appendix).
5 Robustness of independence testing
This section describes IT methods for cause-effect learning in LSNMs, including a new IT method based on affine flows. Theoretical results below explain why IT methods are robust to noise misspecification and misleading CVs in LSNMs.
5.1 The independence testing method
Inspired by the breakthrough DirectLiNGAM approach [25], independence testing has been used in existing methods for various SCMs [7, 19, 28, 9]. Like ML methods (Equation (4)), IT methods fit the model parameters in both directions, typically maximizing the data likelihood (Equation (3)). The difference is in the model selection step, where IT methods select the direction with the highest degree of independence between the fitted model residuals and the putative cause.
Algorithm 1 is the pseudo-code for our IT method CAREFL-H. We fit the functions  $f$  and  $g$  in Equation (1), with the affine flow estimator  $\mathbf{T}$  from CAREFL-M [11], implemented using neural
(a) Log-likelihood difference under correct noise specification: Gaussian(0,1) noise
(c) HSIC difference under correct noise specification: Gaussian(0,1) noise
Figure 2: Visualization of Table 1. First row (2a,2b): ML methods. Second row (2c,2d): IT methods.  $Y$ -axis  $< 0.0$ : A ML method returns the incorrect anti-causal direction.  $Y$ -axis  $> 0.0$ : An IT method returns the incorrect anti-causal direction. ML methods may fail under misspecification and misleading CVs (2b, when  $X$ -axis  $> 1$ ). IT methods are more robust (2d).
(b) Log-likelihood difference under noise misspecification: uniform(-1, 1) noise
(d) HSIC difference under noise misspecification: uniform  $(-1, 1)$  noise
networks. For details on CAREFL-M and learning the flow transformation  $\mathbf{T}$  see Appendix I. This combination of affine flow model with IT appears to be new. Another IT method is to test the independence of both residuals [6] (Appendix J). We found that this performs similarly to CAREFL-H and therefore report results only for the more common DirectLiNGAM-style method.
As in previous work [14], we use the Hilbert-Schmidt independence criterion (HSIC) [5] to measure (in)dependence throughout the paper. HSIC measures the squared distance between the joint probability of the two variables and the product of their marginals embedded in the reproducible kernel Hilbert space. We have  $HSIC(U,V) = 0$  if and only if  $U \perp V$ .
5.2 Theoretical comparison between IT and ML under noise misspecification in LSNMs
The intuition for why independence testing works is that if LSNM training is consistent, testing the independence of the putative cause and the residual will indicate the correct causal model. The next theorem provides a formal statement of this intuition.
Theorem 5.1. For data pairs  $(X,Y)$  generated by an LSNM model  $X\to Y$ , let  $\widehat{f} (X)$  denote the conditional mean estimator and  $\widehat{g} (X)$  denote the conditional standard deviation estimator. Under the consistency conditions that  $\widehat{f} (X) = \mathbb{E}[Y|X]$  and  $\widehat{g} (X) = STD[Y|X]$ , the reconstructed noise is independent of the cause, i.e.  $\widehat{Z}_Y\perp X$ , even under noise misspecification.
Proof. According to Equation (1),
$$
\widehat {f} (X) = \mathbb {E} [ Y | X ] = \mathbb {E} [ f (X) + g (X) \cdot Z _ {Y} | X ] = f (X) + g (X) \cdot \mathbb {E} [ Z _ {Y} | X ] = f (X) + g (X) \cdot \mathbb {E} [ Z _ {Y} ]
$$
$$
\widehat {g} (X) = S T D [ Y | X ] = S T D [ f (X) + g (X) \cdot Z _ {Y} | X ] = | g (X) | \cdot S T D [ Z _ {Y} | X ] = g (X) \cdot S T D [ Z _ {Y} ]
$$
Thus,  $\widehat{Z}_Y = \frac{Y - \widehat{f}(X)}{\widehat{g}(X)} = \frac{f(X) + g(X)\cdot Z_Y - f(X) - g(X)\cdot\mathbb{E}[Z_Y]}{g(X)\cdot STD[Z_Y]} = \frac{Z_Y - \mathbb{E}[Z_Y]}{STD[Z_Y]}$ , where  $\mathbb{E}[Z_Y]$  and  $STD[Z_Y] > 0$  are constants. Therefore,  $\widehat{Z}_Y$  and  $Z_Y$  are identical up to shift and scale. Since  $Z_Y \perp X$ , therefore,  $\widehat{Z}_Y \perp X$ .
Note that the proof does not involve the specification of the noise prior distribution  $P_{Z_Y}$ , which shows that under the consistency condition, the conclusion  $\widehat{Z}_Y \perp X$  holds whether the noise prior distribution is misspecified or not.
Algorithm 1 CAREFL-H
1: Input: data pairs  $D \coloneqq (X, Y)$ , the flow estimator  $\mathbf{T}$  of CAREFL-M with prior  $P_Z$ , and an HSIC estimator
2: Output: estimated causal direction dir
3: Split  $D$  into training set  $D_{train} \coloneqq (X_{train}, Y_{train})$  and testing set  $D_{test} \coloneqq (X_{test}, Y_{test})$
4: Optimize  $\mathbf{T}_{\widehat{\theta},\widehat{\psi},\widehat{\zeta}}(D_{train};P_Z)$  in  $X\rightarrow Y$  direction via ML to estimate  $\widehat{f}$  and  $\widehat{g}$
5: Compute the residual  $\widehat{Z}_Y\coloneqq \frac{Y_{test} - \widehat{f}(X_{test})}{\widehat{g}(X_{test})}$
6: Optimize  $\mathbf{T}_{\widehat{\theta}^{\prime},\widehat{\psi}^{\prime},\widehat{\zeta}}(D_{train};P_Z)$  in  $X\gets Y$  direction via ML to estimate  $\widehat{h}$  and  $\widehat{k}$
7: Compute the residual  $\widehat{Z}_X\coloneqq \frac{X_{test} - \widehat{h}(Y_{test})}{\widehat{k}(Y_{test})}$
8: if  $HSIC(X_{test},\widehat{Z}_Y) < HSIC(Y_{test},\widehat{Z}_X)$  then
9: dir :=  $X \to Y$
10: else if  $HSIC(X_{test}, \widehat{Z}_Y) > HSIC(Y_{test}, \widehat{Z}_X)$  then
11: dir := X ← Y
12: else
13: dir := no conclusion
14: end if
In contrast, the likelihood of a causal model depends on the specification of the noise prior distribution  $P_{Z_Y}$  (Equation (2)). This explains why IT methods are more robust than ML methods under noise misspecification in LSNMs.
In the context of ANMs, Mooij et al. [14] provide another sufficient condition for  $\widehat{Z}_Y \perp X$  called suitability. In Appendix K, we show how the suitability concept can be adapted for LSNMs to provide another argument for the robustness of IT methods.
6 Limitation of IT-based methods
Limitations of IT-based methods include the following: 1) When the noise prior distribution is correctly specified, ML-based methods can utilize this information to be more sample efficient. Conversely, since IT-based methods are less sensitive to the noise prior distribution, they require a larger sample size to infer causal direction accurately. 2) IT-based methods have higher computational cost, especially with nonlinear independence testing. 3) IT-based methods require high-capacity function estimators to model the unknown functions  $f$  and  $g$  for a good noise reconstruction.