When Additive Noise Meets Unobserved Mediators: Bivariate Denoising Diffusion for Causal Discovery
Authors: Dominik Meier
Sujai Hiremath
PROMIT GHOSAL
Kyra Gan
Venue: NeurIPS 2025 poster ()
Keywords: causal discovery
additive noise model
diffusion
Primary area: probabilistic_methods
TL;DR: We introduce a diffusion-based method for bivariate causal discovery that remains robust to unobserved mediators in additive noise models.

Abstract: Distinguishing cause and effect from bivariate observational data is a foundational problem in many disciplines, but challenging without additional assumptions. Additive noise models (ANMs) are widely used to enable sample-efficient bivariate causal discovery.
However, conventional ANM-based methods fail when unobserved mediators corrupt the causal relationship between variables. This paper makes three key contributions: first, we rigorously characterize why standard ANM approaches
break down in the presence of unmeasured mediators.  Second, we demonstrate that prior solutions for hidden mediation are brittle in finite sample settings,
limiting their practical utility. To address these gaps, we propose Bivariate Denoising Diffusion (BiDD) for causal discovery, a method designed to handle latent noise introduced by unmeasured mediators. Unlike prior methods that infer directionality through mean squared error loss comparisons, our approach introduces a novel independence test statistic: during the noising and denoising processes for each variable, we condition on the other variable as input and evaluate the independence of the predicted noise relative to this input. We prove asymptotic consistency of BiDD under the ANM, and conjecture that it performs well under hidden mediation. Experiments on synthetic and real-world data demonstrate consistent performance, outperforming existing methods in mediator-corrupted settings while maintaining strong performance in mediator-free settings.

When Additive Noise Meets Unobserved Mediators: Bivariate Denoising Diffusion for Causal Discovery
Dominik Meier
Sujai Hiremath" Cornell Tech sh25838cornell.co edu
Cornell Tech dm9540cornel1.edt edu
Promit Ghosal
Kyra Gan Cornell Tech
University of Chicago
promit@uchicago. edu
kyragan@cornell.co edu
Abstract
Distinguishing cause and effect from bivariate observational data a foundational problem in many disciplines, but challenging without additional assumptions. Ad- ditive noise models (ANMs) are widely used to enable sample-efficiens bivariate causal discovery. However, conventional ANM-bascd methods fail when unob- served mediators corrupt the causal relationship between variables. This paper makes three key contributions: first, rigorously characterize why standard ANM approaches break down in the presence of unmeasured mediators. Second, we demonstrate that prior solutions for hidden mediation are brittle in finite sample settings, limiting their practical utility. To address these gaps, we propose Bi- variate Denoising Diffussion (BiDD) for causal discovery, method designed handle latent noise introduced by unmeasured mediators. Unlike prior methods that infer directionality through mean squared error loss comparisons, our approach introduces a novel independence test statistic: during the noising and denoising processes for each variable, we condition on the other variable as input and evaluate the independence of the predicted noise relative to this input. We prove asymptotic consistency of BiDD under the ANM, and conjecture that it performs well under hidden mediation. Experiments synthetic and real-world data demonstrate consistent performance, outperforming existing methods in mediator-coruuted settings while maintaining strong performance in mediator-free settings.
Introduction
Determining the causal direction between variables (X-Y) is fundamental scientific domains ranging from genomics to economics. However, traditional discovery methods, such as constraint- based [61, 60] and scoring- methods [8, 27, 17] can only identify causal graphs up to an equivalence class, leaving them unable to distinguish the causal direction between a variable pair. Additional assumptions are necessary to enable bivariate discovery [46], and they mostly fall under three categories: (1) the location scale noise model (LSNMs), (2) the principle of independent mechanisms (ICM), and (3) the additive noise model (ANM).
LSNMs express the outcome Y with heteroskedastic, multiplicative noise relative to the treatment X, i.e. Y ((X) g(X),, where IL X. While LSNMs allow for increased flexibility, existing ap- proaches require additional parametric assumptions for identifiability [62, 64, 18, 7]. ICM approaches assume that the marginal distribution of the cause and the conditional mechanism generating the effect
Shared first co-author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

are independent components of the data- gettratennaningggg. process (DGP) [57, 22]. While they impose no explicit functional form, these methods rely on unverifiable structural asymmetries [39, 19], often fail under non-invertible mechanisms [20], and often lack theoretical guarantees [62]. In contrast, additive noise models offer unique advantages for bivariate discovery, allowing for consistent recovery of causal directions without strong parametric assumptions [67], permitting sample complexity charac- terization under Gaussian noise [70], and enabling polynomial- time guarantees for global discovery on large graphs [47]. These properties have spurred both methodological advances [35, 13, 65, 14] and real-world applications [54, 28].
However, these strengths vanish when hidden variables corrupt the observed causal relationships near-ubiquitous scenario in real-world systems like biomedicine [28] and economics [2]. Indeed, as Peters et al. [48] point out, although the joint distribution of all variables may admit an ANM, the joint distribution over subset that excludes some mediators may not allow for an ANM (see Appendix D.3). To the best of our knowledge, despite the rapid advances in statistical tests that handle unobserved confounding of causal pairs [21, 33, 34, 66, 30), only one bivariate discovery method [6] addresses the problem of unobserved mediators. However, Cai et al. [6] provides no correctness guarantees, requires nonlinearity, and has poor empirical performance (Section 5). This leaves a glaring gap in practical bivariate causal discovery.
Contributions. In this paper, we propose bivariate denoising diffusion (BiDD), causal direction identification method that works for general ANM, even in the presence of unobserved mediators. Our contributions are fourfold:
Analysis of Unmeasured Mediators: We first introduce the ANM-UM, a novel approach for modeling unobserved mediators (Section 2). We then characterize how unobserved mediators break the ANM assumption over observed variables, finding that this occurs if and only if there are nonlinear mechanisms induced after the initial transformation of the cause (Lemma 2.3).
Mode of Existing Methods: We first categorize conventional ANM-hased methods into three types: Residual-Indepen Indeeendenee,,, MSE-Minimization, and Score-Matching based (Section 3). For each category, we show that existing methods will fail to correctly recover the directionality when unobserved mediators break the ANM assumption (Lemmas 3. 1-3.4). We then analyze the only method developed to handle hidden mediation, discussing potential issues.
Diffusion Methodology and Guarantees We develop BiDD, a practical alternative to existing ANM based methods, hypothesizing that the noise predictions from conditional diffusion model will be less dependent on the condition when the condition is the cause, rather than the effect (Section 4). We show a consistency result under the assumption of an ANM (Theorem 4.2), and conjecture that a similar result may hold in the ANM-UM setting.
Comprehensive Evaluation: We extensively evaluate BiDD on synthetic data, demonstrating that only our approach able to achieve uniformly strong performance across DGPs with linear, nonlinear non-invertible, and nonlinear invertible mechanisms (Section 5.2). We then validate BiDD on a large real world dataset, the Tübingen Cause-Effect pairs [41], where it achieves comparable results to the best baselines, highlighting BiDD's robustness across diverse domains (Section 5.2). The source code for BiDD publicly available.
2 Problem Setup
In this work, we focus on the discovery of the causal direction between a causal pair (X, Y). which is generated by an ANM with Unobserved Mediators (ANM-UM). In this section, first formally introduce the structural causal model describing ANM-UM. We then establish identifiability conditions and characterize when ANM-UM cannot be simplified standard ANMs. A complete notation table is included in Appendix A.
Under ANM-UM, the outcome Y is generated from cause X through unobserved mediators [ (Figure 1), with each Z, introducing independent noise while remaining unmeasured. Formally, given T unmeasured mediators, the DGP between X and Y can be described as follows:
hnttps:/ggithul htttas///gttaaa.coo////mmmsse// www./oomooie//biite..
2

---

Frtnl
fr41
Figure 1: ANM-UM (Eq 2.1), where mediators Z1,... Zer and noises 60.... ...1+1 all unobserved.
Z1 [i(X) + -1:
Z2 = f2(Z1) +
(2.1)
ZT - (T(Pa(ZT) + ET,
y f + +T+11
where X, HeTr are mutually independent. The functions ear, and the can be arbitrary (Gaussian or non-Gauss...). We choose to model the process with multiple latent variables, instead of collapsing all Z; into one Z, to preserve the independent additive noise structure.
(J1...., can be linear or nonlin-
Assumption 2.1 (ANM-UM Setting). Suppose X, Y,Y follow ANM-UM described by Eq. (2.1). Then, assume: 1) no confounders among X, (Z. and Y; 2) acyclicity; 3) no selection bias (noise independence is preserved in the data collection process); 4) X 1 Y, ie., f is nonzero almost everywhere (otherwise, X IL detectable via simple independence testing).
The conventional ANM and the related Post-Nonlinear (PNL) Model [67] are special cases of ANM- UM. ANM corresponds to zero mediators (i.e., T = 0), while PNL corresponds to one mediator (i.e., T = 1) and no additive noise Y (i.e., ST+1 = 0). Our ANM-UM also generalizes the Cascade Additive Noise Model (CANM) [6], which assumes all functions
fr+11 are nonlinear. See
Appendix D.1 for details.
Prior work (Theorem Cai et al. [6]) shows that certain joint distributions over (X, admit ANM-UM representations in both directions X --+ Y and Y --+ X, rendering the causal direction unidentifiable without further assumptions. We refer to such distributions as Backward ANM UM, the structural assumptions of ANM-UM hold in both the causal and anticausal directions. These distributions can only arise under pathological conditions, such as when all functions are linear and the noises are Gaussian. We thus impose:
Assumption 2.2 (Identifiability Constraint). Under Eq. (2.1) and Assump. 2.1, no backward ANM- UM exists where X =
.E) + ēr41 with Y, .,....,,E4 mutually independent.
Appendix D.2 provides explicit constraints on the backward mechanism g and noise terms (éiheem that preclude non-identifiability under Assumption 2.2.
While CANM [6] requires all mediators to be nonlinear, ANM-UM permits identifiability with unobserved mediators even under linear transformations, reducing standard ANM when the causal effect admits an additive decomposition:
Y (1 + A2(E1, ST) 6T+1,
(2.2)
where functions A₁ and A (determined by f1...., fr) are separable without X-s interaction terms. For example, in a variable ANM-UM X - Zı - Y. if fi is nonlinear and f2 is linear, it reduces to the ANM setting, whereas if f1, are both nonlinear, it does not (see Appendix D.3). Lemma 2.3 (proof in Appendix D.4) formalizes this: ANM-UM is irreducible to ANM and only if there exists mediator Z, such that Y depends nonlinearly on Z;:
Lemma 2.3 (Irreducible ANM-UM and Nonlinear Mediator). Under ANM-UM (Eq. (2.1)) and Assump.s 2.1 I and 2.2, Y does not admit decomposition in Eq. (2.2) if and only if there exists mediator Z. such that Y -h(2,) + E for some nonlinear ht with E 11. Ze. Additionally, we call such mediator Z; nonlinear mediator.
3

---

3 Failure-Modes of Prior Work
In this section, we illustrate how both standard ANM methods and one existing hidden mediator approach fail under ANM-UM settings (Eq.(2.1) and Assumption 2.1), assuming identifiability (Assumption 2.2) and irreducibility of ANM-UM (Lemma 2.3).
3.1 Traditional ANM-based Bivariate Methods
Existing methods mostly fall into three categories: 1) Residuall-nddeendeen (RI): identify the cause via an independent residual, 2) Score-Matching: identify the effect via conditions on the score function, 3)
identify the cause via the smallest residual. For each class, we present its core decision rule and construct ANM-UM counterexamples where it fails.
Residual-Independenec Key methods include DirectLiNGAM [59], its nonparametric general- ization RESIT [47], and PNL [67]. The former two leverage ANM-induced residual independence asymmetries via common decision rule (Decision Rule E.2): if the residual from regressing Y on X is independent of X but the residual from regressing X on Y depends on Y, we conclude X causes Y, and vice versa. If both residuals independent or dependent,
conclusion can be drawn.
As a counterexample, consider the following ANM-UM: X (0, 1), = X2 +1. = 72 + with 52 NO, 1). The residual e1 := Y E[y X] can be simplified as
Y - E[y]x]= = Y [ + eî + 82) + (2,, x²)] oî + + 1 X2 -
(3.1)
Since e1 ... X, Decision Rule E.2 does not return the correct causal directionality and fails to identify
X --9 Y. We formalize this intuition in Lemma 3.1 (proof in Appendix E.3y:
Lemma 3.1 (Regression
Fails). Assuming a consistent estimator for re- gression residuals and access to infinite data, Decision Rule E.2 fails to identify the correct causal direction when least one mediator nonlinear.
PNL assumes more complicated structure between X,Y: Y fg((X) + Ea), where g. E1, f are the nonlinear effect, independent noise, and invertible post-nonlinear distortion, respectively. As ε can be represented as the difference (Y) - ). [67] proposes to identify the causal direction by recovering independent noise. If they can find functions 4, l2 such that IL X for e1 = (YY - 4 (X), then they say that the causal hypothesis X - Y 'holds' (Decision Rule E.3).
While valid for restricted ANM-UM cases (c.g., single nonlinear mediator), this approach may fail with multiple mediators due to f's invertibility requirement (Lemma 3.2, proof in Appendix E.5).
Lemma 3.2 (PNL Residual- Independence Fails). Assuming a consistent ICA residual estimator and access to infinite data, Decision Rule E.3 fails to recover the correct causal direction when there exists at least one non-invertible nonlinear mediator.
Prior work [59, 47, 67] propose alternative rules to compare measures of dependence, rather than independence, to improve finite sample performance (see Appendix E.4 for more details). However, empirically, we find this heuristic often fails (see Section 5).
Score-Matching
The original score- matching method SCORE [52] (with several followup works [36, 56] leveraging the same fact) relies on the assumption of Gaussian noise and nonlinear mech- anisms identify the effect via condition the Jacobian of the score function (V log p(X, Y)). Montagna et al. [35] prove that SCORE can fail to correctly decide causal direction when the noise is non-Gaussian, proposing NoGAM noise agnostic solution for nonlinear ANM. They further extend NoGAM to Adascore [37], which they prove correctly recovers the causal direction for all identifiable ANM.
Adascore identifies the causal direction by proving that only the residual from nonparametrically regressing the effect onto the cause is a consistent estimator of particular expression involving the score (Rule E.4). However, their theory relies on the estimated residual being independent from the cause, which, demonstrated in Eq. (3.1) may be false in some ANM-UM. Thus, Decision Rule E.4 fails to identify X --+ Y. We formalize this intuition in Lemma 3.3 (proof in Appendix E.6)
Lemma 3.3 SScore-Matching Fails). Assuming consistent estimator of the conditional expectation and access to infinite data, Decision Rule E.4 fails recover the correct causal direction when there exists at least one nonlinear mediator.

---

MSE Minimization Key methods include CAM [5], NoTEARS [68] and GOLEM [43], and NoTEARS-MLP [69]. The causal direction is determined by comparing prediction error: whichever variable better predicts the other (lower MSE) is designated the cause (Rule F.5). While effective in some synthetic settings, this rule suffers from two key flaws: 1) standardizing degrades performance [50], and 2) the 1-2 loss is only lower in the causal direction under restrictive variance conditions [44], which may not hold under ANM-UM (Lemma 3.4, proof in Appendix E.7). Intuitively, the causal direction becomes unidentifiable when R2 -ortaabbyyy vanishes (i.e., equal prediction errors in both directions), a problematic limitation since DGPs may exhibit arbitrary R2 values [51].
Lemma 3.4 (MSE-Minimization Fails), Given a consistent estimator of the conditional expecta-
tion and infinite data, Rule E.5 fails to recover the correct causal direction when E, E[[ar[X <
B[
3.2 Hidden Mediator Method-CANM
Assuming nonlinear mechanisms, CANM [6] uses a variational autoencoder (VAE) framework to:
1) learn latent noise via VAE (X,Y + WN )))) 2) compare the evidence lower bound (ELBO, Eq. (3.2)) scores for both causal directions, and 3) infer causation via higher ELBO (Rule E.6). Critically, CANM minimizes reconstruction loss combined with regularization term on the latent space, which does not align directly with the assumptions of the ANM-UM. This means that, while CANM succeeds empirically on synthetic non-invertible Gaussian DGPs, it lacks theoretical guarantees, even for the standard ANM without mediators. We further note that VAB-based methods in general require specifying the dimensionality of the latent space, i.e., the number of unobserved mediators, which is difficult to estimate and can affect performance. In contrast, denoising diffusion does not require estimating this unknown parameter. Therefore, for tasks where aspects of the underlying causal structure (such as the number mediators) are both important and hard to ascertain, propose that it is theoretically justified prefer diffusion over VAEs.
Our experiments (Section 5) show failure cases with: 1) linear/invertüble mechanisms, 2) non- Gaussian noise (often exhibiting posterior collapse, see Appendix E.9). As VAE training often encounters posterior collapse in practice, next we examine the behavior of CANM under this phenomenon. Posterior collapse causes CANM's learned N M(u, o²) to degenerate to A A(0, 1). This eliminates the KL term in ELBO and reduces the objective to the sum of negative entropy of and the conditional log-likelihood of Y X (Eq. (3.3)):
ELBOx-Y = B(log + -KLL (go(1) x. v) p(n)) + (n logp y f(x, m; 0))
(3.2)
= -X) + E Elooprix (E= = f(x; 6))
(3.3)
When posterior collapse occurs, CANM is provably inconsistent for ANM-UM where this sum is not higher for the causal direction (Lemma 3.5, proof in Appendix 10):
Lemma 3.5 (CANM Fails). Assuming infinite data and a consistent estimator of the conditional expectation, Rule E.6 fails to recover the causal direction if posterior collapse occurs and the expected conditional likeeiill minus the entropy higher in the causal direction.
4 Bivariate Causal Discovery Using Diffusion
In this section, we develop our conditional diffusion- method for distinguishing between cause and effect generated by the ANM-UM. We first warm up by developing intuition in the linear setting about when denoising leads to predicted noise that is independent of one of its input variables. We then spell out decision rule for deciding the causal direction that leverages the developed intuition, providing theoretical guarantees of correctness under certain restrictions the ANM-UM. We end by introducing practical method for denoising-diffusii for bivariate discovery, BiDD, and providing its computational complexity.
4.1 Denoising and Independence
To better understand what asymmetries may arise from denoising in the causal vs. anticausal direction, we start with simplified setup, restricting ANM-UM to only linear mechanisms without unobserved
5

---

mediators. We let the DGP of X, Y follow
X X E1:
where 51 is non aan-Gassiinn (to ensure identifiability [59]). To show the asymmetry in this setup, we formulate two noising processes: one where keep X fixed and noise Y, and one where we keep Y fixed and noise X. In the noising processes, we inject independent Gaussian noise into the noised variable, obtaining the noised terms
= + +ex and Y -x, 1).
(4.1)
Now, in the denoising processes, we aim to find the best estimators fex Jey such that MSE losses
(ey (V, and -ex (X,)))
(4.2)
minimized. Intuitively, the unnoised variable contains information about the noised one, so including it can enhance noise prediction and reduce the loss. However, this inclusion may also introduce dependence between the predicted noise and the unnoised variable. Consider the case where X is the cause and Y is the effect. Under the ANM-UM assumption (Assumption 2.1), this implies that Y is function of X and independent noise terms. Let (Xi, Y;) be fixed sample from the dataset. Suppose we add noise to Y; to obtain noised version Y Y; + εy, When predicting εY, from Y, and X,, the estimate Ey; may depend on X;, since X; contains information about the true value of Y, This induces statistical dependence between X and the predicted noise eý; The same argument applies in the anticausal direction. However, the dependence is not symmetric, as the functional relationship between X and Y is not symmetric. This asymmetry in dependence may provide a useful signal for causal identification and motivates our use of a conditional diffusion model. Specifically, we expect that the independence test outcomes for the pairs (X, Jey (Ý, X)) and fex ()) to differ. We now formalize this intuition.
Causal Direction: Denoising Y and Testing Independence between X and fey (Y, X) Given infinite data, the best estimators - f of the MSE loss converges to the conditional expectation [35]. This implies that the prediction of injected ey equals
E = EEsy X.
(4.3)
Substituting Y X + s1 into y = + ty, we have:
y =1 =1
(4.4)
Next, we will show that y --X X is a sufficient statisties for Eley Y, ie, Bley |Ý, X] ] ElEY
Y X]. To see this, we observe that since
E1 and X 11 Ey, we have Ey 11 X E1 + =
EY 1L x Y X. This implies that
Ły ElEy Y X] = BlEY X X] EiEy Y Y-X Eley E1 + +yl,
X]
(4.5)
where the second equality is due to the parametrization of Y and X; the third equality is due to
εy IL X &1 + = εy 11 X y - X, and the last equality is due to Eq. (4.4).
Now, as our conditional expectation in Eq. (4.5) shown consist of terms entirely independent of X, we have that our predicted noise is independent of the un-noised conditioning variable:
ē X.
(4.6)
Anticausal Direction: Denoising X and Testing Independence between Y and SEx (X, Y) In the anticausal direction, we repeat the same calculation and observe that the noise prediction is no longer independent of the input unnoised variable. First, substituting Y = X + E1 into X X +ex, we obtain
ğ-Y E1 + EX.
(4.7)
We note that the same argument in the causal direction longer works here as X - Y is not sufficient statistic for E[Ex X, Y]. In fact, show that

---

Lemma 4.1. εx B X, Y + Y.
The proof of Lemma 4,1 (Appendix E.1 proceeds by contradiction. While prior diffusion-based approaches have focused on leveraging diffusion to estimate the Jacobian of the score function [56], to our knowledge we are the first to point out an asymmetry arising from the independence of the predicted noise. Although the intuition is developed on a simple linear DGP, we hypothesize that the same argument generalizes to nonlinear DGPs, leading to more dependent predicted noise in the anticausal direction.
4.2 Theoretical Guarantees
Building on the intuition that we developed in Section 4.1, we build a decision rule that identifies the correct causal direction according to which denoising process (denoising Y or X) leads to prediction that is less dependent on the unnoised variable.
Decision Rule 1 (Bivariate Denoising Diffusion (BiDD)). Let X), Êx EX,o Y) be the predictions of the noise added to Y, respectively. Given a mutual information estimator MI(-). if MI(Éy,X) X) < MI(xx,)), conclude that X causes Y. If MI(Éy, MI(Ex. Y), conclude that X causes Y. Else, do not decide.
When the ANM-UM reduces to ANM (i.c., when Lemma 2.3 does hold), we can guarantee the correctness of Decision Rule 1 (Theorem 4.2, proof in Appendix E.12).
Theorem 4.2 (Consistency of Decision Rule 1). Suppose X, Y follow Eq. (2.1), Assumptions 2.1 and 2.2 hold, and no nonlinear mediator exists. Then, given a consistent mutual information estimator and infinite data, Decision Rule I correctly recovers the causal direction between X, Y.
We note that, to our knowledge, Theorem 4.2 represents the first theoretical result on using denoising diffusion for bivariate causal discovery by leveraging asymmetries arising from the independence structure of ANMs. We discuss the major challenges in extending Theorem 4.2 to the irreducible case, i.e., when conditions described in Lemma 2.3 hold, in Appendix E.13.
Based on intuition from the ANM case, we conjecture that Decision Rule I remains consistent for cases when the ANM-UM does not reduce to ANM, such as when there is nonlinear mediator. Note that X and Y are functionally related, conditioning noise can induce dependence between the predicted noise and the conditioning variable in either direction. Under the ANM-UM, Y can be modeled as a series of transformations of X where at each step an independent error is injected additively. Intuitively, we hypothesize that the additive noise is casier to decouple from Y when denoising Y while conditioning on X, thereby rendering the estimate of the noise added to Y (fy) less dependent on X. In contrast, in the anticausal direction, X can only be modeled as a series of transformations of Y where the error is not additively injected; this would imply that denoising X while conditioning on Y leads to a predicted noise that is more dependent on Y. Therefore, we focus the gap between the dependencies in the two directions as the signal for inferring causal direction. We validate this conjecture empirically (Section 5), finding that our approach performs well across wide variety of DGPs.
either variable when predicting the added
4.3 BiDD: A Practical Bivariate Denoising Diffusion Approach
Guided by the intuition developed in the linear case, we now present BiDD, practical method for inferring causal direction based on asymmetries in the independence of denoising estimates.
BiDD fits two conditional diffusion models, one for each direction. For B A. we corrupt A with noise and train to recover it given B, and vice versa. We then compare dependence between predicted noise and the condition, choosing the direction with lower dependence. We now describe each of these steps in detail. Additional information can be found in Appendix F, where we also formalize the procedure in Algorithm
Noise Prediction We train a neural network to reconstruct the Gaussian noise injected into a noised sample År, conditioned on B. Our training follows the standard denoising diffusion framework of Ho et al. [15] and its conditional extensions [53].
7

---

Let ferter denote a fixed noise schedule and let â. = -II be its cumulative product. For variable A and a diffusion timestep we define the noised version:
A v vā A + V E e~,
(4.8)
Given (A,, B,t) , the model A trained minimize the noise prediction loss:
LCDM = E,,.,s, ||e - c0(A,, B, B, t レ~
At each iteration, we sample a timestep t, generate A,, and update Eq by minimizing LCDM with stochastic gradient descent over E' epochs. This yields trained model E,,., which predicts A EA,0 (Ar, B, t).
Note that our theoretical results Section assumes that, for each direction denoising, the noised variable satisfies À A + with A TL E, where A denotes either X or Y. In practice, BiDD uses the rescaled form = V + /1 - ū,ξ to ensure that the forward process converges to a standard Gaussian, is common in denoising diffusion models. However, our analysis still remains valid because independence is preserved under linear transformations.
Dependence Testing After training, we evaluate the model on the set. For each diffusion timestep & = l,... T", we generate noised inputs to estimate the dependence between the predicted noise and the conditioning variable. Specifically, for each test sample (A,, B:), we construct noised versions {A (8)
1,1 t.k by sampling independent noise E ~ ~N(0, 1) & times.
We then apply the trained model to obtain noise predictions ÊA,i.j = c..o (A, noised sample. The mutual information MI is computed between the predicted noises and the conditioning variable B as MI MI (É B).
t.j B,, t) for each
We repeat the procedure in the reverse direction by training a second model that predicts &g from noised B and conditioning on A, and compute MIg,; analogously.
Inferring Causal Direction To determine the causal direction, we compare MIA, and MI r for each timestep i. We count how often one direction yields lower mutual information value and select the direction that does so more frequently across timesteps. A formal description of this procedure is provided in Subroutine 3 in Appendix G.2.
While our theoretical framework assumes sample splitting between training and testing, we find in practice that using the full dataset for both training and dependence estimation often improves performance, consistent with observations from prior work [18]. Therefore, we empirically evaluate two variants: BiDDrest which uses a held-out test set for dependence estimation, and BiDD which uses the full dataset. Additional implementation details, including learning rate, optimizer, noise schedule, and estimator configuration, are provided in Appendix G.2.
Computational Complexity The computational complexity of BiDD involves two main stages. Firstly, the training of two conditional denoising diffusion models, each for E epochs over Miain training samples. If Csee denotes the cost of single neural network training step (forward pass, loss computation, backward pass, and parameter update)
sample, this stage has a complexity of O(F MTrrin Cste)). Secondly, the inference stage as per Decision Rule requires generating noise predictions for 1 = k- meval evaluation samples per timestep T in the model (costing O( k- T Ciwa), where Ciwd is the neural network forward pass cost) and computing two mutual information (MI) estimates. If CMI.2 is the cost for one MI estimation on / samples, this adds CMI.2 to the inference cost. The overall computational complexity of BiDD is thus O Wirrain Cstep + T(Meval k. Crww + CMT,)), which is typically dominated by the O(E 17 Cstep) training component.
5 Experimental Results
We evaluate BiDD on synthetic data with linear, nonlinear invertible, and nonlinear non-invertible mechanisms, as well as real-world dataset [55]. BiDD achieves oate-of-the-ar and consistent performance across settings, while most basclines performs poorly in least one setting.

---

Method Noise
Linear Unif.
Neural Net
Quadratic
Tanh
Gauss.
Unif.
Gauss.
Unif.
Gauss.
Unif.
BiDD Total
0.77
.93
.90
1.00
1.00
0.80
.93
BiDD Test
0.73
.97
.97
1.00
1.00
0.60
0.83
Table Accuracy of BitD and BiDDrest across different transformation-noie combinations, with no mediators and n 1000.
Method Noise
Linear Unif.
Neural Net
Quadratic
Tanh
Gauss.
Unif.
Gauss.
Unif.
Gauss.
Unif.
BiDD Total
0.83
0.87
0.97
1.00
1.00
0.80
0.83
BiDD Test
0.80
0.87
1.00
1.00
1.00
0.63
0.77
CANM
0.10
0.93
0.87
1.00
1.00
0.50
0.10
Adascore
0.93
0.73
0.77
0.43
0.13
0.67
1.00
NoGAM
1.00
0.43
0.43
0.00
1.00
0.63
1.00
SCORE
1.00
0.73
0.57
0.43
1.00
0.50
1.00
DagmaL
0.13
0.17
0.10
0.00
0.00
0.00
0.07
CAM
0.03
0.77
0.80
1.00
1.00
0.93
0.13
PNL
0.73
0.83
0.70
0.83
0.83
0.67
0.70
RESIT
0.93
0.70
0.67
1.00
1.00
0.87
1.00
DLiNGAM
1.00
0.13
0.13
0.10
0.40
0.17
1.00
Var-Sort 0.43
0.57
0.63
0.47
0.57
0.33
0.60
Table 2: Accuracy of methods across different transformation-note combinations, with one mediator and n 1000. Bold indicates best, underline indicates second-bes
5.1 Setup
Synthetic Data Details. We produce synthetic bivariate causal pairs under the following ANM-UM (Eq 2.1), with varying causal mechanisms, exogenous noise distributions, sample size, and number of mediators. We use linear mechanisms with randomly drawn coefficients; we use both invertible (tanh) and non-invertible (quadratic, neural networks with randomly initialized weights [29, 23, 14)) nonlinear mechanisms. We use both uniform and Gaussian noise (excluding the linear Gaussian case to ensure identifiability) We standardize the data to mean 0 and variance 1 to ensure that the simulated data is sufficiently challenging; methods are evaluated on randomly generated seeds in each experimental setting. See Appendix G.1.1 for details on parameters choice for each DGP.
Real- World Data Details. To confirm the real-world applicability of our approach, we test BiDD on the Tübingen Gause-Effeet dataset [41], a widely used bivariate discovery benchmark that consists of 99 causal pairs that may have unobserved mediators. Due to runtime issues with baselines, we subsample the dataset of each causal pair by randomly selecting up to 3000 data points.
Baselines and Evaluation. We benchmark BiDD against mix of classical and SOTA methods: compare against three Residual-Inndependence methods (DirectLiNGAM, RESIT, PNL). three Score- Matching methods (SCORE, NoGAM, Adascore), MSF-Minimizztion methods (DagmalLinear [3], CAM), and the only hidden mediator method in the literature (CANM). We include the heuristic algorithm Var-Sort, which exploits artifacts common simulated ANMs [50], to show that BiDD performance is not driven by such shortcuts. Similar to [41], we use the accuracy for forced decisions, which corresponds forcing the compared methods to decide the causal direction.
5.2 Results
Synthetic Data. We first examine the performance of BiDD in the traditional ANM setting (no unobserved mediator): Table shows results for BiDDToual on data generated by different mechanism-noise combinations and sample size = 1000. We observe the robust performance of BiDDTotal, achieving 2 77% accuracy across all mechanisms. This empirically confirms theoretical correctness guarantee given in Section 4 (Theorem 4.2).

---

0.8
0.8
0.6
0.6
BIDD
BiDD
CANM
0.4
BIDDia
0.2
0.2
RESIT
2000
2
3
Number Meddators
Sample Size
1000 with increasing number of mediators.
(b) One unobserved mediator, with increasing n.
Figure 2: Tanh mechanism, uniform noise setting.
Method
BiDD Total
BiDD Test
CANM
CAM
Adascore
Entropy
Accuracy
0.64
0.60
0.47
0.56
0.06
0.36
Method
DagmaLinear
DircetLiNGAM
NoGAM
RESIT
PNL
SCORE
Accuracy
0.30
0.51
0.69
0.62
0.61
0.65
Table 3: Accuracy for Tübingen Causal Pairs dataset, 3000
We examine how BiDDrotal performs under unmeasured mediators: in Table 2 we display results for different mechanism-noise combinations, each generated with one unobserved mediator (i.e., Y f2(f((X) + E1) +2) and sample size 1000. We observe the robustness of BiDD as it achieves 2 80% accuracy across all experimental setups, getting the first or second best accuracy 5/7 times. In contrast, all baselines except PNL and RESIT perform extremely poorly (s 50%) in at least two settings. PNL's performance is significantly lower (~~ 10% 20%) than BiDD in almost every setting, while RESIT struggles the neural network setting (< 70%). The other hidden mediator method, CANM, performs poorly (< 50%) for invertible mechanisms (linear, tanh), even for Gaussian noise, which consistent with our analysis of CANM's behavior under posterior collapse (see Appendix E.9). The degraded baseline performance when the ANM assumption is violated highlights the limited applicability of current bivariate ANM methods. The performance gap becomes more pronounced when adding a second mediator (see Appendix H.3).
In Figure 2, investigate how BiDDTotal performs under fixed sample size (n 1000) and varying depth (Figure 2a), or fixed depth (one mediator) and varying sample size (Figure 2a), in the tanh mechanism, uniform noise setting. In Figure 2a we observe that as the number of mediators increases, the performance of RESIT and PNL both degrade (to ~ 60%), while BiDDTotal remains performant (~~ 95%). This shows that the performance of Residuall Independence based methods (RESIT and PNL) is sensitive to mediator number, while our diffusion approach remains robust. We observe similar trend for the non-invertable case (see Appendix H.4)
In Figure 2b we observe the consistency of BiDDToual as its accuracy approaches 100% while CANM does not improve, and in fact seems decrease in performance. This points to CANM being inconsistent in settings with unmeasured mediators, rather than merely having finite sample issues.
Real-world Data The results of Tübingen dataset are presented in Table 3: BiDDTotal (64%) performs comparable to the best baselines, NoGAM (69%) and SCORE (65%), outperforming the rest of the methods. This confirms the robustness of BiDD across diverse real-world setups.
Diseussion. Future work includes extending BiDD to be robust to latent confounding, and analyzing its potential consistency in settings where ANM-UM cannot be reformulated as an ANM.
10

---

Acknowledgments
This work was supported in part by an AWS Cloud Credit Grant from Cornell's Center for Data Science for Enterprise and Society.
References
[1] Aczél, On applications and theory of functional equations. Academic Press, 2014.
[2] Addo, P. M., Manibialoa, C., and Mclsaac, Exploring nonlinearity on the co2 emissions, economic production and energy 6196-6204, 2021.
causal discovery approach. Energy Reports,
[3] Bello, K., Aragam, B., and Ravikumar, P. DAGMA: Learning DAGs via M-matrices and hog-Determinant Acyelicity Characterization. In Advances in Neural Information Processing Systems, 2022.
[4] Bloebaum, P., Janzing, D., Washio, T., Shimizu, S., and Schoelkopf, B. Cause-effect inference by comparing regression errors. In Storkey, A. and Perez-Cruz, F. (eds.). Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 18a. html.
PMLR, 09-11 Apr 2018. URL
https:/ petps://proceedingg mlr gress/v84/blee
[5] Bühlmann, P., Peters, J., and Ernest, J. CAM: Causal additive models, high-dimensional order search and penalized regression. The Annals of Statistics, 42(6), December 2014. ISSN 0090-5364. doi: 10. (0.1214/14-A0112666 URL http://arxixi http:///xxxvv.or/aaabb wwtp://////a//////eeee 1533. arXiv:1310. 1533 [cs, stat].
[6] Cai, R., Qiao, J., Zhang, K., Zhang, Z., and Hao, Z. Causal discovery with cascade nonlinear additive noise models. arXiv preprint, arXiv:1905.0944 2019.
[7] Cai, R., Ye, J., Qiao, J., Fu, H., and Hao, 7. Fom: Fourth-order moment based causal direction identification on the heteroscedasticc data. Neural Networks, 124:193-201, 2020. ISSN 0893-6080. doi: acc.encc//aattceeee///////eeooo0 sciencedirect.co com/ http://///ccccccccca/rreeeeeee/e
.006. URL https: //wp:://wwwww.
[8] Chickering, D. M. Learning Equivalence Classes of Bayesian Network Structures. Journal of Machine Learning Research, 2013.
[9] Fu, H., Li, C., Liu, X.. Gao, J., Celikyilmaz, A., and Carin, L. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. In Burstein, J., Doran, C., and Solorio, (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 240-250, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.11653/1//119-100 URL /
org/N1gg// Atg/N19-1021/
[10] Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., and Smola, A. A kernel statistical test of independence. Advances in neural information processing systems, 20, 2007.
[11] He, J., Spokoyny, D., Neubig, G., and Kerg-Kirkpatrick, T. Lagging inference networks and 2019.
posterior collapse in variational autoencoders. arXiv preprint
[12] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and
Lerehner, A. Beta-vae: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017.
[13) Hiremath, S., Maasch, J., Gao, M., Ghosal, P., and Gan, K. Hybrid top-down global causal discovery with local search for linear and nonlinear additive noise models. NewrIPS 2024, 2024. URL httt::ttts/arreee org/ abs/ /2005240020 14496. https//arxivvorr/abb///// 14496.
[14] Hiremath, S., Ghosal, P., and Gan, K. Losam: Local search in additive noise models with mixed mechanisms and general noise for global causal discovery. UA/ 2025, 2025. URL https: arxii.ivvvv..vva org/ abs/2410. 11759. https://ariv.or//a//411...1111
11

---

[15] Ho, J., Jain. A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6833 2020.
[16] Hoyer, P. O., Janzing, D., Mooij. M., Peters, J., and Schölkopf. B. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems 2/ (NIPS 2008), 2008.
[17] Huang. B., Zhang, K., Lin, Y., Schölkopf, B., and Glymour, C. Generalized Score Functions for Causal Discovery. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1551-1560, London United Kingdom, July 2018. ACM. ISBN 978-1-4503-5552-0, doi: 10. (0.11445/2218819332201
URL https://td.... acm.
org/doi/10, 1145/3219819 3220104.
[18] Immer, A., Schultheiss, C., Vogt. J. E., Schölkopf, B., Bühhmann, P., and Marx, A. On the Identifiability and Estimation of Causal Location-Scale Noise Models, June 2023, URL http:///axxvvv.og//a// /210..2210000 09054. arXiv:2210.0905: [cs, stat].
[19] Janzing, D. and Scholkopf, B. Causal Inference Using the Algorithmic Markov Condition. IEEE Transactions on Information Theory, 56( 36(10)55168 11175518811444 October 2010. ISSN 0018- 9448. 1557-9654, doi: 10. 00.1009/TIT22000200000999 URL http:///eeexploree ieee org/ document/5571886/
[20] Janzing. D., Mooij, J., Zhang, K., Lemeire, J., Zscheischler, J., Daniušis, P., Steudel, B., and Schölkopf, B. information-geomettee approach to inferring causal directions. Artificial Intelligence, 182-183:1-2 May 2012. ISSN 00043702. doi: 10. 10.10166fjartim222112. 2012000000...0...........0...... URL
elsevier com///errieoo//11/80333300000000
[21] Janzing, D., Peters, J., Mooij, J., and Schölkopf, B. Identifying
ceniiiyiincoooooudeee confounders using additive
noise models. arXiv preprint arXiv: 1205. 2012.
[22] Jonas Peters, Dominik Janzing, and Berhard Scholkopf. Elements of causal infer- ence: foundations and learning algorithms. The MIT Press, Cambridge, Massachusetts, 2017. ISBN 978-0-262-03731-0 URL https://mitpress wtteess......uuuuuu..mmm ciii.ech//770002203330/// el elements-of causal inference/.
[23] Ke, N., Chiappa, S., Wang, J., Bornschein, J., Goyal, A., Rey, M., Weber, T., Botvinick, M., Mozer, M., and Rezende, D. Learning induce causal structure. International Conference on Learning Representations, 2023.
[24] Kelly, M., Longjohn, R., and Nottingham, K. The UCI Machine Learning Repository. https: // //archive. ics. uci. edu, 2025, Accessed 22 May 2025.
[25] Kolmogorov, A. N. Three approaches the quantitative definition ofinformation'. Problems of information transmission, 1(1):1-7, 1965.
[26] Kraskov, A., Stögbauer. H., and Grassberger, P. Estimating mutual information. Physical Review E, 69(6))066138,, June 2004. ISSN 1539-3755, 1550-2376. doi: 10. 100.//hy.eee..6.0666613 URL https://link //link. aps. org/ 10. 1tw.0...11PPPP.. 1103/PhysRev...comm 69. 06611118...666.... 066138.
[27] Lam, W.-Y., Andrews, B., and Ramsey, J. Greedy Relaxations of the Sparsest Permutation Algorithm, 2022. URL https::/ http://pooooogggg....ppeeeeee///
mlr. wwwps//ewe//aae//////ee hatt://mmmmmammmmmm/ 2df.f.f..........
[28] Lee, J., Srinivasan, R., Ong. C. S., Alejo, D., Schena. S., Shpitser, I., Sussman, M., Whitman, G. J., and Malinsky, D. Causal determinants of postoperative length of stay in cardiac surgery using causal graphical learning. The Journal of Thoracie and Cardiovaseular Surgery, pp. $002252232209900X, August 2022. ISSN 00225223. doi: 10. elsevier cmm//eerrivee/p1//022222220000
URL
[29] Lippe, P., Cohen, T., and Gavves, Efficient Neural Causal Discovery without Acyelicity Constraints, February 2022. URL htttp://arxxv..or/abb//// wwtp:////////.....///////// 10483. arXiv:2107.10483 [cs, stat].
[30] Liu, M., Sun, X., Qiao, Y., and Wang. Causal discovery via conditional independence testing with proxy variables. ICML 2024, 2024. URL https: //tpx/axvvvv///////////e //t///awwvvoogppdf.///// /2305 05281.
12

---

[31] Maeda, T. N. and Shimizu, S. Causal Additive Models with Unobserved Variables. In Pro- ceedings of the Thirty-Seventh Conference on Uncertainty Artificial Intelligence, pp. 10, 2021.
[32] Marx, A. and Vreeken, J. Identifiability of cause and effect using regularized regression. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 852-861, 2019.
[33] Miao, W., Geng, Z.,, and Tchetgen Tchetgen, E. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika, 105(4):987-993, 2018. ISSN 0006-3444.
[34] Miao, W., Hu, W., Ogburn, E., and Zhou, X.-H. Identifying effects of multiple treatments in the presence of unmeasured confounding. Journal of American Statistical Association, 2022. URL https:/ htwpp::///www.tt////// wandfonllan com/doi, wwwp////wl/oo/////eeeeeeeeeeeeee htt.://10.11000001221221100001 2021. 2023551# abstract.
[35] Montagna, F., Noceti, N.. Rosasco. L., Zhang, K., and Locatello. F. Causal Discovery with Score Matching on Additive Models with Arbitrary Noise. In Proceedings of the 2nd Conference on Causal Learning and Reasoning. arXiv, April 2023. URL http: /tt//axxxvvvoo//eeee..e www..oorgg/bb/////ee 03265. arXiv:2304. ..02655 [cs, stat].
[36] Montagna, F., Noceti, N., Rosasco, L.,, Zhang, K., and Locatello, F. Scalable Causal Discovery with Score Matching. In Proceedings of the 2nd Conference on Causal Learning and Reasoning. arXiv, April 2023. URL http: / arxiv. org/ abs 2304. 03382. #xXiv:2304.03386 [cs, stat].
[37] Montagna, F., Faller, M., Bloebaum, P., Kirschbaum, E.,, and Locatello, Score matching through the roof: linear, nonlinear, and latent variables causal discovery. arXiv preprint arXiv:2407. 18755, 2024.
[38] Mooij, J. and Janzing, D. Distinguishing between cause and effect. JMLR Workshop and Conference Proceedings, 6:147-156, 2010.
[39] Mooij, J., Janzing, D., Peters, J., and Schölkopf, B. Regression by dependence minimization and its application to causal inference in additive noise models. Proceedings of the 26th annual international conference on machine learning, pp. 745-752, 2009.
[40] Mooij, J. M., Peters, J., Janzing, D.,, 7scheischler, J., and Schölkopf, B. Distinguishing cause from effect using observational data: methods and benchmarks. Journal of Machine Learning Research, 17032):1-102, 2016.
[41] Mooij. J. M., Peters, J., Janzing, D., Zscheischler, J., and Scholkopf, B. Distinguishing Cause from Effect Using Observational Data: Methods and Benchmarks. Journal of Machine Learning Research, 17, 2016.
[42] Mooij, J. M., Peters, J., Janzing, D., Zscheischler, J., and Schölkopf, B. Distinguishing cause from effect using observational data: methods and benchmarks. Journal of Machine Learning Research, 17(32):1-102, 2016.
[43] Ng. I., Ghassami, A., and Zhang, K. On the role sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems,
2020.
[44] Park, G. Identifiability of additive noise models using conditional variances. Journal of Machine Learning Research, 21(75): 2020.
[45] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z.. Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang. L., Bai. J.. and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, volume of NeurIPS, pp. 8026 8037. Curran Associates Inc., Red Hook, NY, USA, December 2019.
[46] Pearl, J. Myth, confusion, and science in causal analysis. Technical Report R-348, University of California, Los Angeles, Los Angeles, CA, 2009. URL http::/ftt...../ cs ucla. edu/pub/ stat_ser/r349 pdf.
13

---

[47] Peters, J.. Mooij, J., Janzing, D., and Schölkopf. B. Causal Discovery with Continuous Additive Noise Models, April 2014. URL http:://axiii org aaaaaaas/000 1709.999... arXiv:1309.6779 [stat].
[48] Peters, J., Janzing, D., and Schölkopf, B. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[49] Pham, T., Maeda, T. N., and Shimizu, S. Causal additive models with unobserved causal paths and backdoor paths. arXiv preprint arXiv:2502.072 07646, 2025.
[50] Reisach, A., Seiler, C., and Weichwald, S. Beware of the simulated dag! causal discovery benchmarks may be easy game. Advances in Neural Information Processing Systems, 34: 27772-27784, 2021.
[51] Reisach, A. G., Tami, M., Seiler, C., Chambaz, A., and Weichwald, S. A Scale-Invariant Sorting Criterion Find Causal Order in Additive Noise Models. In 37th Conference on Neural Information Processing Systems. arXiv, October 2023, URL httt::/tta/rrteeee www.i.owgg/a....a..oo.o.aaoowooe wtw.i.oorg/bbb/20333 [cs, stat].
18211. arXiv:2303.
[52] Rolland, P., Cevher, V., Kleindessner, M., Russel, C.,, Scholkopf, B., Janzing, D., and Lo- catello, F. Score Matching Enables Causal Discovery of Nonlinear Additive Noise Models. In Proceedings of the 39 th International Conference on Machine Learning, 2022.
[53] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image syn- thesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
[54] Runge, J., Bathiany, S., Bollt, E., Camps-Valls, G., Coumou, D., Deyle, E., Glymour, C., Kretschmer, M., Mahecha, M. D., Muñoz-Mari, J., van Nes, E. H., Peters, J., Quax, R., Re- ichstein, M., Scheffer, M., Schölkopf, B., Spirtes, P., Sugihara, G., Sun, J., Zhang, K., and Zscheischler, J. Inferring causation from time series in Earth system sciences. Nature Commu- nications, 10(1):2553, December 2019. ISSN 2041-1723, doi: 10. (0.1038/441467-019-100 URL http::t::10 www. nature. com/ wttaaaaeeeeee1ee-ee///////// -19- 10105-33.
[55] Sachs, K., Perez, O., Pe'er, D., Lauffenburger, D. A., and Nolan, G. P. Causal protein-signalinn signaling networks derived from multiparameter single-cell data. Science, 308(5721):523-553 22222,22-22111 2005.
[56] Sanchez, P., Liu, x., O'Neil, A. Q., and Tsaftaris, S. A. Diffusion models for causal discovery via topological ordering. arXiv preprint arXiv:2210.06201, 2022.
[57] Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., and Mooij. J. On causal and anticausal learning. arXiv preprint arXiv: 222.66766, 2012.
[58] Shimizu, S., Hoyer, P. O., Hyvarinen, A., and Kerminen, A. Linear Non-Gaussian Acyclic Model for Causal Discovery. Journal of Machine Learning Research, 1-2003-2030 2006.
[59] Shimizu, S., Inazumi, T., Sogawa, Y.. Hyvarinen, A., Kawahara, Y., Washio, T., Hoyer, P. O., Bollen, K., and Hoyer, P. Directlingam: A direct method for learning a linear non non-gaussian structural equation model. Journal of Machine Learning Research-JMI.R, 12(Ap)):1255-1248 2011.
[60] Spirtes, P. An Anytime Algorithm for Causal Inference. In Proceedings of the Eighth Inter- national Workshop on Artificial Intelligence and Statistics, volume R3, pp. 278-285. PMLR, 2001.
[61] Spirtes, P., Glymour, C., and Scheines, R. Causation. Prediction. and Search, volume 81 of Lecture Notes in Statistics. Springer New York, New York, NY, 2000. ISBN 978-1-4612-765000 URL http::///i/li springer com/
978- 178-1-4612-2745 doi: 10. 1-117778--111127-----11
[62] Tagasovska, N., Chavez-Demoulin V., and Vatter, T. Distinguishing cause from effect using quantiles: Bivariate quantile causal discovery. In International Conference on Machine Learning, pp. 9311 9311-9323, PMLR, 2020.
14

---

[63] Turing, A. M. et al. On computable numbers, with an application to the entscheidungsproolem.. J. of Math, 58(345-363):5, 1936.
[64] Xu, S., Mian, O. A., Marx, A., and Vreeken, J. Inferring cause and effect in the presence of heteroscedastic noise. In International Conference Machine Learning, pp. 24615-24630. PMLR, 2022.
[65] Xu, Z., Li, Y., Liu, C., and Gui, N. Ordering-based causal discovery for linear and nonlinear relations. In Proceedings of the 37th Conference Neural Information Processing Systems (NeurIPS), 2024.
[66] Yuan, Y. and Qu, A. De-confounding causal inference using latent multiple-mediitorr pathways. Journal of the American Statistical Association, 119(547):2551-2065 2023. doi: 10.10 01621459.2023.224444
URL https:////do.ootr http://tooooogg///////o 10. 1080/01621499 202.122215992222.....22 2240461.
[67] Zhang, K. and Hyvarinen, A. On the Identifiability of the Post-Nonlinear Causal Model. Uncertainty Artificial Intelligence, 2009.
[68] Zheng, X., Aragam, B., Ravikumar, P. K., and Xing, E. P. Dags with no tears: Continuous optimization for structure learning. Advances in neural information processing systems, 31, 2018.
[69] Zheng, X., Dan, C., Aragam, B., Ravikumar, P., and Xing, E. Learning sparse nonparametric dags. In International Conference on Artificial Intelligence and Statistics, pp. 3414-3425. Pmir, 2020.
[70] Zhu, Z., Locatello, F. and Cevher, V. Sample Complexity Bounds for -ctre-Matching: Causal Discovery and Generative Modeling, 2023.
URL
htttps:/pproceeddnggg neurips cc/paper #asccca229.caals5see124444eedddd Paper-Conference.pp pdf.
15

---

Appendix
A Notation
Pa(x:)
The set of parent vertices of 24.
Z
The i'th Mediator
fi
Arbitrary function, generating Z,
An independent noise term sampled from arbitrary distribution.
X
is independent of Y
X
X is not independent of Y is parent
X-YY
in the ANM-UM X is parent of Y in the ANM-UM Set integers {1, .., 7}
X --+Y
[T]
(Zihke
Set of Mediators {21, ZT}
A;
i'th data point
(A, Bibiel
Collection of data points
Conditional expectation of Y given X
The inverse function of /
V
Gradient operator
Var[V[x]
Variance Y given X
&
Noised version of X
Sex
Predictor of the noise added X
fex
The best predictor of the noise added to X. Best means lowest MSE Empirical mutual-information estimator
MI(-)
EA
Noise added to = E
EA.
A learned model for predicting сд with parameters 0
0())
Big-O (Landau) notation for asymptotic upper bound
E
There exists
Partial derivative
Ex[]
Expectation given X
TH
Minimal-sufficient statistic
Partial derivative with respect to x
de
Af(u)
Difference of function shifted by constant at Af(u) f(u + f(u)
fx
Probability density function of X
ox.Y
Joint density of X and Y
Sigma algebra
C))(X,Y)
Covariance of X and Y
||
Euclidian norm X
10
Normal distribution with mean and variance 1
u(-1,1)
Continous uniform distribution between -1 and
Unif([1,...,.
Discrete uniform distribution between N elements
Indicator function
D
Bivariate Dataset, consisting of n observations of A and B
16

---

B Further Discussion of Related Bivariate Methods
In this section, we further clarify the difference in modeling assumptions employed by ANM-based methods, LSNM-based methods, and ICM-based methods.
Location Scale Noise Models. LSNMs follow Y = f(X) g(X),, where X IL E, allowing the noise term to be scaled and shifted for each X value, according to g(X). This increased flexibility models the possibility of heteroskedastic noise (noise dependent on the input X).
LSNMs are similar to ANM-UM, in that both include the vanilla ANM (Y = f(x + g(X))) as a subcase. However, we note that while LSNMs and ANM-UM overlap, ANM-UM cover many cases which LSNMs do not. For example, y = e + E2 can be modeled as a ANM-UM, while LSNMs do not admit such representation. In general, ANM-UM can admit much more complicated joint distributions because they attempt to account for unobserved mediators: this introduces multiple (rather than one) independent noise distributions, with multiple transformations (rather than one) of the original input.
Additionally, methods developed to exploit LSNMs have several drawbacks. They either lack theoretical guarantees, or they require parametric assumptions than the general LSNM case. For example, [64] require linear mechanisms, while [18, 7] require Gaussian noise for correctness results.
Principle of Independent Mechanism Approaches The independence of cause and mechanism postulate [57] (ICM) states that the cause X should be independent of the mechanism that maps X to the effect Y. More concretely, this means that X --+ Y only if the shortest description of Px,y is given by separate descriptions of Pyxx and Px, in the sense that knowing Px does not enable shorter description of Pylx (and vice versa) [20]. Here description length is understand in the sense of algorithmic information ("Kolmogorov complexity") [25].
The overall ICM postulate is true generalization of the ANM-UM (as well as the LSNM), as the and noise distributions ε ST+1 (which make up the data generating process from X Y) do not change for different input distributions of X. However, concrete methods developed to exploit ICM have many drawbacks. These generally follow from the fact that Kolmogorov complexity is known generally to not be computable [63]. Therefore. methods use approximations or proxies of the Kolmogorov complexity to develop heuristic approaches.
functional mechanisms f1,..., fr41
For example, [38] substitute the minimum message length principle, [62] use quantile scoring as proxy for Kolmogorov complexity, and [32] leverage condition on the parameter size of the true causal model implied by the Kolmogorov struction function. In general, methods based on the ICM do not come with strong identifiability results [32].
C Further Discussion of Related Global Methods
Recent work in global causal discovery has grappled with the issue of unobserved mediators, to various degrees of success. The multivariate version of Adascore [37] is the first score-matching method to handle unobserved confounders, but the authors clearly state that fails to correctly recover the graph when unobserved mediators are present (See Examples 4, 5, and 7 in [37]). [31] showed that, under further restriction of the ANM, where both the causal effects and error terms are additive (causal additive models, CAM, subcase of ANM-UM), it is possible to recover the correct causal edge when all parents of a variable are measured, and otherwise leave the causal edge undecided if an unobserved mediator is parent of an observed variable. In a recent extension of this work, Pham ct al. [49] have shown that, under the CAM restriction, the correct causal edge can be recovered if the unmeasured mediator that is parent of an observed variable is embedded in certain types of global graphical structures. However, neither of the latter two works comment on the general bivariate case involving unobserved mediators (AMM-UM), where additional global information may not be present.
Our bivariate method for handling unobserved mediators (BiDD) is motivated by the drawbacks of current ANM-based global discovery methods, as either they cannot handle unobserved mediators at all, fail to recover edges under hidden mediation, or can only do so under very narrow global graphical structures. Future work can incorporate BiDD as subroutine in a global ANM-based discovery method, providing utility in real-world systems where hidden mediation abounds.
17

---

D Problem Setup
D.1 Relation of ANM-UM to ANM, PNL, and CANM
Note that ANM-UM can be represented as = (f+1(fr(...)) + ST) ET+1, where the last term inside the is
is fi (X) + S1.
The traditional ANM [47] models f(X + E1, where the key constraint that X TT E1. If T = O for the ANM-UM, i.e., there are no unobserved mediators, then fi and fr+1 coincide, and Eq 2.1 reduces to = (x) E1, which is exactly the ANM.
The PNL [67] models Y = g(
) + S1) where X Il e1 and is an invertible nonlinear transforma- tion. If = for the ANM-UM, f2 is nonlinear and invertible, and 0T+1 0, then Eq 2.1 reduces to Y = f2(f((X +1), which is exactly the PNL.
The CANM [6] models Y fr all f Eq 2.1 are nonlinear, then ANM-UM reduces to the CANM.
ET) + ST+1 where all fi...., f nonlinear. If
D.2 Identifiability Discussion
We first note that any ANM-UM (Eq 2.1) can be represented equivalently as
(D.1) where E1......, ET+1 are all mutually independent, and F fr+1 + er)) For the ANM- UM to be identifiable, we require that there is no backwards ANM-UM that fits the anticausal direction X --+ Y, i.e. there does not exist G, ,T such that
F(X, .T) + ET+1
X
Er+1
(D.2)
where G= 01+ (9r-))) e and, additionally, E,é,,..., ,,,,-
-T4 are mutually independent. Theo- rem from [6] shows that for any G, , which satisfy Eq D.2, we have that Y, ,,,,,,..,,r, are mutually independent (and thus ANM-UM unidentifiable) if and only if, êTT1 takes very partic-
ular form:
/ | |f p(x) Per. f(x, e 22xiv dn dæ
Pêr+i (Ex+1) =
dv (D.3)
plu) / p(ñi) 2xig dî
where n = (s,,,..,,,T) and (
Therefore, to ensure identifiability. Assumption 2.2 requires that for any such G, C, which satisfy Eq D.2, ÊT+ does not satisfy Eq D.3.
Cai et al. [6] show that, when ANM-UM can be represented as a traditional ANM (i.e., ANM-UM satisfies conditions in Lemma 2.3), the Assumption of 2.2 reduces known identifiability constraints.
For example, in Corollary of Cai et al. [6]. they show that if all mechanisms fi,..., fr+1 in the ANM-UM are linear, then Assumption 2.2 reduces to requiring that at least one of X, {=,)
non-Gaute Gaussian. This is exactly the constraint described by Shimizu et al. [58]. In Corollary 2 of Cai al. [6], they show that if T 0 in ANM-UM (no unobserved mediators), then Assumption 2.2 reduces to requiring that X, f1,&1 81 satisfy the differential equation described the identifiability assumptions of the general ANM in Hoyer al. [16].
D.3 Nonlinear ANM Not Always Closed Under Marginalization
X Z - Y, which corresponds to the ANM-UM X, Zi =xx))xxtxxyx->
+ As
Y fe(fi(X) + E2, is straightforward that Y follows an ANM if and (f can be decomposed into the addition of a function (1 + A2(E1). Note that this follows the form of Pexider's equation C(r + D(x)+ Ely)- is known [1] that if C, D, F satisfy this equation, C, D. must all be linear functions. Therefore, if fe is nonlinear, the ANM-UM does not reduce. and if fe is linear, then the ANM-UM does reduce.
X and a function of e i.e. (2( +
18

---

D.4 Proof of Lemma 2.3
Lemma 2.3 (Irreducible ANM-UM and Nonlinear Mediator). Under ANM-UM (Eq. (2.1)) and Assump.s 2.1 and 2.2, Y does not admit a decomposition in Eq. (2.2) if and only if there exists mediator Z; such that Y h(.))) + E for some nonlinear ht with E Zi. Additionally, we call such mediator Zi nonlinear mediator.
Proof. Suppose that there exists a mediator Z, such that Y = h(22) + where h is nonlinear, and Z; 11 Then, 3 some function F such that Y ((F(X, E,....,,. E =2) + Suppose for contradiction that Y admits an additive decomposition, i.e. Y
An (X) + A2(E1,...;e) es) + Az(E). Note that this implies h(F(X, (1. D(x) + Ely). (1(X) A2(51..... E;),, which follows the form of Pexider's equation, C(x is known that C, D, E satisfy this equation, C, D, must all be linear functions [1]. However, this contradicts that h is nonlinear. Therefore, Y does not admit a decomposition in Eq 2.2.
+
Suppose that Y does not admit decomposition in Eq 2.2. Suppose for contradiction that there does not exist nonlinear mediator Z, This implies that .......ff+. are all linear functions. Then, Y can be written as linear function of Z1 and noise terms E2, ................................ =T+1, i... Y = aZ1 + B,a,. Then, Y = a (X) + ase + + 8:,; which follows the additive decomposition in Eq 2.2. Therefore, there must exist nonlinear mediator Z₁.
E Failure-mode of Prior Work + Proof of Lemma 4.1, Theorem 4.2, Experimental Analysis of CANM
E.1 Decision Rules
Decision Rule E.2 (Regression Residual--Independenee Lef e1, e2 be the residuals obtained from regressing Y onto X, and X onto Y (respectively). Ife1 1I. X, e2 V.. y, then conclude X causes Y. es V x,e II Y, then conclude Y causes X. Ifei 11. Xx,2 Il Y, conclude neither causes each other. Otherwise, do not decide.
Decision Rule E.3 (Nonlinear ICA Residual-Indeee Residual-Indeppedenne). Check to see if the hypothesis X --+ Y holds and the hypothesis Y --s X holds. If only one hypothesis holds, we conclude that the causal direction. If they both hold, conclude there is no causal relationship. Otherwise, do not decide. Decision Rule E.4 (Adascore Score-matching), Let T1.T2 be the residuals obtained from regressing Y onto X, and X onto Y (respectively).
Let $1,82 be the val- (E Joy logp(X (X, Y) [n]] - Öy logp(X,Y))* and
ues obtained by plugging T1, into E
E [E 0x log Y) r2] 0x logp(X, Y)) respectively. If $1 0, + O. conclude that X- Y. 1f + 0, 82 = O, conclude that - X. Else, do not decide.
Let Lossx-..y, Lossy be the MSE obtained from predicting Y from X, and X from Y respectively. Then if Lossx x--S < Lossy -.X, conclude that X - Lossx-..Y > Lossy-..x. conclude that X Y. Otherwise, do not decide.
Decision Rule E.5 (MSE-Minimization))
x be the ELBOs obtained from training a VAE to predict Y from X, and X from Y respectively. Then if ELBO x..Oxx..Y+ >>> ELBOy conclude that X --+ ELBOx-..Y ELBOY -.x conclude that Y X. Otherwise, do not decide.
Decision Rule E.6 (ELBO-Maximization) Let ELBOx ELBOY -
E.2 Decision Rule Discussion
We note that while the Decision Rules E.2-E.6 are generally representative of each type of ANM- based bivariate methods (Regression Residual- Innituaal--nddeeddenne.eee Score- Matching, MiSi-iiniiization, etc.), there exist subclasses of methods each category. For example, while Decision Rule E,4 reflects the method Adascore [37] (and NoGAM [35] to some extent), the score-matching method SCORE [52] leverages slightly different condition on the score function to recover the causal direction. In our analysis, choose each Decision Rule to reflect the methodology of the most
19

---

general (and typically most recent) method developed in each category. For example, we choose to focus on Adascore over SCORE, as Adascore handles linear, nonlinear, and non-Gaussian ANM, while SCORE requires nonlinear Gaussian ANM.
E.3 Proof that Regression Residual-Indeeendeen Fails (Lemma 3.1)
Innitiaal-ndeeeedenee Fails). Assuming consistent estimator for re- gression residuals and access to infinite data, Decision Rule E.2 fails to identify the correct causal direction when least one mediator nonlinear.
Lemma 3.1 (Regression
Proof. We note that Decision Rule E.2 identifies the causal direction if and only if the residual obtained from regressing Y onto X is independent of X, i.e. 61 IL. X. Suppose for contradiction that
61 IL X. Then, we have that for 6 = g(X) = h(E1, ..... Er-1). where g(X) = E [X] X. Therefore, there at least one nonlinear mediator, then by Lemma 2.3, the ANM-UM underlying X, Y does admit additive decomposition. Instead, Y A₁ (X) A2(Er+1) + As(X, E1,-... ST), where A3 E Cr) contains nonlinear interaction between X and noise terms &. Therefore, e1 M X, and therefore Decision Rule E.2 fails to identify the causal direction.
. e1 _ rewrite Y (XX) h(E1...... (TT+)) However, this leads to contradiction:
E.4 Discussion of Residual-Dependenus Comparisons
We note that despite Decision Rules E.2 being framed in terms of the outcomes of independence tests, most implementations of Residual-Dependenn tests leverage the comparison of test statistic values that measure dependence, rather than strictly comparing the outcome of independence tests. For example, DirectLiNGAN [58] estimates and compares the mutual information, while RESIT [47] estimates and compares the p-value of the HSIC independence test ([10). We analyze the empirical performance of such an approach in Section 5, the baselines leverage these dependence comparisons to boost performance.
(DirectLiNGAM, RESIT)
E.5 Proof that Post-Nonlinear Residual Independence Fails (Lemma 3.2)
Lemma 3.2 (PNL Residual-Independendd Fails). Assuming consistent ICA residual estimator and access to infinite data, Decision Rule E.3 fails to recover the correct causal direction when there exists at least one non-invertible nonlinear mediator.
Proof. As there exists at least one non-invertible nonlinear mediator (i.e., a function ft where t 2 2 and f, non-invertible and nonlinear in the ANM-UM generating Y from X), we note that by Lemma 2.3 we can rewrite Y as Y = ( + A2($1,...,8TT 1) + A(J(()() E1,,...,,T. where Az(-) produces nonlinear interaction between X and and Aa(-) is non-trivial (non-7ero), and non-invertible in (X) E1.
We note that Decision Rule E.3 identifies the causal direction if and only if the causal hypothesis X -- Y holds, and the hypothesis Y X does not hold. We note that the causal hypothesis X -- holds if and only if functions 4, l2 such that for e1 (2 (X),, E1 IL æ. Suppose for contradiction that functions 1,, le such that for e₁ I2(Y)) (X), e1 IL æ. Note that e must be some function of noise terms i.e., = h Ey+1) h(=). Note that 12 must be invertible, as otherwise it would contradict that a proper function X and noise terms E, as there exists an original DGP Y E1,
Suppose 25-1 is linear. Then, we can write y = (X)) + a(e)). However, this contradicts the non-triviality of Ag(-). Then 151 must nonlinear and invertible. However, that contradicts the fact that Aa(-) is non-invertible in /(X) e1. Therefore, there cannot exist functions 4, la such that for e = (Y (X), el II. Therefore Decision Rule E.2 fails to identify the causal direction.
20

---

E.6 Proof Score-Matching Fails (Lemma 3.3)
Lemma 3.3 (Score-Matching Fails). Assuming consistent estimator of the conditional expectation and access infinite data, Decision Rule E.4 fails recover the correct causal direction when there exists af least one nonlinear mediator.
Proof. We note that for Decision Rule E.4 to correctly identify the causal direction, it requires that E (E [oy log p(g,(X Y) r1] - 0y log (XX, Y)) O. Notably, [37] shows (Proposition 4) that this holds and only for Y (x)) + AU) we have A2(U) X. However, as there
least
one nonlinear mediator, then by Lemma 2.3, the ANM-UM underlying X, Y admits the following decomposition Y (X) (2(#T+1) + A3( 61, ,r), where A3(X, Aa(X,,1,.. .1,,..., ST) contains nonlinear interaction between X and noise terms and is non-trive non-trivial. Therefore, by it follows
from Proposition [37] that E (F) [ay logp(X, Y) - dy logp(X X, Y))² * 0, and therefore Decision Rule E.4 fails recover the right causal direction. In fact, [37] explicitly states that their method fails to recover causal relationships when unobserved mediators occur (see Examples 4, 5, 7 [37]).
E.7 Proof that MSE-Minimization Fails (Lemma 3.4)
Lemma 3.4 (MSE-Minimization Fails), Given consistent estimator of the conditional expecta- tion and infinite data, Rule E.5 fails to recover the correct causal direction when E [/Va < E. E|Var[Y] [X]]
Proof. We note that under infinite data the optimal estimator of the MSE
MSB(J) EE((Y= ,(X))²}
converges to the conditional expectation
f'() E|Y x].
This implies that as the sample size n goes to infinity, the MSE converges to the expected conditional variance of Y|X:
Ex [MS((f) = Ex((Y (*)))² XX]
Exx(Y E[Y X)) X Ex (Vr(Y X)]
Therefore, if [V] < E[Var[Y|X]]] this implies that Lossx...y Lossy which implies that Decision Rule E.5 fails to recover the causal direction.
We note that []] < |]) can occur if the R²-sortability favors the anti-causal direction. We note that the coefficient of determination R² simple function of the expected conditional variance, when the variables are standardized:
R (f=) EI(Y f*) = 1 Ex(Var(Y X)]
Var(Y)
(+) =1 E((X - 1+)
= Ey [Var(X Y)]
Var(X)
Reisach et al. [51] show that linear ANM may or may not always be -o-sortability; as linear ANM subset of ANM-UM, this justifies our claim that MSF-Minimizztion methods will fail some ANM-UM.
21

---

E,8 Necessary Constraints for MSE-Minimization methods
We note that the conditions under which MSE-Minimization actually does actually correspond to causal direction identification, i.e.. the conditions that ensure B[Var[X[Y]] > E[Var|Y|XX) have been discussed in other work. For example, [4] show that under the assumption of independence between the function relating cause and effect, the conditional noise distribution, and the distribution of the cause, as well as a close to deterministic causal relation, the errors are smaller in the causal direction. Marx & Vreeken [32] build up on Bloebaum et al. [4], and show that, under the assumption that the best anti-causal model requires at least as many parameters as the causal model (leveraging Kolmogorv's structure function), the regression errors should be smaller in the causal direction. However, these assumptions are quite distinct from our ANM-UM setting, and we leave further investigation to future work.
E.9 Failure Mode of VAE
CANM uses a variational autoencoder (VAE) framework to decide causal direction by picking the direction with the lower ELBO. The training objective used consists of three parts: the log likelihood of x (which does not depend on the model parameters 6, the KL divergence of the latent code, and the reconstruction error [6. Equation 4].
Which of these terms dominates the loss is highly dependent on the training procedure for the VAE, since training VAEs is known to suffer from posterior collapse [11], which we observed during running the method. In Figure 3, we plot decomposition of the training loss of the VAE for CANM, consisting of the KL-divergence term in the latent space and the reconstruction error, for different training regimes. Under the standard training setup (Figure 3a), the KL divergence remains close to zero, while the reconstruction error is relatively high, indicating that the model relies only on X to reconstruct Y, effectively ignoring the latent representation.
Different mitigation strategies for mitigating posterior collapse have been proposed in the literature. Among them, the line following 3-VAE, which introduces a factor in front of the K L term, and uses a scheduling of the 3 part during training [9, 12).
In our experiments, we found that training the VAE with cyclical 3 annealing schedule, following Fu et al. [9]. led to better reconstruction results. The corresponding loss decomposition is shown in Figure 3b. Compared to the constant 3 setting, the reconstruction error is lower, indicating that the model reconstructs the noise effectively. This improvement comes at the cost of a higher KL divergence penalty, suggesting that the latent space is being utilized more meaningfully.
Comparing the two schedules, Figure 3 shows that the dominant loss term varies with the training procedure: with a constant 3, the reconstruction error dominates, while with a cyclical 3, the KL divergence becomes more prominent.
As shown in Table 4, cyclical scheduling failed to improve performance for the invertible cases (tanh and linear) under uniform noise. The algorithm predicts the opposite causal direction with high probability (2 90%). While achieving better reconstruction after improved training, the accuracy in the tanh + Gaussian setting declined using the new training schedule. Both phenomena imply that is exploiting a heuristic signal rather than truly recovering the correct causal orientation.
Method
Linear Unif.
Neural Net Unif.
Quadratic
Tanh
Noise
Gauss.
Gauss.
Unif.
Gauss.
Unif.
CANM
0.10
0.93
0.87
1.00
1.00
0.50
0.10
CANM (constant 3)
0.00
0.97
0.83
1.00
0.90
0.83
0.10
Table 4: Accuracy of CANM variants across different transformation-noiss combinations, with one mediator and n 1000. Bold indicates best.
The original CANM paper proposes selecting the number of latent variables by comparing model likelihoods across different dimensionalities. In our implementation, we instead provide CANM with the ground-truth number of mediators as a hyperparameter, which defines the size of its latent space. In contrast, BiDD does not require knowledge of the number of mediators.
22

---

2.0
6.0
0.2
5.2
Eooch
Dierppacc
Earth
250
Recovsmedtor
200
MCC02M0000 Eror
200
30
100
-100
1200
2200
220
(a) Constant schedule
(b) Cyclical schedule
Figure 3: Comparison between different training schedules for 3 in adapted 3- for the tanh uniform noise case. From top to bottom: B(t), KL divergence, example reconstructions. Mean and 95% bootstrap confidence interval (20 runs) from estimating Ey from Y and X.
E.10 Proof that CANM Fails (Lemma 3.5)
Lemma 3.5 (CANM Fails), Assuming infinite data and a consistent estimator of the conditional expectation, Rule E.6 fails to recover the causal direction posterior collapse occurs and the expected conditional log- likeliihoood minus the entropy is higher in the causal direction.
Proof. We that Decision Rule E.6 identifies the causal direction when posterior collapse occurs and only if that expected conditional log-likelihood minus the entropy is lower in the causal direction. As assume is higher in the causal direction, this implies that Decision Rule E.6 must fail. O
E.11 Proof of Lemma 4.1 Anticausal Direction in Linear ANM
Lemma 4.1. εx
E|sx X Y
23

---

Proof. This proof proceeds through the following steps:
First, we will restate the DGP E,Y, X, and all assumptions.
2. We define the regression function h(X,Y) Y) = Blax|X; YJ. This proof will proceed in the following steps.
(a) We will characterize h(X, Y), splitting it up into univariate functions and functions with interaction terms.
(b) We will outline all possible cases (and subcases) for how the functional form of h(X, Y) might look.
(c) For each case (and associated subcases) we will show that 1(X Y) is a function of noise term dependent Y.
(d) We conclude that, as h(X,, Y) is always non-trivial function of noise that is dependent on Y, we have that ((X, Y) J Y.
Step 1: Restate DGP and Assumptions
Note, X, Y follow
Y X 11.
(E.1)
We Assumption 2.1 and 2.2); note that 2.2) stipulates that E is non Gau-Saas,,,,, for identifiability. We inject independent Gaussian noise into both X, obtaining the noised term X:
X = X +xx ex 1).
(E.2)
Additionally, we require that E1, ex are random variables with sverywhere-positiee absolutely continuous and differentiable densities fx, Ser, and fex.
Step 2: Structure of h(X,
Note that h(X,Y can be decomposed as h(X,Y) = (x) + A2(Y) + Aa(X, Y). where As contains only (linear or nonlinear) interaction between X and Y, while A1, A2 are univariate.
Step 3:
General Cases
We note that the functional form h(X, Y) can be divided into four general cases: 1) Ag is non- trivial, 2) A3 and Ag are trivial A₁ is non-trivial, 3) Ag and A₁ are trivial, Ag is non-trivial, and 4) Ag is trivial while A₁ and A2 are non-trivial
Case 1): Suppose Ag is non-trivial. This implies that there exist interaction terms between X, Y. Suppose for contradiction that h(X,, Y) is not dependent implies that A, A₂ somehow cancel out all Y terms A3- This leads contradiction, since the space of additive functions-i.e.. those expressible as a linear combination of univariate functions of each variable-canno represent interaction terms, which require andadiitive combinations such as products of variables. Therefore, h( h(X,y Y) is non-trivial function of Y. making a non-trin non-trivial function of noise term ε1 M Y.
noise term dependent on Y. Then, it
Case 2): Suppose Ag, A2 are trivial, and A, is non-trivial. Then, for some function h(X,Y) = g(X) = g(X + +x). This implies that f(X,, Y) is non-trivial function of X and noise jointly independent of X and Y. As Y contains X noise term (Y X s1), this implies that Y Y YY.
Case 3): Suppose Ag, A₁ trivial, and Ag is non-trivial. This directly implies that h(X, Y) is non-trivial function of Y. Therefore, Y J. 1(X
Case 4): Suppose Ag is trivial while A₁ and A are non-trivial. We analyze what happens here in
Step 4, further splitting Case 4 into subcases based on the linearitynoolineariyy of A1,

---

Step 4:
Analyzing Case 4
We split Case 4 into the following three subcases: A) A is nonlinear, B) A₂ is nonlinear, C) both A1, Ag are linear.
Subcase A): Suppose (X) is nonlinear. Then, as X = X A1(X) will contain interaction term between Ex Note as A3 is trivial, the interaction term between X, Ex cannot be cancelled out - therefore, h(X, Y) must a non-trivial function of X. As Y = X + S1, this implies that Y Y YY.
Subcase B): Suppose A2(Y) is nonlinear. Then, Y X + 31, A2(Y) will contain an interaction term between 61. Note as A3 trivial, the interaction term between Y, cannot be cancelled out therefore, h(X, Y) must be non-trivial function of ε and Y. As Y X + E1, this implies that Y V. 1(X,)) Y).
Subcase C): Suppose A and A2 are both linear, i.e.
(x aX + aex A²(Y) 3Y.
(E.3)
(E.4)
Suppose for contradiction that 1(XX, Y) 11 Y. Then, we have
h(X, Y) = (X) + A2(Y) M(X, Y) =xx + asx +3..
(E.5)
(E.6)
X 1(X, Y) - BY - aEX
(E.7)
Q
X B
h(X,Y) Y) OEx
(E.8)
a
Note thath(X, Y) T1 Y.Y Y. IL EX. Additionally, when - is added to A(X.Y) cannot cancel out
any term dependent on Y. Therefore, the sum must be independent of Y, i.e. Y IL -(XxY)-oex However, this contradicts assumption in Step I that there does not exist backwards ANM model X Y + where Y IL Thus, h(X, Y) Y.
Step 5: Conclusion
We have shown that h(X, Y) is a non-trivial function of Y for all possible forms that h(X, Y) can take. Therefore, we conclude that B[Ex [X, Y Y.
E.12 Proof of Diffusion Correctness for ANM
Theorem 4.2 (Consistency Decision Rule 1). Suppose X, follow Eq. (2.1), Assumptions 2.1 and 2.2 hold, and no nonlinear mediator exists. Then, given consistent mutual information estimator and infinite data, Decision Rule I correctly recovers the causal direction between X, Y.
Proof. We note that if no nonlinear mediator exists, then by Lemma 2.3 X, Y can be represented by standard ANM, where
f(X) + X IL E1 and Assumptions 2.1, 2.2 still hold. We additionally require that X, 1 are random variables with weryyhher-posstiive. absolutely continuous densities fx, Jey and Jex- We further assume that f is continuous and three-times differentiable. Note again, that we inject independent Gaussian noise into both X and Y, obtaining the noised terms
X = X Ex and Y = Y EY, Ex, <X, 1).
(E.9)
We note that, under infinite data, Decision Rule I correctly recovers the causal direction and only if both of the following statements hold(written equivalently in terms of mutual information and independence):
MI(Ey, x)= - Elay|: I. V, X I1 X
=
(E.
MT(xx, .)> - E|Ex| X, Y J Y
>0
(E.11)
25

---

We will first show the causal direction (Eq E.10 holds), then the anticausal direction (Eq E.11 holds).
E.12.1 Causal Direction
We will find a statistic such that E Y, x] E(ey/7) and is jointly independent of and Ey. Then, we will conclude that Eq E. 10 holds.
Note, the joint density 98, N,Sy can be written as
x 2. (x0=)))((( c) (c)
(E.12)
using the change of variables
Y= f(x + +EY,
(E.13)
as X, Ey are mutually independent.
Let T(Ý. x) = = f(X). Then, we have that
gY Xisv (1, x fx(x) (T(Y, x) e) =j(z ((),)))
(E.14)
(E.15)
for some functions j, k. Note that this means that, when conditioned on a constant T(Ý, X), the joint distribution 99 X,ey can be factorized into the product of two distributions, one involving X, Y, and the other involving , T(Y,))
follows that X(Y"X) renders By conditionally independent of Y, X:
11 Y, T X).
(E.16)
Now, this implies that T(Y, X) is sufficient for estimating the conditional expectation:
B E
Y, X = E[sy Y T(V x)] Elsy (T(,))) x)).
(E.17)
Finally, note that
x)Ý = - -(X) = + f(X) + E1 f(X)
(E.18)
(E.19)
= =s + E1.
(E.20)
= E|sy Y. X| Eisyley + =1].
(E.21)
As X is jointly independent both Ey by assumption, it follows that E|sy|T((, 11 X. Therefore, we conclude that |E|sy|Y y, X] X, and thus Eq E.10 holds.
E.12.2 Anticausal Direction
f is linear, then by Lemma 4.1 have ElEx X Y. Suppose f is nonlinear.
Let h(X, Y) Elex | YJ. This proof will proceed in the following steps.
1. We will characterize ((X, Y), showing that it must be non-trivial function of Y and X.
2. We will outline all possible cases (and subcases), in which h(X, Y) non-trivial function of Y.
3. For each case (and associated subcases) we will show that h(X Y) is a function of a noise term dependent on
4. We conclude that, h(X,Y) is always a function of noise dependent on Y, h(X, Y) 1 Y.
26

---

Note that h(X, can be decomposed as h(X,Y) = A1( + A2(Y) + As(X, where As contains only (linear or nonlinear) interaction between X and Y, while A1, A2 are univariate.
In the graphical model (see Figure 4), there is an active path Y + Ex when conditioning on the collider X. Due to d-separation rules [61], follows that x J YJX. Note that this implies that, under regularity conditions assumed above, the conditional distribution P C)C) # P(ex(C) on set of positive measure. This implies that B[Ex [X,, Y] + [(ex|X] and therefore h(X, Y) is non-trivial function of Y. Therefore, at least one of A2, Ag is non-trivial. Similarly, as Ex J. X|Y, least one of A1, A3 must be non-trivial.
We will now walk through the following 3 cases: 1) that A3 is non-trivial, 2) that A3 is trivial and A2 non-trivial and f invertible, and 3) that A3 is trivial and A2 non-trivial and f non-invertible. In each have 4 subcases.
will show that h(X, Y) is function of noise term dependent on Y. Case 2 will
Suppose A3 is non-trivial. This implies that there exist interaction terms between X, Y. Suppose for contradiction that &xX,Y' Y) is not dependent that A, somehow cancel out all terms in A3. This leads to a contradiction, since the space of additive functions-icc those expressible as linear combination of univariate functions of each cariable-cannot represent interaction terms, which require non-additive combinations such as products variables. Therefore, h(X,Y) Y) is a non-trivial function of Y, making a non non-trivial function of noise term E1 Y.
noise term dependent on Y. Then, it implies
Suppose Ag is trivial. Then, A₁ and A₂ must be non-trivial.
Suppose f is invertible. There are possible subcases. In each case, is modelled by a different function of Y, and different noise. We list each case explicitly:
1. fi(Y) + +e,, 11 ee
2. X fe(Y, e2), 1L
3. X Ja(Y, e3), Y e/
4. X = fA(Y) + E4, Y e4
We note that, in Case 2 and 3 functions f2 and fa induce nonlinear interactions between their inputs Y, e2 and ez.
Note that Case 1 cannot occur, it violates Assumption 2.2 by allowing for the existence of backwards model X + Y with additive noise.
Let's assume Case 2. Then,
h(X,Y) = Ai(fa(Y, + +xx) A2(Y).
As induces nonlinear interaction between Y and 62, where 62 1l Y, e2 11 Ex, the collection of terms in A₁ containing Y, cannot be equal to a univariate function of Y. Therefore, the residual r
A (1 e2) + +xx) A2(Y must be dependent on both Y and e2.
Let's assume Case 3. Then
M(XY) = Ai(Ja(Y +6x) A2(Y).
Note that as induces nonlinear interaction between Y and Eg, needs to be function of if it would be only function of Y, that would imply a deterministic relationship between X and Y. which contradicts our setup. Due to the nonlinear dependence between Y and e3 in fa. the eg term can not be canceled out by the univariate function A²(Y). Therefore, h(X, Y) must contain the noise term c3, C3 $. Y.
Let's s assume Case 4. Then
h(X,Y) = Ai(f((Y) + +e A²(Y)
(E.22)
Similar to the argument in Case 3, although J. Y, cannot solely be function of - would again imply deterministic relationship between X, Y. Therefore, es must also function of e Then, the residual noise A)(f((Y) + + A²(Y) # 0, and r r Y.
27

---

EX
X
Y
Figure 4: Dependence in the Anticausal Direction: X X Ex and Y f(X) 51. All root nodes are drawn from independent noise.
Therefore, h(X,Y) is always a function noise term dependent on Y.
Suppose is non-invertible. Then there exists no backwards model where X can be written as function of and noise. Therefore, A₁ cannot written as function of Y, noise and &x. Then, the residual r = A1 + A2 must contain Y, which means that (X, Y) contains Y (which contains e1).
We have shown, either that always contains E1. that h contains different term dependent on Y. Taken together our results imply that Y M h h(XY)) [[ X, Y, Y.
E.13 Challenges in Extending Theorem 4.2
The major challenge in extending Theorem 4.2 to the irreducible case, i.e., when ANM-UM does not reduce to ANM due to the presence of a nonlinear mediator, lies in the increased difficulty in analyzing the properties of the estimator E[Ey Y. x]. This difficulty arises when X becomes non-additively related to the latent noise as result of nonlinear mediators.
Note, procedure rests the intuition that noise predictions from conditional diffusion model will be less dependent on the condition when the condition is the cause, rather than the effect.
When the ANM-UM reduces to ANM, we can actually show that the noise prediction is entirely independent of the condition when the condition the cause. We can write as (potentially nonlinear) function of X plus noise, i.e. Y f(X) + E1. This clean additive separation between the noise and f(X) allows to show that the predicted noise é Ejey IÝ X] B[sy|y, X,
f(X)] = E[sy| |Ý, Xy EI] Eley lsy + @1). This reformulation of é into a term that is clearly independent of X is the backbone of our proof of Theorem 4.2.
However, when the ANM-UM does not reduce to ANM, i.e. because of nonlinear mediator, Y can only be written as a function of X and noise (Lemma 2.3): Y = f(X) + g(E) + h(X, E). where X, E are nonlinearly combined h. Here, its not possible to cleanly project X out of Y, i.e., there does not exist function k(-) such that &(Ý, x))
) is independent of X. Therefore, to extend Theorem 4.2 to this setting, we would need to show that EE[yy XX has a smaller dependence on X, when compared to the dependence between E[Ex [X, Y] and Y. Analyzing the strength of the dependence would probably require very different set of proof techniques, which are beyond the scope of this paper. One potential first step to bridge the gap between the reducible and irreducible settings is to identify conditions under which the interaction term h(X, (,&) becomes sufficiently 'weak', such that E[ey|Y Y,, X]] becomes 'close to independent'. This will probably involve some investigation into the interaction by functional forms, noise distributions and the resulting mutual information between X and the interaction term e(X,e).
28

---

Details on Bivariate Denoising Diffusion (BiDD)
F1
Adapting Diffusion for Causal Discovery
BiDD works by training diffusion model in both direction and selecting the direction with the lower noise prediction. We will now describe the details of training the diffusion model, estimating the mutual information, and deciding on causal direction. We describe the full procedure of BiDD in Algorithm 1. We formalized all three components subroutines 2 and 3.
Algorithm BiDD: Causal-direction discovery via conditional diffusion
Require:
Training set Drrain Test set Drest, epochs E, timesteps T, mutual information estimator MI, noise schedule {āt (r(mT]
Ensure: Causal direction A + B or B + A
EA <- TRAINCONDIIIONA DIFFUSION E,T. T, {othe)))
2: (etA < ESTIMAYYEMUTTAAALNFFORAAAATOOTOO rest. E, T, MI, (arhetT))EAIp)
(eest
3: "BJA <- "BANNCODTTOMMIDDIEESSOSSSSSSSDSD E, E, {ūx he(r))
4: (MTz,shEt)) - ESTINAATEMMITTAAAAAARRRRAATTTTTT
F, MI, (a))a(())
5: return COMPAREENTTUULLINOORRTTTTOMOACCC (Mla. {MIB.)
F1111 Training Conditional Diffusion Model
Subroutine adapts the denoising-diffine diffusion training loop of Ho et al. [15, Alg. to our conditional setting. In our training, we perform single, full-dataset update per epoch.
Subroutine Train diffusion model
Require: Training set Dirain {(A() [e))) epochs E, total timesteps noise schedule
(e
Ensure: Trained model Eæ
for ., E do
2: Independently sample timesteps {(0) ~ Uff....... T3 for all i ,,...,n
3: Draw noises e(t) ~ NO0 1) for all
4: Construct noised inputs
1 Vanc A + vI Cxt (1)) for 1,...,
5: Compute loss
1 n
L=
>
((0) -Ed(A B(0, {(0) 1|
6: Take gradient step on 0
7:
end for
8: return
F11.2 Mutual Information Estimation
After training the diffusion model, we use Subroutine 2 to estimate the mutual information between the condition and the predicted noise.
Note that in order to improve the finite sample performance of the mutual information estimators, the test set input to Subroutine 2 can be oversampled. This will lead to datapoints with identical conditions B, but differently noised version A. We oversampling factor & 10 in experiments.
E.1.3 Deciding causal direction
After obtaining ( (MTr.)) (denoising À) and B, (denoising B) for both directions, we decide the direction of the causal dependence.
29

---

Subroutine Estimate mutual information
Require: Test set D - {A(0) ( Neeu) epochs E, timesteps T. mutual information estimator MI, noise schedule frr) trained diffusion model Eg
Ensure: Series of MI estimations {MIr)e for = . do
Draw random noise e (0, 1) for i.
Generate A from test set as At A + vI- -1-0; a
Predict e = (A() (000,(6))
Calculate MI. MI((EX 1 ( {B() ドー)
end for return (MT)
Voting rule For the voting rule, we conclude B - if and only MTA. < MTB. B for the majority of timesteps. We use this rule in the main body of the paper. We formalize it in Subroutine 3.
Subroutine 3 Compare mutual information
Require: MI sequences (MI and (eTT))
Ensure: Chosen causal direction (A + B or B + A)
I: e + Σ {(MIA < MIB,+}
2: i T/2 then
3: return B + Ak
4: else
5:
return A + B
6: end if
Mean rule the mean rule, we conclude > if and only + MI < + E MlR,, We present results on this decision rule in Appendix H.
F.2
Implementation details
Models were implemented in PyTorch[45). Training employed the AdamW optimizer. We used a Hilbert-Schmidt Independence Criterion (HSIC) implementation in PyTorch. For the alternative mutual-information estimator evaluated in Appendix H, we used the NPEET package.
F2211 Mutual-information estimation
Dependence between the predicted noise and the conditioning variable quantified with HSIC [10], which we treat surrogate for mutual information. HSIC is computed with Gaussian kernel whose bandwidth is selected via the median pairwise-distance heuristic. The robustness of our results to the choice of dependence measure is examined in Appendix H.
F.2.2 Training hyperparameters
In our diffusion setup, we use T 256 timesteps, scheduling 3 linearly from 8min 0.0001 to Amax 0.02. For stochastic gradient descent, we use the AdamW optimizer with cosine annealing. We set initial learning rate of 0.0001 and decay to 0.00001. We train our model for total of 4000 epochs. For BisDTestt we used 80% of the data for training and evaluated performance on the remaining 20%. For BiDDTotal. trained 80% of the data and evaluated on the entire dataset.
F.2.3 Model Architecture
The diffusion model for BiDD uses an MLP architecture to predict cAA, B,t). Each of the three inputs is first fed through dedicated input projection: a 1 - 512 linear layer for A,, small conditioning network for B, and sinusoidal time embedding followed by two SiL.U-activated linear layers for The resulting (512 + + 512) 1028-d feature vector is concatenated and processed
30

---

Stage
Operation / Activation
Output shape
Input projections
Åg proj.
Linear (1~+512)
[B,512
proj.
Linear - (1->16))Rell ReLU
[, 16)
Linear (06-+32)--0 (16-33))
(B,32)
Linear (32 (3)++4)
(B,4)
embedding
Sinusoidal (1 (1-+10
(B,16)
Linear (16 + +
(B,512)
Linear (512 ++12)
(B,512)
CONCAT
(B, 1028)
Residual MLP blocks (repeat twice)
Hidden
Linear (1028 + 2056) + SiLU
(B, 2056)
Linear (2056 + 1028)
(B, 1028)
Residual add
I < block(
(B, 1028)
Output projection Output proj.
Linear (1028 + SILU Linear (512+1))
(B,512)
(B,1)
Table 5:
Layer specification for the BiDD denoising model predicting e(Ae, B, t). B is batch size. SiLU is the Sigmoid-weighted Linear Unit, ReLU is the Rectified Linear Unit.
by two residual MLP blocks, each expanding to two times width and returning to the original size. A final projection (1028 +512 ++12-IJ produces the noise estimate Eg (A,, B, t). We document the exact layer setup in Table 5. Overall, the model contains 9, 2260,893 learnable parameters.
G Experimental details
G.1 Evaluation Data
G.1.1 Synthetic Data
To generate the synthetic data, used different link functions in combination with different noise types.
Details for the link functions
Quadratic:
f(x) = (x) + E,
Tanh:
f(x) = tanb(x o) + e.
0~ U(-1,1),
Linear:
f(x) az + +
a a~ u((55,,5) 0 (((3,3).
Neural network: f(z) = tanb( x w + 1b))) Wout +E,
where the weight vectors lie & and are sampled i.i.d. from
Wins b Wont 4(-5,5).
The parameters o, a, Win- bys Wout are randomly drawn for each mediator and run.
Noise types We evaluated BiDD two different noise types; uniform and Gaussian. For noise inside the mediators (e, till en Figure 1), we used noise with mean 0 and variance .5. For noise generating X and Y (Eu and e141 in Figure I), we used noise with mean 0 and variance 1.
Data generating process We used the same noise type for generating the cause and noise in the process for our experiments to generate X and Y. For each mediator, redrew the random
31

---

parameters for the link functions. More specifically, to synthesise single cause- cause-effect observation (X with T' latent mediators, we begin by drawing the cause X D 1). where D(u, o²) denotes the chosen base noise family (e.g- .N L)) Setting 7 = X, we traverse chain of T unobserved mediators. For each index j ...... 7 we independently (i) sample noise term Ej D(0, 0.5) and (ii) select link function 9j at random. The mediator is then evaluated as
Zj (z(())1)) +5g.
After the final mediator we draw 8T41 ( 1) and an additional link function fr+1. generating the effect variable by
Y = / 8T+1-
Finally, both X and Y are centered and rescaled to unit variance. Repeating this procedure indepen- dently yields i..dd dataset that follows ANM-UM with T' unobserved mediators.
G.1.2 Real-world Data
We evaluated the performance on the first 99 pairs of the Tübingen dataset [42], loaded from the package. For each pair, we randomly subsampled 3000 data points each run for faster execution. calculated the accuracy as the simple average over pairs.
G.2 Implementation Details
We coded our experiments using python 3. 11. 11 with PyTorch 2.5.1 [45], and ran the experi- ments AWS g4dn. xlarge ec2 instances. We provide public repo https://g// //gttttt..b/../ com/ commeier/biin
G.3 Baseline Implementation
We imported CANM from the code provided in Cai et al. [6]. We did not use the described method find the correct number of mediators, but instead called the method with the correct number of mediators. We trained the VAE for 2000 epochs.
DirectLiNGAM and RESIT were imported from the lingam package. CAM, SCORE and NoGAM were imported from the dodi scdier package. CAM-UV was imported from the lingam package. Adascore was imported using the causal-score score secore-matching package. Var-Sort was implemented using NumPy. DagmaL was imported from the dagma package. PNL was implemented following the logic in the causal-less library. but with slight modifications in order to execute model training on GPU for faster execution.
We list the hyperparameters used in Table 6.
Method
Hyperparameters
DirectLiNGAM
None
CAM
prune=False
RESIT
RandommorrsstRggressoo (max
(mpx
SCORE
prune=False
NoGAM
n_crossval=2, prume-Fllee
var_sort
None
entropy_kee
k=100, base=2
PNL
None
AdaScore
alpha_orientaaiionn.. 05 alphaa_confounddd_eeee alpha_soparations-00..0
DagmaL
lambda1=0. 05, T=4, mu mu_ini
s=[1, 0.9, 0.8, 0.7], mu mu_factor=0.11
Table 6: Hyperparameters used for each method.
32

---

G.4 Runtime
We provide the mean runtime for 80 independent runs across all mechanisms (linear, tanh, neural network, quadratic) and noises (Gaussian, uniform), measured wall-clock time. The runtime experiments were conducted on AWS g4dn. xlarge ec2 instances with GPU support, with no parallelization.
All methods that do not rely on soohastticggrdiien training finish in under seconds. Among the baselines, PNL is an order of magnitude slower because it trains small neural network.
The scenario with unobserved mediators is inherently more complex than fully observed settings tackled by the baselines, which justifies longer runtime. When compared with the only other method that explicitly addresses unobserved mediation, CANM, BiDD has comparable runtimes.
For PNI., BiDD, and CANM the wall-clock time is governed mainly by (i) the size of the training set, (ii) the complexity of the model, and (iii) the number of training epochs. We didn't t exhaustively tune model size, epoch count, or code efficiency, so runtimes could be reduced with further optimisation.
Method
Runtime (s)
BiDD
172.0
CANM
145.5
Adascore
1.25
NoGAM
0.23
SCORE
0.28
DagmaL
0.85
CAM
0.15
PNL
35.80
RESIT
0.43
DLiNGAM
0.01
Var-Sort
0.00
Table 7: Average runtimes of methods for n 1000 seconds.
G.5 Asset information
external code we import open-source under permissive licences: lingam, dodis acoisr causal-leamd and NPEET are MIT;
is MIT-0; dagma is Apache-2.0.
We use the Tübingen causc-effect dataset curated by Mooij et al. [40]. Several of its variable pairs originate from datasets released by Kelly et al. [24] in the UCI Machine Learning Repository.
H Ablations
H.1
Mutual information estimate
H.1.1
Setup
To test the sensitivy of BiDD to the choice of mutual information estimator, we test it's performance under two different mutual information estimators. We test three different sets of hyperparameters for each of them.
For HSIC, in our main experiments, we heuristically pick the width of the Gaussian kernel as the median in the distance metric. As an ablation, we scale this width by . and 2.
As alternative estimator, use the NPEET package, which implements the nor-maraettric estimator by Kraskov et al. [26], using k-nearest neighbors approach. We vary the number of neighbors & = 3, 5, 10.
We use the same experimental setup as in our previous synthetic experiments, setting 1000, and using a test split of 20%.
33

---

Method
Linear Unif.
Neural Net
Quadratic
Tanh
Average
Noise
Gauss.
Unif.
Gauss.
Unif.
Gauss.
Unif.
Full data, voting
HSIC(1)
0.80
0.95
0.90
1.00
1.00
0.85
0.80
0.85
HSIC((5)
0.85
0.95
0.85
1.00
0.80
0.90
0.80
0.83
HISIC(2)
0.80
0.90
0.90
1.00
1.00
0.85
0,80
0.86
N3EET(3)
0.85
0.95
0.85
1.00
0.70
0.80
0.95
0.81
NPEET(()
0.90
0.95
0.85
1.00
0.70
0.75
0.95
0.81
NPEET(10)
0.90
0.90
0.85
1.00
0.60
0.80
0.95
0.82
Full data, mean
HSIC(1)
0.80
0.95
0.95
1.00
1.00
0.80
0.70
0.84
HSIC(55)
0.85
0.90
0.90
0.80
0.75
0.90
0.70
0.79
HSIC(2)
0.70
0.95
1.00
1.00
1.00
0.70
0.65
0.83
N3EET(3)
0.90
0.85
1.00
0.65
0.70
0.95
0.81
NPEET(S)
0.90
0.95
0.85
1.00
0.55
0.75
0.95
0.81
NPEET(IO)
0.85
0.95
0.80
1.00
0.45
0.75
0.95
0.79
Test data, voting
HISIC(I)
0.80
0.85
0.80
1.00
0.95
0.75
0.70
0.79
HSIC((.))
0.75
0.90
0.85
1.00
0.75
0.75
0.75
0.76
HSIC(2)
0.65
0.90
0.90
1.00
1.00
0.80
0.65
0.80
NPEET(3)
0.75
0.85
0.80
1.00
0.70
0.70
0.90
0.79
NSEET(()
0.70
0.85
0.85
1.00
0.80
0.60
0.90
0.78
NPEET(10)
0.75
0.90
0.85
1.00
0.85
0.70
0.90
0.81
Test data, mean
HSIC(1)
0.75
0.85
0.90
1.00
0.95
0.65
0.65
0.79
HSIC(.5)
0.75
0.90
0.95
0.60
0.70
0.77
HSIC(2)
0.75
0.85
1.00
1.00
1.00
0.60
0.65
0.81
N3RET(3)
0.75
0.80
0.85
1.00
0.45
0.75
0.75
0.74
NPEET(5)
0.80
0.80
0.80
1.00
0.40
0.70
0.75
0.73
NPEET(10)
0.80
0.95
0.90
1.00
0.55
0.70
0.95
0.80
Table Accuracy various MI estimators across transform noise combinations. Best scores per column are in bold, second-best are underlined. Accuracy is averaged over 20 runs per mechanism. HSIC(s): Factor s applied to kernel width. NPEET(K): Number of neighbors
For every run we report two accuracy criteria: (i) voting, which counts a direction correct if the majority of timesteps agree; (ii) mean, which averages the MI over all timesteps and picks the lower-dependence direction (see Section F.1.3).
H.1.2 Results
Complete results appear Table 8.
Overall robustness. Every configuration achieves at least 76% mean accuracy, rising 79% or better when the full data is used for estimating mutual information.
Retimatto-meecaniism interactions. HSIC dominates on the quadratic + uniform-noise mechanism, whereas NPEET best on tanh + uniform. These preferences are consistent across both voting and mean rules.
Voting vs. mean. Discrepancies between the two decision rules widen the test split. For example, NPEET with 3 misses the quadratic + uniform case under the mean rule but is working good under voting. Conversely, HSIC occasionally loses 5% 10% on tanh normal when switching from voting to mean.
Hyperparameters for estimators. Within each estimator family, hoper-parameter choices have second-order impact: HSIC's wider bandwidth (x2) and NPEET's larger neighbourhoods (k= 5 or 10) bring modest, but consistent, improvements.
34

---

Method Noise
Linear Unif.
Neural Net
Quadratic
Tanh
Gauss.
Unif.
Gauss.
Unif.
Gauss.
Unif.
With conditioning:
BiDDrotal Total
0.83
0.87
0.97
1.00
1.00
0.80
0.83
BiDD Test
0.80
0.87
1.00
1.00
1.00
0.63
0.77
No conditioning:
BiDD Total
0.15
0.90
0.75
0.30
0.35
0.60
0.05
BiDD Test
0.15
0.85
0.80
0.05
0.55
0.40
0.10
Table 9: Conditioning is necessary for identification across different mechanisms: Results for same setup as Table 2, but with modified training objective without conditioning. report the mean accuracy over 20 runs.
H.2
Conditioning vs Non-Conditioning
H.2.1
Setup
To test how conditioning impacts the performance of our methods, we provide an ablation study where we train an unconditional loss. That means that the diffusion model does have access the conditioning variable B anymore, and needs to predict the noise only from A and t. We replace the original loss function:
LCDm E.,.,.. [|e s0 (Ar, B.0)]] t Unit((L,...., ,T)
with the unconditional one:
Lom EAa, |ll- s0 =o(AA, to Unif((1......
and keep the setup and training identical.
H.2.2 Results
Conditioning is important Table 9 shows the importance of conditioning. While accuracy in the neural-network setting remains similar, deteriorates across all other mechanisms, and the model often selects wrong directions for both linear uniform and tanh + uniform. These results indicate that conditioning the diffusion model is an important part for reliable causal discovery in our framework.
H.3 Two mediators
Method Noise
Linear Unif.
Neural Net
Quadratic
Tanh
Gauss.
Unif.
Gauss.
Unif.
Gauss.
Unif.
BiDDTootll
0.65
0.80
0.95
0.70
0.30
0.45
0.85
BiDD Test
0.70
0.75
0.95
0.70
0.60
0.45
0.85
RESIT
0.90
0.50
0.60
0.00
0.00
0.60
0.90
PNL
0.55
0.55
0.55
0.75
0.40
0.30
0.50
Table 10: Accuracy of methods across different transformation-noiss combinations, with two media- fors and 1=1000.
Results for the quadratic link function with high number of mediators tend to become unreliable due to numerical instability from repeated quadratic operations, reported in previous work [13]
'Results for the tanh link function and Gaussian noise can become unreliable if sample size not high enough, because the Tanh function is close to linear across -1 to 1. where most of the mass of a Gaussian distribution is, rendering the empirical distribution close the unidentifiable linear Gaussian case.
35

---

1.0
0.8
0.6
Accuracy
0.4
BiDD
0.2
BiDD
PNL
RESIT
0.0
1
2
3
4
Number of Mediators
Figure 5: Accuracy for Neural Network + Gaussian setting with varying number of mediators
H.3.1 Setup
All baselines, except PNL and RESIT, fail in at least two settings. achieving accuracy of at most 50%. This renders them unreliable in scenarios where only the ANM-UM assumption holds. Of the three remaining methods, all use residual independence as the decision criterion, their difference in performance probably depends on how much the underlying model violates ANM or PNL. While they might perform well in settings with low number of mediators, they might fail with an increased number of mediators.
To analyze how RESIT and PNL perform when their assumptions are violated, we re-ran the synthetic experiments using the same setup in Table 2, but with two mediators.
H.3.2 Results
While the average performance of all methods declines when increasing the number of mediators from one to two mediators (e.g.. from 0.90 0.67 for BiDDrocal. 0.87 0.71 for BiDD 0.76 to 0.51 for PNL. and 0.88 to 0.50 for RESIT), the drop is particularly pronounced for PNL and RESIT in settings with neural net link functions: Both BiDD variants consistently achieve least 75% accuracy across noise types, whereas the best bascline score does not exceed 60%.
We explain this difference in performance with DGP which is not following the assumptions from PNL or RESIT anymore, while still following the ANM-UM, which BiDD can exploit.
H.4 More mediators in non-invertible Gaussian settings
H.4.1
Setup
We extend the experiments for Figure 2a to the Neural Network + Normal setting, to investigate how a non-invertible mechanism and noise combination behaves under varying number of mediators.
H.4.2 Results
We report the accuracy of 20 runs across different methods in Figure 5. For mediator, all methods perform well. When increasing the number of mediators, both PNL and RESIT exhibit sharp decline in performance for one mediator, and keep declining for more mediators. Both BiDD versions are more stable and decline slower.
36

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: We address all claims made in the abstract and contributions section throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly state the assumptions we make throughout the paper, and state the settings in which our theoretical results hold.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
37

---

Justification:
Each theoretical results includes the necessary assumptions, and links to full proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We thorougly describe experimental procedures in Section 5 and give additional details in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
38

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Answer: [Yes]
Justification: We describe experimental setup Appendix G and provide the code for the experiments on Github (https github. com/
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips cc/public/ gww.d/gg/////eeeee//ooo/co /ooeeSumma ss wwwionoolllli onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5, Appendix G and code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We report the sample size and number of runs we use for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
39

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the computational resources we used in Appendix G.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : /ttp:////eups./eeeeeeeeeeereeeee cttpiipp..ppppppeeeeee cc/public/Etiiiie httpo//iccehiiccccudeeeeeeeeeee
Answer: [Yes]
Justification:
We followed all parts of the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper addresses a foundational problem in the field of causal discovery, which is not fundamentally tied to any specific set of applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out
40

---

that a generic algorithm for optimizing neural networks could enable people to train
models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Appendix G.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets introduced.
41

---

Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The core method development does not include LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
42

---

Please refer to our LLM policy hattps:/hhuurips.c//// /c//////oooooooooo/22////1
for what should or should be described.
43


Reviews:
[{"review_id": "LJUYNTn9SR", "reviewer": "Reviewer_UBVq", "summary": "The proposed method in this paper offers a causal direction identification method while addressing the hidden mediator case. The authors show precisely where existing work does not perform well and why. The proposed method uses denoising diffusion to obtain a new conditional independence test.  Finally, they demonstrated empirical performance on both synthetic and real-world datasets.", "questions": "Below I share my questions.\n\n* Why can we not consider all $Z_i$ as a single variable $Z$ and then apply existing methods?\n* The authors mentioned (line 202) that “inclusion may also introduce dependence between the predicted noise and the unnoised variable”. Why does this happen?\n* Line 271: Is it possible that the inferred direction keeps flipping at different timestamps $t$? What does this behavior indicate? Does the method eventually converge to a specific direction?\n* (Line 297) For which setup was the causal direction the hardest to infer?\n* How many mediators can the proposed method accommodate before performance degrades?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Below I share my comments.\n\n### Strength:\n* The paper is written in a clear and well-organized manner. The proposed method appears to be theoretically sound.\n* I appreciate the authors' discussion on the failure modes of prior works in different scenarios. The analysis of failure cases is helpful for understanding the motivation and contributions of the proposed method.\n\n### Weakness:\n* Line 198: Typo. I think it should be “noising/forward process”.\n* Line 200: In $(\\epsilon_Y - f_{\\epsilon_Y} (\\tilde{Y}, X))^2$, why is it $X$ and not $\\tilde{X}$, given that noises were added to both $\\tilde{X}$ and $\\tilde{Y}$? Are the noising processes in Eq $4.1$ and $4.2$ applied to $X$ and $Y$ at the same time, or are they applied individually while keeping the other variable fixed? The authors should clarify this point.\n* The authors mentioned that they choose the direction with lower dependence (mutual information). Shouldn’t there be no dependence in the anti-causal direction? What does it mean when both directions have high dependence (but unequal), or both have low dependence (but unequal)? What does the gap between the two dependence values imply? These questions should be addressed in the paper.", "was_revised": true, "final_justification": "The authors have addressed my concerns and added additional experiments. Thus, I keep my positive score.", "rebuttal": "We thank the reviewer for highlighting several strengths of our submission, including the discussion of failure modes in existing work described as “helpful for understanding the motivation and contributions of the proposed method”, as well as the “clear and well-organized” presentation and the “sound” methodology.\n\nWe also appreciate the reviewer’s thoughtful questions, which have helped improve both the clarity and depth of our work. Below, we address each of the points raised in detail.\n\n\n**Weaknesses**\n\n**(W1)** *Typo*\n\nWe thank the reviewer for pointing out the typo. We address it with our update to the paragraph in (W2).\n\n\n\n\n**(W2)** *Noising process*\n\nWe formulate two noising/denoising processes. One for each direction $X \\rightarrow Y$ and $Y \\rightarrow X$.\nIn each of the processes, we keep one variable fixed, while the other is noised. To clarify this setup, we will update the paragraph in Section 4.2. containing line 200:\n\n> To show the asymmetry in this setup, we formulate two noising processes: one where we keep $X$ fixed and noise $Y$, and one where we keep $Y$ fixed and noise $X$. In the noising processes, we inject independent Gaussian noise into the noised variable, obtaining the noised terms\n\\begin{align}\n    \\widetilde{X} = X + \\varepsilon_X~~~ \\text{and}~~~ \n    \\widetilde{Y} = Y + \\varepsilon_Y, ~~~ \\varepsilon_X,\n    \\varepsilon_Y, \\sim \\mathcal{N}(0,1).\n\\end{align}\nNow, in the denoising processes, we aim to find the best estimators\n$f_{\\varepsilon_X}, f_{\\varepsilon_Y}$ such that MSE losses\n\\begin{align}\n    (\\varepsilon_Y-f_{\\varepsilon_Y}(\\widetilde{Y},X))^2 ~~~\\text{and}~~~ (\\varepsilon_X-f_{\\varepsilon_X}(\\widetilde{X},Y))^2\n\\end{align}\nare minimized. \n\n**(W3)** *Dependence in the anti-causal direction*\n\nAs $X$ and $Y$ are functionally related, conditioning on either variable when predicting the added noise $\\hat{\\varepsilon}$ can induce dependence between the predicted noise and the conditioning variable, potentially in both directions. In our analysis of the ANM sub-case, we show that in the causal direction, the predicted noise remains independent of the conditioning variable, whereas in the anticausal direction, statistical dependence is introduced (Theorem 4.2).\n\nUnder the ANM-UM, $Y$ can be modelled as a series of transformations of $X$ where at each step an independent error is injected additively. Intuitively, we hypothesize that the additive noise is easier to decouple from $X$ when denoising $Y$ while conditioning on $X$, thereby rendering the estimate of the noise added to $Y$ ($\\hat{\\varepsilon_Y}$) less dependent on $X$. In contrast, in the anticausal direction, $X$ can only be modelled as a series of transformations of $Y$ where the error is _not additively injected_; intuitively, this would implying that denoising $X$ while conditioning on $Y$ would lead to a predicted noise that is more dependent on $Y$.\n\nEmpirically, we observe a similar pattern empirically in ANM-UM: the amount of dependence differs between directions, with the causal direction consistently yielding the smaller dependence. The magnitude of the estimated dependence is influenced by several factors (link function, noise dist, etc.).  Therefore, we focus on the gap between the dependencies in the two directions as the signal for inferring causal direction.\n\nWe have included a summary of this discussion in the paragraph before Section 4.1 on page 5.\n\n\n\n**Questions**\n\n**(Q1)** *Multiple $Z_i$ or only one $Z_i$*\n\nThe core challenge in causal discovery with unobserved mediators lies not in their number, but in the fact that all $Z_i$ are unobserved. Existing methods cannot be applied in this setting, as they rely on access to observed mediators. In contrast, our method is explicitly designed to handle entirely unobserved mediation. \n\nWe choose to model the process with multiple latent variables, instead of collapsing all $Z_i$ into one $Z$, to preserve the independent additive noise structure. If we used only one latent $Z$, then when there is nonlinear mediator $Z_i$ in the causal chain between $X$ and $Y$, then $Y$ would be a non-additive function of $Z$ and noise. This breaks the underlying ANM structure we exploit.\n\n\n**(Q2)** *How inclusion introduces dependence*\n\nTo clarify why adding this conditional information might introduce dependence, we will add the following paragraph in Section 4.1 after `However, this inclusion may also introduce dependence between the predicted noise and the unnoised variable`:\n\n> Consider the case where $X$ is the cause and $Y$ is the effect. Under the ANM-UM assumption, this implies that $Y$ is a function of $X$ and independent noise terms. Let $(X_i, Y_i)$ be a fixed sample from the dataset. Suppose we add noise to $Y_i$ to obtain a noised version $\\tilde{Y}_i = Y_i + \\varepsilon_Y$. When predicting $\\varepsilon_Y$ from $\\tilde{Y}_i$ and $X_i$, the estimate $\\hat{\\varepsilon}_Y$ may depend on $X_i$, since $X_i$ contains information about the true value of $Y_i$. This induces statistical dependence between $X$ and the predicted noise $\\hat{\\varepsilon}_Y$. The same argument applies in the anticausal direction. However, the dependence is not symmetric, as the functional relationship between $X$ and $Y$ is not symmetric. This asymmetry in conditional dependence provides a useful signal for causal identification and motivates our use of a conditional diffusion model.\n\n**(Q3)** *Timsteps*\n\nWe would like to clarify that the timestep $t$ in our model does not correspond to an iterative sampling process with varying outputs. Rather, it represents a fixed point in the denoising trajectory of the learned diffusion model. \n\nAt each timestep $t$, we generate predictions of the added noise in both directions, $\\hat\\varepsilon_{X | Y}$ and $\\hat\\varepsilon_{Y | X}$. For each direction, we provide the model with three inputs: (1) the conditioning variable (either $X$ or $Y$), (2) the noised variable (either $\\tilde{Y}$ or $\\tilde{X}$), and (3) the timestep $t$.\nWe then compute the dependence between the predicted noise and the conditioning variable, yielding a dependence estimate for each direction at each timestep.\n\nEmpirically, we found that comparing the dependence estimates across timesteps (i.e., using a majority vote) outperformed simply averaging them over time. For a detailed comparison of how this choice affects BiDD’s performance, we refer the reader to Appendix F.1, where we evaluate both the voting and averaging strategies empirically.\n\n\n**(Q4)** *Differences in setups*\n\nWe investigated several factors that influence the difficulty of inferring the correct causal direction, including the choice of linking mechanism and noise type (Table 2), as well as the number of mediators and sample size (Figures 2a and 2b).\n\nWe found that BiDD generally performs better when the link function is non-invertible and when the noise follows a uniform distribution (Table 2). As expected, increasing the sample size consistently improves performance (Figure 2b). The effect of the number of mediators is more nuanced, and depends strongly on the specific combination of linking function and noise type. We provide a detailed analysis of this interaction in our response to *(Q5)* below.\n\n**(Q5)** *Number of mediators*\n\nGenerally, as the number of mediators increases, the dependence between $X$ and $Y$ weakens, and in the limit (under finite samples), they become independent. In such cases, our method will output each direction with approximately 50\\% probability.\n\nThe number of mediators that our method can accommodate depends on the specifics of the data-generating process. To determine the influence of different link functions and noise types, we re-ran the synthetic exps. using two mediators instead of one:\n|                      |  Linear, Uniform | Neural Net, Normal | Neural Net, Uniform | Quadratic$^A$, Normal | Quadratic$^A$, Uniform | Tanh, Normal $^B$| Tanh, Uniform |\n|-----------------------|----------------|--------------------|---------------------|-------------------|--------------------|--------------|---------------|\n| $BiDD_\\textrm{Total}$  |       0.65       |         0.80         |         0.95          |        0.70         |         0.30         |      0.45      |       0.85      |\n| $BiDD_\\textrm{Test}$   |       0.70       |         0.75         |         0.95          |        0.70         |         0.60         |      0.45      |       0.85      |\n\n[$A$] Results for the quadratic link function with high number of mediators tend to become unreliable, because the repeated application of the quadratic operation tends to lead to numerical instability, as reported in previous work [1]. [$B$] Results for the Tanh link function + Gaussian noise can become unreliable if sample size is not high enough, because the Tanh function is close to linear across -1 to 1, where most of the mass of a Gaussian dist. is, rendering the empirical distribution close to the unidentifiable linear Gaussian case.\n\nWe observe that BiDD’s performance with two mediators remains stable in some settings (e.g., Tanh + Uniform), while it degrades significantly in others (e.g., Linear + Uniform). To investigate this behavior further, we extended our synthetic experiments by increasing the number of mediators for two cases: Tanh + Uniform and Neural Net + Normal. The results are summarized below:\n| Number of mediators     | 1   | 2    | 3    | 4    | 5    | 10   |\n|-------------------------|-----|------|------|------|------|------|\n| $BiDD_\\textrm{Total}$ (Tanh + Uniform)          | 1.0 | 0.9  | 0.8  | 0.95 | 0.85 | 0.95 |\n| $BiDD_\\textrm{Total}$ (Neural Net + Normal) | 1.0 | 0.95 | 0.75 | 0.75 | 0.8  | 0.55 |\n\nWhile performance remains relatively robust for Tanh + Uniform, the Neural Net + Normal setting exhibits a clear decline.\n\nWe propose to add the results of both additional experiments to the paper.\n\n\n**References**\n\n${[}1{]}$ Hiremath et al. \"Hybrid top-down global causal discovery...\" NeurIPS 2024."}, {"review_id": "YYAOIZ6GkB", "reviewer": "Reviewer_mAkY", "summary": "This paper aims at inferring causal direction between two variables when unobserved mediators are present with nonlinear relations with the output , where additive noise model (ANM) assumptions may not hold. The author provided a through discussion why existing methods failed, and then proposed a Bivariate Denoising diffusion that is provable to learn the causal relation under ANM. Experiments on synthetic and real-world data are conducted.", "questions": "See the weakness above.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths. \n\n1. The paper considered an important scenario in bivariate causal discovery, and was overlooked by the literature. \n2. The authors rigorously analyze when and why existing methods fail under hidden mediators, and provide consistency guarantees for BiDD under standard ANM.\n3. BiDD outperforms existing methods on synthetic datasets across a range of mechanisms (linear, nonlinear, invertible/non-invertible).\n\nWeakness. \n\n1. The theoretical consistency of BiDD is only proved in the standard ANM setting, and only conjectured in more realistic settings with nonlinear mediators.\n2. Training two conditional denoising diffusion models and estimating mutual information is computationally intensive, which may limit scalability to large datasets.\n3. Assumption 2.2 is difficult to decipher. It is difficult to comprehend which constraints on f1,...,fN and noise to make the assumption holds.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. I maintain my positive score.", "rebuttal": "We thank the reviewer for recognizing that our work addresses an \"important scenario\", provides a \"rigorous analysis of when and why existing methods fail\", offers consistency guarantees for ANM, and demonstrates superior performance on synthetic datasets.\n\nWe appreciate the reviewer’s constructive feedback, which has helped us enhance both the clarity and conceptual depth of our work. In the following, we address all points raised:\n\n**Weaknesses**\n\n**(W1)** *Theoretical consistency*\n\nWe agree that a theoretical consistency guarantee for BiDD in the general case of ANM-UM, beyond our consistency result in the ANM, would significantly strengthen our contribution. However, we argue that our proof of consistency in ANM (Theorem 4.2) is an important first step, and an extension is nontrivial due to the presence of unobserved mediators, which introduces fundamental identifiability challenges. For more context on these issues, we refer to our answer to Q1 of Reviewer wazL.\n\n**(W2)** *Scalability*\n\nWe can decompose the scalability of our approach into two aspects:  \na) the ability to handle larger datasets or higher-dimensional input/output spaces, and  \nb) the runtime of the method when used as a subroutine, a common scenario for bivariate causal discovery, e.g. in global discovery methods.\n\n\n- **(W2a)** *Ability to handle large datasets*\n\n  - While diffusion models are computationally intensive, they are known to scale well to large datasets. For high-dimensional data, prior work has proposed latent diffusion models [1], which operate in a compressed latent space to significantly reduce memory and compute requirements while maintaining generation quality.\n\n- **(W2b)** *Runtime of BiDD*\n\n  - The scenario with unobserved mediators is inherently more complex than fully observed settings, which justifies a longer runtime. Nonetheless, when compared with the only other method that explicitly addresses this case, the CANM, BiDD is competitive:\n\n| Method | Runtime (s) |\n|--------|-------------|\n| BiDD   | 172.0       |\n| CANM   | 145.5       |\n\n\nWhile the primary focus of this work is on providing a theoretically grounded and consistent method, we agree that improving runtime is a valuable direction for future research. Since BiDD involves both training and generation, advances in accelerating diffusion models apply directly. Prior work has proposed techniques to make training more efficient [2], to speed up generation [3], and to fine-tune pre-trained models for downstream tasks rather than training from scratch [4], that all could be applicable here. We will include this as a potential future direction at the end of the paper.\n\n**(W3)** *Clarity of Assumption 2.2*\n\nOur identification results rely on the assumption that the noise injected into each unobserved mediator is mutually independent, which means that we can express $Y$ as \n$$\n    Y = F(X, \\varepsilon_1,\\ldots,\\varepsilon_{T})+\\varepsilon_{T+1}\n$$\nwhere $X,\\varepsilon_1,\\ldots,\\varepsilon_{T+1}$ are all mutually independent, and $F=f_{T+1}(f_T(\\ldots)+\\varepsilon_{T})$.\nFor the ANM-UM to be identifiable, it is essential that there does not exists a Backward ANM-UM, a model that satisfies the ANM-UM assumptions in the reverse direction ($Y \\rightarrow X$)\nThis would require the existence of functions $G$ and noise terms $\\hat\\varepsilon_1,\\ldots,\\hat\\varepsilon_{T+1}$ such that $X$ can be decomposed as a function of $Y$ and independent noise terms, i.e.,: \n\\begin{align}\n    X = G(Y, \\hat\\varepsilon_1,\\ldots,\\hat\\varepsilon_T)+\\hat\\varepsilon_{T+1}\n\\end{align}\nwhere $G=g_{T+1}(g_T(\\ldots)+\\hat{\\varepsilon_{T}})$\nand, additionally, $Y,\\hat\\varepsilon_1,\\ldots,\\hat\\varepsilon_{T+1}$ are mutually independent.\nFor a formal discussion of the conditions under which backward ANMs can arise, we refer the reviewer to Appendix D.2.\n\nTo improve clarity and provide additional intuition for the Backward ANM-UM concept, we will revise the paragraph preceding Assumption 2.2 in the main text as follows:\n\n> Prior work (Theorem 1, [Cai et al., 6]) shows that certain joint distributions over $(X, Y)$ admit ANM-UM representations in both directions, $X \\dashrightarrow Y$ and $Y \\dashrightarrow X$, rendering the causal direction unidentifiable without further assumptions. We refer to such distributions as Backward ANM-UM, as the structural assumptions of ANM-UM hold in both the causal and anticausal directions. These distributions arise only under pathological conditions, such as when all functions are linear and the noises are Gaussian. \n\n\n**References**\n\n${[}1{]}$ Rombach et al. \"High-resolution image synthesis with latent diffusion models.\" CVPR, 2022.\n\n${[}2{]}$ Choi et al. \"Perception prioritized training of diffusion models.\" CVPR, 2022.\n\n${[}3{]}$ Song et al. \"Denoising diffusion implicit models.\" arXiv, 2022.\n\n${[}4{]}$ Watson et al. \"De novo design of protein structure and function with RFdiffusion.\" Nature 620, 2023."}, {"review_id": "E3KuXVuE40", "reviewer": "Reviewer_UfQZ", "summary": "This paper proposes Bivariate Denoising Diffusion (BiDD), a method for identifying causal direction in the presence of unobserved mediators. BiDD leverages diffusion models, hypothesizing that noise predictions are less dependent on the cause than the effect. Empirical evaluations demonstrate BiDD’s superiority over baselines on synthetic and real-world data.", "questions": "- What properties of diffusion (e.g., noise-perturbation process) make them more effective in handling unobserved mediators? Can you offer a theoretical analysis to it?\n- What is the identifiability when nonlinear mediator exists.", "limitations": "See Weaknesses.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n- BiDD adapts diffusion models to bivariate causal discovery.  \n- Extensive experiments validate BiDD’s performance.\n\nWeaknesses:\n- First, the paper does not adequatly explain why diffusion models are theoretically better suited for this task compared to alternatives like VAEs.   \n- Decision Rule 1 appears incorrect, as it ignores the edge case where \\( MI(\\epsilon_Y, X) = MI(\\epsilon_X, Y) \\).\n- Theorem 4.2 only addresses identifiability when no nonlinear mediator exists. The paper should further provide guarantees for the more general case where nonlinear mediators are present.", "was_revised": true, "final_justification": "Thank you for the authors’ response. Most of my concerns have been addressed, and I will raise my score accordingly. I recommend that the key differences from [1] be well discussed in the final version.", "rebuttal": "We thank the reviewer for recognizing the \"superiority over baselines\" and appreciating the \"extensive experiments\". We also value the reviewer's thoughtful questions, which have helped us improve both the clarity and conceptual rigor of our work. In the following, **we respond to all points raised**.\n\n**Weaknesses and Questions (combined)**\n\n**(W1+Q1)** *Diffusion models vs. VAE*\n\nWhile both VAEs and diffusion models are generative frameworks, how this paper and the CANM paper used them was both theoretically different and practically different.\n\n*Theoretical analysis:*\nCANM [1], based on a VAE, minimizes a reconstruction loss combined with a regularization term on the latent space. This does not align directly with the assumptions of the additive noise model with unobserved variables (ANM-UM). Other methods relying on MSE-Minimization therefore need to include other assumptions for theoretical guarantees (see Section 3 --> Failure Modes -> MSE Minimization) As discussed in Section 3, such approaches may fail even under ANM-UM, as proven in Lemma 3.4 (*MSE-Minimization Fails*)\n\nIn contrast, BiDD utilizes a diffusion model trained via mean squared error (MSE), but evaluated through an independence test between the predicted noise and the conditioning variable. This explicitly encodes asymmetry in the data-generating process, as implied by the ANM-UM assumption (independence of residuals in one direction only). As a result, our approach yields a formal theoretical guarantee for causal identifiability in the ANM setting—something the VAE-based method is unable to provide.\n\nAnother difference between VAE and Diffusion based methods, beyond how they are specifically implemented in CANM and BiDD, is that VAE guarantees are likely to depend on whether the dimensionality of the latent space, i.e. the number of unobserved mediators assumed by a VAE method, actually matches the true number of unobserved mediators in the causal DGP.  In contrast, denoising Diffusion does not require an estimate of an unknowable parameter for its procedure. Therefore, apriori, for tasks where aspects of the underlying causal structure (such as the number of mediators) are both important and hard to ascertain, we propose that it is theoretically justified to prefer Diffusion over VAEs.\n\n*Empirical performance:*\nIn practice, VAEs are known to be sensitive to hyperparameters and prone to instability. Their training objective includes two components—the KL divergence of the latent representation and the reconstruction loss—and the relative strength of these terms is highly dependent on optimization details. Avoiding posterior collapse often requires extensive tuning or auxiliary tricks.\n\nEven with an improved training schedule designed to mitigate posterior collapse (see Appendix E.9), we observed poor performance of the VAE baseline in synthetic experiments with invertible link functions (Tanh + Linear) and uniform noise. In contrast, our diffusion-based method trains robustly across a range of configurations and performs consistently well across all evaluated settings.\n\nIn summary, our choice of diffusion modeling is both theoretically motivated and practically effective, offering advantages over VAE-based alternatives in the context of causal discovery with unobserved mediators.\n\n\n\n**(W2)** *Edge case for decision rule 1*\n\nWe thank the reviewer for pointing out that edge case.\nWe updated our decision rule so that we do not decide when the edge case of equality occurs:\n> **Decision rule 1** (Bivariate Denoising Diffusion (BiDD))\nLet $\\hat\\varepsilon_Y=\\varepsilon_{Y,\\theta}(\\widetilde{Y},X)$, $\\hat\\varepsilon_X=\\epsilon_{X,\\theta}(\\widetilde{X},Y)$ be the predictions of the noise added to $Y,X$, respectively.\nGiven a mutual information estimator $MI(\\cdot)$,\nif $MI(\\hat\\varepsilon_Y, X) < MI(\\hat\\varepsilon_X, Y)$, conclude that $X$ causes $Y$. If $MI(\\hat\\varepsilon_Y, X) > MI(\\hat\\varepsilon_X, Y)$, conclude that $Y$ causes $X$. Else, do not decide.\n\nWe will apply the same update to the other decision rules presented in the paper to ensure consistency.\n\n**(W3 + Q2)** *Theoretical consistency in the nonlinear mediator case*\n\nOur theoretical analysis assumes a linear mediator, as this preserves a simple Additive Noise Model (ANM) structure even when the mediator is unobserved. In contrast, nonlinear mediators break this structure—making the relationship between cause and effect no longer additive. This loss of structure complicates the analysis, as it does not allow for a relatively straightforward decomposition anymore. However, we argue that our proof of consistency in ANM (Theorem 4.2) is an important first step, and an extension is nontrivial: for more context on these issues, we refer to our answer to Q1 of Reviewer wazL.\n\n**References**\n\n${[}1{]}$ Cai et al. \"Causal discovery with cascade nonlinear additive noise models.\" arXiv, 2019."}, {"review_id": "Jx9ftlQosW", "reviewer": "Reviewer_wazL", "summary": "This paper discusses the problem of causal direction identification under the addictive noise model with unmeasured mediator (ANM-UM). The authors first analyze the equivalent condition for ANM-UM to reduce to common ANM. Then, they analyze how existing methods based on ANM can fail under unmeasured mediator. Next, under the linear setup, they develop an identification rule of the causal direction via denoising. Experiments on synthetic and real-world datasets are conducted for verification.", "questions": "**1.** Can the authors explain the major challenge when extending Thm. 4.2 to the irreducible case? Is there any potential idea for the extension?\n\n**2.** If BiDD works under the irreducible case (as conjectured in Sec. 4.2) and the baselines do not work under ANM-UM (as shown in Sec. 3), why their performance seems to be comparable?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n**1**. This paper establishes the necessary and sufficienct condition for the ANM-UM to reduce to the ANM.\n\n**2**. The authors conducts exhausive analyses on why existing methods fail under unmeasured mediation.\n\n**3**. They conduct both synthetic and real-world experiments for validation.\n\n**Weakness**\n\n**1**. The consistency of the decision rule is shown only for the case where ANM-UM reduces to ANM. For the irreducible case, e.g., nonlinear causal relationship, there is no theoretical intuition why this method can work. \n\nIndeed, the authors consider empirical verifications in Sec. 5. But the improvement of BiDD over baseline methods seems to be insignificant. I not sure whether this result can support the conjecture that BiDD also works under the irreducible case.", "was_revised": true, "final_justification": "The author's response has partly addressed my concern. So I keep my positive score.", "rebuttal": "We thank the reviewer for highlighting our establishment of the \"necessary and sufficient condition for the ANM-UM to reduce to the ANM\", our \"exhaustive analysis\" of failure modes in existing methods, and our use of \"synthetic and real-world experiments for validation\". We also appreciate the reviewer's insightful questions, which has helped us improve the clarity and conceptual strength of our work. In what follows, **we address all points raised**.\n\n**Weaknesses and Questions (combined)**\n\n**(W1+Q1)** *Extending Thm 4.2 to Irreducible Case*\n\nWe first note that Theorem 4.2 is non-trivial and, to our knowledge, represents the first theoretical result on using denoising diffusion for bivariate causal discovery—not merely to estimate score function Jacobians, as in prior work [1], but to reveal asymmetries arising from the independence structure inherent to ANMs.\n\nThe major challenge in extending Theorem 4.2 to the irreducible case, i.e., when ANM-UM does not reduce to ANM due to the presence of a nonlinear mediator, lies in the increased difficulty in analyzing the properties of the estimator $E[\\varepsilon_Y|\\tilde{Y},X]$. This difficulty arises when $X$ becomes non-additively related to the latent noise as a result of nonlinear mediators.\n\nNote, our procedure rests on the intuition that noise predictions from a conditional diffusion model will be less dependent on the condition when the condition is the cause, rather than the effect. \n\nWhen the ANM-UM reduces to ANM, we can actually show that _the noise prediction is entirely independent of the condition_ when the condition is the cause. We can write $Y$ as a (potentially nonlinear) function of $X$ plus noise, i.e. $Y = f(X)+\\varepsilon_1$. This clean additive separation between the noise $\\varepsilon_1$ and $f(X)$ allows us to show that the predicted noise $\\hat\\varepsilon=E[\\varepsilon_Y|\\tilde{Y},X]= E[\\varepsilon_Y|\\tilde{Y},X, \\tilde{Y}-f(X)]=E[\\varepsilon_Y|\\tilde{Y},X, \\varepsilon_Y+\\varepsilon_1]= E[\\varepsilon_Y|\\varepsilon_Y+\\varepsilon_1]$. This reformulation of $\\hat\\varepsilon$ into a term that is clearly independent of $X$ is the backbone of our proof of Theorem 4.2.\n\nHowever, when the ANM-UM does not reduce to ANM, i.e. because of a nonlinear mediator, $Y$ can only be written as a function of $X$ and noise (Proposition 2.3): $Y = f(X) + g(\\varepsilon) + h(X,\\varepsilon)$, where $X,\\varepsilon$ are nonlinearly combined in $h$. Here, its not possible to cleanly project $X$ out of $\\tilde{Y}$, i.e., there does not exist a function $k(\\cdot)$ such that $k(\\tilde{Y},X)$ is independent of $X$. Therefore, to extend Theorem 4.3 to this setting, we would need to show that $E[\\varepsilon_Y|\\tilde{Y},X]$ has a smaller dependence on $X$, when compared to the dependence between $E[\\varepsilon_X|\\tilde{X},Y]$ and $Y$. Analyzing the strength of the dependence would probably require a very different set of proof techniques, which are beyond the scope of this paper. One potential first step to bridge the gap between the reducible and irreducible settings is to identify conditions under which the interaction term $h(X,\\varepsilon)$ becomes sufficiently 'weak', such that $E[\\varepsilon_Y|\\tilde{Y},X]$ becomes 'close to independent'. This will probably involve some investigation into the interaction by functional forms, noise distributions and the resulting mutual information between $X$ and the interaction term $h(X,\\varepsilon)$ .\n\n**(W2+Q2)** *Comparable performance of the baselines*\n\nFor a method to be effective, it should work across all mechanisms that follow the ANM-UM, not only selected ones. In our synthetic experiments, we therefore tried to incorporate different types of noise (Normal, Uniform) and different invertible (Linear, Tanh) and non-invertible (Neural Net, Quadratic) link functions.\n\nAll baselines, except PNL and RESIT, fail in at least two settings, achieving accuracy of at most 50\\%. This renders them unreliable in scenarios where only the ANM-UM assumption holds.\n\n*Performance of RESIT, PNL, BiDD for two mediators*\nOf these three methods, all use residual indepenced, so their difference in performance probably depends on how much the underlying model violates ANM or PNL.\nWhile they might perform well in settings with a low number of mediators, they might fail with an increased number of mediators.\n\nTo analyze how RESIT and PNL perform when their assumptions are violated, we re-ran the synthetic experiments using the same setup as in Table 2, but with two mediators:\n|                      |     Linear ,  Unif.   |     Neural Net ,  Gauss.   |     Neural Net ,  Unif.   |     Quadratic$^A$ ,  Gauss.   |     Quadratic$^A$ ,  Unif.   |     Tanh ,  Gauss.$^B$   |     Tanh ,  Unif.   |     Tanh ,  Unif.   |\n|---------------------|------------------------|-------------------------------|--------------------------------|--------------------------|---------------------------|---------------------|----------------------|----------------------|\n| $BiDD_\\textrm{Total}$           |                    0.65 |                           0.80 |                            0.95 |                      0.70 |                       0.30 |                 0.45 |                  0.85 |                  1.00 |\n| $BiDD_\\textrm{Test}$ |                    0.70 |                           0.75 |                            0.95 |                      0.70 |                       0.60 |                 0.45 |                  0.85 |                  0.80 |\n| RESIT        |                    0.90 |                           0.50 |                            0.60 |                      0.00 |                       0.00 |                 0.60 |                  0.90 |                  0.90 |\n| PNL                  |                    0.55 |                           0.55 |                            0.55 |                      0.75 |                       0.40 |                 0.30 |                  0.50 |                  0.50 |\n\n[$A$] Results for the quadratic link function with high number of mediators tend to become unreliable, because the repeated application of the quadratic operation tends to lead to numerical instability, as reported in previous work [2].\n[$B$] Results for the Tanh link function + Gaussian noise can become unreliable if sample size is not high enough, because the Tanh function is close to linear across -1 to 1, where most of the mass of a Gaussian dist. is, rendering the empirical distribution close to the unidentifiable linear Gaussian case.\n\nWhile the average performance of all methods declines when increasing the number of mediators (e.g., from 0.86 to 0.68 for $BiDD_\\textrm{Total}$, 0.82 to 0.72 for $BiDD_\\textrm{Test}$, 0.75 to 0.52 for PNL, and 0.83 to 0.54 for RESIT), the drop is particularly pronounced for PNL and RESIT in settings with neural net link functions. Both BiDD variants consistently achieve at least 75\\% accuracy across noise types, whereas the best baseline score does not exceed 60\\%.\n\n\n**References**\n\n${[}1{]}$ Sanchez et al. \"Diffusion models for causal discovery via topological ordering\". ICLR, 2023.\n\n${[}2{]}$ Hiremath et al. \"Hybrid top-down global causal discovery with local search for linear and nonlinear additive noise models.\" Advances in Neural Information Processing Systems, 2024."}]

Author rebuttal:

Meta-review: This paper introduces Bivariate Denoising Diffusion (BiDD), a method for causal direction identification in the presence of unobserved mediators under the Additive Noise Model with Unmeasured Mediators (ANM-UM) setting. 

Strengths:

- The authors articulates why existing methods fail under unmeasured mediation and motivate BiDD's use of denoising diffusion models for causal discovery 

- The paper provides an analysis of failure modes in existing methods and establishes conditions for ANM-UM reducibility. 

Weaknesses:

- theoretical guarantees only apply to the reducible ANM-UM case.

- proposed method is computationally intensive, potentially limiting applicability to large-scale problems. 

- empirical improvements are sometimes marginal
Decision: Accept (poster)