Title: Score-informed Neural Operator for Enhancing Ordering-based Causal Discovery
Venue: NeurIPS 2025
Accepted: yes
Keywords: Causal Discovery

Score Matching

Diffusion Models

Neural Operator
Abstract: Ordering-based approaches to causal discovery identify topological orders of causal graphs, providing scalable alternatives to combinatorial search methods. Under the Additive Noise Models (ANMs) assumption, recent causal ordering methods based on score matching require an accurate estimation of the Hessian diagonal of the log-densities. However, previous approaches mainly use Stein gradient estimators, which are computationally expensive and memory-intensive. Although DiffAN addresses these limitations by substituting kernel-based estimates with diffusion models, it remains numerically unstable due to the second-order derivatives of score models. To alleviate these problems, we propose Score-informed Neural Operator (SciNO), a probabilistic generative model in smooth function spaces designed to stably approximate the Hessian diagonal and to preserve structural information during the score modeling. 
Empirical results show that SciNO reduces order divergence by 42.7% on synthetic graphs and by 31.5% in real-world datasets on average compared to DiffAN, while maintaining memory efficiency and scalability. Furthermore, we propose a probabilistic control algorithm for causal reasoning with autoregressive models that integrates SciNO's probability estimates with autoregressive model priors, enabling reliable data-driven causal ordering informed by semantic information. Consequently, the proposed method enhances causal reasoning abilities of LLMs without additional fine-tuning or prompt engineering.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on a key challenge of ordering-based causal discovery: the computationally expensive and memory-intensive issues of score's Jacobian (Hessian) diagonal estimation. Motivated by FNO and DiffAN, this paper proposes a new time-conditioning score-based generative model, Score-informed Neural Operator (SciNO), to stably approximate the Hessian diagonal by modeling the score function with neural operators. This architecture can improve the scalability of the existing algorithms, e.g., SCORE and CaPS, by reducing their cubic computational complexity. Additionally, combining LLM with the leaf node discrimination criteria in previous works, this paper proposes a probabilistic control of autoregressive causal ordering algorithm. With the approximation power of neural operators in Theorem 3.1, the idea is theoretically sounds. Some synthetic and real-world experiments are presented.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. This paper is well-motivated. Improving the estimation efficiency of Hessian diagonal can improve the scalability of many existing ordering-based causal discovery methods.\n\n2. Theoretical motivations are clearly explained and the proof are adequately provided. Theorem 3.1 shows the approximation power of neural operators. The theorems and mathematical details appear to be correct, though I haven't checked all the details.\n\n3. Despite some formatting issues, this paper is well-written and easy to understand.\n\nCons:\n\n1. Lack of some important experiment. Although the main purpose of this paper is to reduce the computational complexity of Hessian diagonal estimation, the accuracy of the approximation is also important. Here I have two main concerns.\n\n* About the accuracy of Hessian diagonal estimation, SciNO can have a better or comparable performance compared to the second-order Stein gradient estimator? The author can illustrate this point by adding the second-order Stein gradient estimator in figure 2 or by presenting a comparison of order divergence.\n\n* Although SciNO can enhance the scalability of CaPS, it seems that it cannot stably improve the performance of CaPS. Does SciNO perform poorly on low-dimensional data? Providing more datasets (like Table 2) can be helpful for this analysis.\n\n2. This paper seems to be concatenated together by two independent parts, one is SciNO and the other is the autoregressive LLM control algorithm. I suggest that at least a comparison between the LLM-based method and the previous methods, such as DiffAN and CaPS, should be provided.\n\nQUESTIONS:\n1. Could you further explain what the subgraph below in Figure 5 intends to express? What specific distributions do the prior and posterior here refer to? What does this change indicate?\n\n2. See strengths and weaknesses. Most of my concerns are about the experimental part. I will adjust the score if the above experiments can reach a consistent conclusion.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of ordering based causal discovery. Identifying that an existing method DiffAN suffers from unstable gradient estimation,  authors proposed Score-inforrmed Neural Operator (SciNO) to stably approximate the Hessian diagonal and to preserve structural information during the score modeling.  This is achieved by introducing two key modifications to the time-conditional Fourier neural operator architecture: a learnable time encoding module and that decomposition of signals in Fourier layers into real and imaginary parts. Further, the paper proposes a probabilistic control method that leverages score-informed statistics inferred by SciNO, guiding autoregressive models to generate more reliable causal reasoning. Empirical results validate that the improved stability of gradient estimation leads to better causal discovery performance compared with DiffAN.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper identifies the key limitation of an existing algorithm, and the modification is sound. \n- The proposed method is efficient, compared with  DiffAN.\n- Writing is generally good (though can be further improved; see below.)\n\nWeakness:\n\n- Lacking comparison with classic algorithms.\n- Novelty is a bit limited.\n\nQUESTIONS:\n1. can you give more details on : \"injecting noise into the data can destroy the functional information inherent in the original score function\" (line 50-51) ？\n\n2. line 150 :\".. allows SciNO to achieve robust performance in high-dimensional causal graphs\": 'robust performance' is vague here. Do you mean the trend of SHDs along number of nodes or the standard deviation? Please make it accurate. \n\n3. regarding Section 3.3: it seems that the probabilistic control can also be applied to the original DiffAN? if so, what would be the difference?\n\n4. **Major Concern**: The major weakness is the limited empirical comparison, lacking classic/existing benchmark methods. In particular, the paper only compares the proposed method with its base method DiffAN. It is not clear how the algorithm performs within the whole causal discovery literature. For example, the CAM method shall  perform well on the synthetic data setup of this paper. See, e.g., [1, 2] for references, where their setup is the identical or at least very similar to the one in this paper. In particular, for ER4d (d=100) case, CAM achieves  98.8$\\pm$ 20.7 SHD, while the proposed method achieves 180.0$\\pm$15.45. Of course, a new algorithm does not need to beat all existing methods, but this large difference needs further clarifying. \n\n    [1] Gradient-Based Neural DAG Learning, ICLR, 2020.\n\n   [2] Ordering based causal discovery with reinforcement learning, IJCAI, 2021.\n\n5. minor on paper writing/organization: \n\n  - line 150: almost half of section 3.2 is about ablation study or case study, which distracts the reading flow. Suggest to move it to experiment section. \n  - tables 1-4: table caption is generally placed on top.\n  - line 126: \"HDM implements time-conditioned Fourier Neural Operator (FNO, [22]) by adding projected tensors from PE (Positional Embedding, [40]) \": be consistent about the format of abbreviation.\n\nLIMITATIONS:\nYes. \n\nAnd see above the limitation on lacking important benchmark methods. At least further clarification is needed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecent methods for causal discovery rely on iteratively finding leaf nodes using the second-order derivative of the log-densities. However, current methods show a lack of stability and accuracy in the estimation of such a derivative. This method leverages and improves upon recent advances in diffusion models for more accurate estimation of derivative. These scores can be used to iteratively find leaf nodes, or to improve LLM-driven causal discovery. Experiments show the improved performance induced by the new estimation method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: the method is original and is a welcome technical contribution, the paper is generally clear.\n\nWeaknesses: the Theorem is a bit narrow and claims in the experiments are dubious.\n\nTheory (Theorem 3.1) :\n\n1a) The assumptions on the Sobolev space are strong; this means k>27 for D=50 and k>52 for D=100!\n\n1b) Further, it is not applied to a score with a time variable, at least when the time is close to 0.\n\nMain concerns on experiments:\n\n2a) Only 5 independent runs are run in Sections 4.1-4.2, and even more concerning, I understand that the best of 3 runs is Section 4.3.\n\n2b) l.230: \"Table 2 shows that SciNO consistently outperforms the original DiffAN across all metrics\" : I see ties in a majority of comparisons in Table 2, both for ER and \"real-world\" (see 3b) graphs.\n\n2c) \"SciNO achieves comparable performances across all datasets, with improved results observed on higher-dimensional graphs such as MAGIC-IRRI and ARTH150.\" The baseline is better with statistical signifiance in a majority of comparisons in Table 3, even most of MAGIC-IRRI.\n\n3) More minor points in experiments\n\n3a) There is no comparison with the original SCORE method while it is extensively discussed before.\n\n3b) BNLearn and Physics are not \"real-world\" data as I understand it, I would downgrade the \"real data\" to \"semi-synthetic data\"\n\n3c) A time complexity comparison between CaPS and the method is done in l.173-174, but Figure 4 does not show runtime.\n\n3d) The analysis of SciNO PE vs LTE is not done on other datasets than ER.\n\nQUESTIONS:\nQuestions are in the Weaknesses above; I am ready to significantly raise my score if they are mostly addressed.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the ordering based causal discovery methods using score matching in the ANM models.\n\nThe causal ordering is identified usually by iteratively identifying the leaf nodes, by estimating Jacobian of the score function, and in particular, its diagonal, denoted as the Hessian diagonal function. However, such estimation is not easy: current methods may suffer from numerical instability, computational complexity, high dimensional issues, etc.\n\nHence, the authors introduce SciNO (Score-informed Neural Operator), a functional diffusion model framework designed to provide stable and scalable approximation of the Hessian diagonal. Score functions are modeled in Sobolev spaces using neural operators.\n\nFurthermore, the authors propose a probabilistic control framework for autoregressive models (e.g., LLMs), integrating SciNO’s score-derived statistics with LLM outputs to guide more accurate causal ordering.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The motivation is clear and the problem is relevant. The identified challenge of unstable second-order derivative estimation is indeed important in score matching based causal ordering identification.\n\n2. The techniques introduced -- the use of Sobolev space approximations and neural operators -- are new to the causal discovery field (at least to my knowledge). While I haven’t verified the theoretical details in full, the framework appears sound. It would be valuable if the authors could comment on whether these theoretical insights could benefit other problems in causal discovery.\n\n3. The section on Probabilistic Control of Autoregressive Causal Ordering is interesting and seems to bring a new perspective to causal discovery, both for evaluation and for debiasing in inference results.\n\n\n**Weaknesses:**\n\n1. While the proposed LLM control framework is interesting and new, it would help to include a deeper discussion on when and why it works. For example, how much do variable semantics and naming conventions influence the predictions? Are the LLM priors generally informative or misleading in high-dimensional settings? \n\n2. For the techniques, could the authors clarify which parts of the proposed architecture are proposed in this paper and which are adapted from existing methods on other problems? If certain components are adopted elsewhere it would be helpful to discuss what problems they were originally designed to address.\n\n3. Since this is an empirical causal discovery method, the experimental comparisons could be more comprehensive. It would be informative to include baselines beyond the ANM category (e.g., constraint-based or score-based methods) to better contextualize.\n\nQUESTIONS:\n/\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the approximation of score functions using a newly proposed method, named “score-informed neural operator.” This method was applied to causal discovery score-based methods, and applied to an LLM reasoning task.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper introduces many topics, and I often find the details missing, hindering the understanding of the paper. Though, the proposed method seems to work better in approximating the score function than prior methods, the framing of the paper focuses on causal discovery, which unnecessarily complicates the problem setting (with key benchmarks in the causal discovery literature missing). The LLM reasoning is disconnected with the rest of the paper, and the setup description lacks clarity.\n\nQUESTIONS:\n1. The paper starts by introducing causal discovery. However, key related work in ANM is missing. There are ANM methods that start building the causal graph from roots from example. To claim that this method works for causal discovery, it is necessary for the authors to compare against methods beyond score-based approaches. I would actually recommend the authors introducing the method as a generic score approximation method instead,\n2. The causal discovery part and the LLM part feel like two standalone topics, and it is strange to put them into the same paper.\n3. Section 3.1 is disconnected from Section 3.2. The authors need to explicitly spell out 1) the input to the Fourier transform and the output , and 2) how theorem 3.1 is leveraged to improved the previous methods. In addition, it is unclear what ‘k’ means in equation (7). Is the Fourier transform used to ensure higher-order derivatives exist? How is the model operating in the function space? I found the high-level description at the beginning of Section 3.2 unclear. \n4. The baseline architecture in [24] is not described in words in the paper, and the authors did not perform ablation to justify the added modifications. LTE seems like a popular heuristic in modern diffusion model trainings. If separating the Fourier layer into the real and imaginary part is the only novel part, I feel the contribution of the paper is incremental.\n5. The authors use the phrase causal representation to mean causal discovery at the end of page 4. I think the authors should correct that as causal representation is often referred to as a different task.\n6. I found Section 3.3 unclear: what is node set V? For the Confidence interval, the authors needs to justify why the data is I.i.d. In order to apply the central limit theorem. \n7. The setup of the synthetic experiments missing key information in the main body, and the authors should consider adding these key information back. For example, the authors said “we sample node values according to an additive noise model with Gaussian noise.” However, it is important to include the information that the ANM is nonlinear, since linear Gaussian mechanisms are not identifiable.\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}