Title: A Hierarchy of Graphical Models for Counterfactual Inferences
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graphical Models

Causal Inference

do-calculus

counterfactual-calculus

Bayesian Networks

Causal Bayesian Networks

Counterfactual Bayesian Networks
Abstract: Graphical models have been widely used as parsimonious encoders of assumptions of the underlying causal system and provide a basis for causal inferences. Models encoding stronger constraints tend to require higher expressive power, which are also harder, and sometimes impossible to empirically falsify. In this paper, we introduce two new collections of distributions that include counterfactual quan- tities which are experimentally accessible under counterfactual randomizations. Correspondingly, we define two new classes of graphical models for encoding empirically testable constraints in these distributions. We further present a sound and complete calculus, based on counterfactual calculus, which licenses inferences in these two new models with rules that are within the empirically falsifiable bound- ary. Finally, we formulate a hierarchy over several graphical models based on the constraints they encode and study the fundamental trade-off between the expressive power and empirical falsifiability of different models across the hierarchy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPCH consists of three layers, observational, interventional, and counterfactual. When the underlying SCM is fully known, then we can compute any query to the model. However, in reality, the model might not be fully known. At the interventional level, it is possible, at least in principle, to obtain full knowledge by doing experiments. This it not true for the counterfactual level. in [19], the concept of counterfactual realizability was introduced, which formalizes the concept of being able to sample from a counterfactual distribution by doing experiments. Not all counterfactual distributions are realizable. In the present paper, the authors formalize two subclasses of the counterfactual layer of PCH by restricting the kind of counterfactual worlds that can be present. These subclasses are realizable in the sense of [19]. \n\nThe main results are the following:\n1) The proof that two corresponding classes of causal Bayesian networks capture the distributions defined by the new layers (Thm 1).\n2) The adaption of the counterfactual calculus to the subclasses and the proof that a quantity is identifiable if it can be expressed in terms of the given distribution using the rules (Thm 2). \n3) The proof that the distributions defined in terms of SCMs of the different layers form a hierarchy (Thm 3) as well as the constraints encoded in the corresponding graphical models (Thm 4).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper makes interesting new theoretical contributions by defining sublayers of the counterfactual layer of PCH which might be easier to falsify by experiments. These layers might be easier to understand, since the way counterfactual worlds can be used is restricted.  \n\nWeaknesses:\n- On the technical side, I think the contributions are rather incremental.\n  - The proof of Theorem 1 is just a check of the properties, which is similar to proofs for the standard layers. Which is to some extend to be expected, as the construction procedures for the CBNs are the same.\n  - In the proof of Theorem 2, soundness is obvious and completeness follows by the same argument as for layer 3, even using the same algorithm.\n  - For the proof of Theorem 3, one only has to check that layer 2.25 is subsumed by layer 2.5, which is quite obvious since Definition 2 generalizes Definition 1, as is the fact that it transfers to the corresponding CBN.\n- One of the selling points of the paper is that the new layers allow empirical falsifiability. The example in the paper are of mathematical/explanatory nature, which is fine. However, the examples one finds in other papers (like changing the color of cars in images) are a little contrived.\n\nComments:\n\n- I think the naming of the new layers is not good. The numbers are arbitrary. Why not 2.33 and 2.66? Furthermore, if someone next introduces a layer between interventional and counterfactual that is not comparable to your new layers, you created a mess, since real numbers have a total order. That being said, already numbering the layers of PCH might have been a mistake, since there is no natural numerical parameter.\n- The paper is not easy to read (which to some part cannot be avoided, since counterfactual reasoning is complicated). Section 2 is pretty dense, while I find section 3 rather lengthy. While Appendix A contains lots of information, I find it to lengthy to be a good reference manual but contains to few explanations to be a good introduction. I do not think that the [..]-notation is explained anywhere, but I might have overlooked it.\n\nMinor comments:\n\n- line 110: V -> U\n- line 113: Val -> \\mathit{Val} (or whatever you prefer)\n- line 118: \\in -> \\subseteq\n- line 158: maybe use x and x'. The indices do not correspond to the indices used in Definition 1\n- page 5, footnote: An \"Appendix\" is missing (most likely a c in \\cref)\n- line 222: Definition 11 only appears in the appendix\n- Eq (35): fi -> f_i\n- Definition 12: The second sentence is not a proper mathematical definition\n- It might be helpfull in Def. 18 to discuss the relation to Def. 13\n- Definition 24 is also not a proper mathematical definition. What is a randomization device is this context?\n- line 693: a blank is missing\n- Lemma F.1 seems to be folklore to me.\n- line 1018 2.25 -> 2.5\n- line 1247: looks like some latex issue\n\nQUESTIONS:\nNo particular question, but I am thankful for any comments regarding the mentioned weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces two new classes of causal graphical models, CBN2.25 and CBN2.5, which encode constraints for distributions made experimentally accessible via counterfactual randomization. These models bridge the gap between interventional (L2) and counterfactual (L3) layers in Pearl's Causal Hierarchy (PCH). The authors demonstrate a novel hierarchy of graphical models where higher-level models offer greater expressive power at the cost of reduced empirical falsifiability. They provide theoretical guarantees for their calculus, proving soundness and completeness for inference within these new frameworks.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality\n### Strength\n1. The paper is technically robust. Definitions of distributions (Def 1/2), graphical constraints (Def 3/4), and theorems linking models to SCMs (Thm 1) are precise. Lemma 1 (CRS criterion) provides a critical tool for verifying distribution membership in PCH layers, ensuring theoretical validity.\n2. The authors prove that ctf-calculus is sound and complete for CBN2.25/CBN2.5 (Thm 2), which is foundational for their utility in causal inference. This strengthens the practical relevance of the proposed machinery.\n3. The refinement of L3 into experimentally accessible sublayers (L2.25/L2.5) and formalization of a hierarchy connecting BN, CBN, and CTFBN is conceptually original. Thm 4 rigorously establishes how constraints propagate across layers.\n### Weaknesses:  \n1. The paper does not provide empirical validation or experiments to demonstrate the practical utility of these models. While theoretical completeness is ensured, real-world performance remains unexplored.\n\n---\n## Clarity\n### Strength\n1. The paper follows a logical flow: problem motivation -> formalisms -> hierarchical analysis. Figures 3–6 and tables like Table 1 aid conceptual understanding.\n2. The paper systematically maps new models into PCH (Thm 3) and establishes their hierarchical relationships via constraint nesting (Thm 4). This enhances conceptual clarity for model selection in causal inference tasks.\n### Weaknesses:  \n1. Dense mathematical notation in Def 1/2 and equations for distributions. For example, the distinction between P^L_{2.25} and P^L_{2.5} could benefit from clearer intuitive explanations alongside formal definitions.\n\n---\n## Significance\n### Strength\n1. The work advances the understanding of counterfactual inference by bridging theoretical limitations (non-experimental L3) with empirical feasibility via counterfactual randomization. This is critical for AI systems requiring robust causality foundations, particularly in fields like healthcare or policy analysis.\n2. The hierarchy framework provides a principled tool for model selection: researchers can choose models that balance expressive power and falsifiability per task.\n### Weaknesses:  \n1. The paper focuses almost exclusively on theoretical contributions without illustrating their application to real-world problems. For instance, it lacks examples of how these models improve over existing methods, like CTFBN, in scenarios like estimating ETT or NDE.\n\n---\n## Originality\n### Strength\n1. The formalization of L2.25/L2.5 as refinements of PCH’s L3 is novel, especially the introduction of AMWN for inferential machinery and the CRS criterion (Lemma 1).\n2. The paper clearly distinguishes its work from prior models and situates them within an expanded PCH framework (Thm 3/4). This contextualizes the contribution without unnecessary redundancy.\n\nQUESTIONS:\n1. How do the computational costs of ctf-calculus in CBN2.25/CBN2.5 compare to Pearl’s do-calculus or ctf-calculus in CTFBN? Could the additional constraints impact scalability?\n2. What concrete applications have been tested with these models, and how do their results compare to prior methods like CTFBN? For instance, in estimating ETT (Example 5), are there cases where CBN2.25/CBN2.5 fail due to insufficient assumptions?\n3. The authors discuss falsifiability but do not address potential societal risks (like biases in counterfactual reasoning). Are there scenarios where CBN2.25/CBN2.5 could amplify ethical concerns compared to prior models?\n\nLIMITATIONS:\nThe authors adequately address limitations by explicitly analyzing trade-offs between model expressiveness and falsifiability (Section 3).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This is a theoretical paper on the lack of falsifiability in the third, counterfactual layer of Pearl's hierarchy. Two new classes of models to express counterfactual quantities that are experimentally accessible are presented, along with a sound and complete calculus (mostly based on Bareinboim's CTF calculus).

STRENGTHS AND WEAKNESSES:
[+] This paper might be very influential, as it extends the classical Pearlian hierarchy using the key concept of experimental falsifiability
[+] The counterfactuals considered by this paper include some relevant queries for applications (ex. NDE, which is important for fairness analysis)
[-] The paper is a shrunk version of a much longer paper (included in the supp-mat). The shrinking process was a bit hurried, which made the clarity of the short version of the paper somewhat poor.

QUESTIONS:
As mentioned, the supplementary material contains a long (~40 pages) version of the paper, which is very clear and well-organised. The same cannot be said for the 10-page version, which is obviously more dense, but also less precise in its organisation of the contents. A natural question is whether the authors believe they could make the conference version of their work entirely self-consistent and clearer than what appears to be now.

Example 7 is excellent to advocate the importance of the "2.5" layer. Is it possible to do something similar for "2.25"?

Does the calculus derived here make the standard CTF calculus useless?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes two novel types of causal queries, labeled $L_{2.25}$ and $L_{2.5}$, that lie between the interventional ($L_2$) and counterfactual ($L_{3}$) layers. Both can be realized using the method of counterfactual randomization. To represent the distributions associated with these intermediate layers, the paper proposes two graphical models based on CBNs with additional constraints. An inference procedure based on the ctf-calculus [Correa & Bareinboim] is developed to identify these queries, and is proven to be both sound and complete. Finally, the paper offers a detailed discussion on the expressivity and falsifiability of different layers within the causal hierarchy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-structured and clearly written overall.\n2. The two new layers in the causal hierarchy are quite novel and theoretically meaningful. Additionally, the paper includes a comprehensive appendix with detailed background and proofs.\n3. I found the examples very effective in aiding readers' understanding of these abstract ideas. Furthermore, the discussion in Section 3 (on expressivity and falsifiability) offers valuable insights from both theoretical and practical standpoints.\n\nWeaknesses:\n1. I believe more real-world examples for Layers 2.25 and 2.5 would help demonstrate their importance. I understand their theoretical significance, but how are they practically meaningful (comparable to L2 and L3)? I know additional details are given in the appendix, but it’s still worth emphasizing the importance (ubiquity) of counterfactual randomization in the main paper.\n2. Theorem 1: Def 10 in the supplementary material does not describe a constructive process. Did you mean Definition 11 here?\n3. Lemma 1: \"set CRS(W*) satisfies (i) and (ii) / An(W*) satisfies (I)\" — I'm confused about this part. What does \"(ii) / An(W*)\" mean here? \n\nMinor typos:\n\n5. footnote 2: \"B\" -> \"Appendix B\"\n6. line 200: \"constrains\" -> \"constraints\"\n\nQUESTIONS:\n1. Definition 1: is the choice of $\\mathbf{x}_i$ unique? It'll be great to clarify it here.\n2. If I understood correctly, the definitions imply that queries in $L_{2.25}$ disallow conflicting values of treatments and outcomes, and queries in $L_{2.5}$ disallow conflicting values in the outcomes?\n3. I wasn't able to find the reference for counterfactual BN (CTFBN)? Is CTFBN the same as AMWN?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-13"}