Title: Conditional Generative Models are Sufficient to Sample from Any Causal  Effect Estimand
Venue: NeurIPS 2024
Accepted: yes
Keywords: causal inference

causal graphs

deep generative models
Abstract: Causal inference from observational data plays critical role in many applications in trustworthy machine learning.
While sound and complete algorithms exist to compute causal effects, many of them assume access to conditional likelihoods,
 which is difficult to estimate for high-dimensional (particularly image) data. Researchers have alleviated this issue by simulating causal relations with neural models. However, when we have high-dimensional variables in the causal graph along with some unobserved confounders, no existing work can effectively sample from the un/conditional interventional distributions. In this work, we show how to sample from any identifiable interventional distribution given an arbitrary causal graph through a sequence of push-forward computations of conditional generative models, such as diffusion models. Our proposed algorithm follows the recursive steps of the existing likelihood-based identification algorithms to train a set of feed-forward models, and connect them in a specific way to sample from the desired distribution. We conduct experiments on a Colored MNIST dataset having both the treatment ($X$) and the target variables ($Y$) as images and sample from $P(y|do(x))$. Our algorithm also enables us to conduct a causal analysis to evaluate spurious correlations among input features of generative models pre-trained on the CelebA dataset. Finally, we generate high-dimensional interventional samples from the MIMIC-CXR dataset involving text and image variables.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper leverages state-of-the-art conditional generative models and algorithms from causal do calculus to perform \"approximately correct\" high-dimensional interventional sampling. Their contribution is ID-GEN, a recursive algorithm that uses diffusion models (among other generative models) to sample from any identifiable causal effect estimand in high-dimensional settings. The efficacy of the method is demonstrated on three diverse datasets: Colored MNIST, CelebA, and MIMIC-CXR.\n\nSTRENGTHS:\n- The paper introduces ID-GEN, a novel algorithm that integrates diffusion models with causal inference for high-dimensional interventional sampling.\n- ID-GEN creatively exploits the causal structure via the known recursive algorithm for sampling in complex causal scenarios, particularly with latent confounders.\n- The algorithm is applied in three applications across diverse datasets (Colored MNIST, CelebA, MIMIC-CXR).\n\nWEAKNESSES:\n- The paper is objectively hard to read. Many important graphical elements that should be paired with the text in the main paper are delayed to supplements. This is notably problematic for Example 4.1, where one would expect the full example to be self-contained in the main text. \n- The contributions are stated in the introduction, but it still seems hard to understand if the proposed method is \"just\" an implementation of the ID algorithm, replacing probability tables by samples from diffusion models. I appreciate that this is hard already, but it has much lower novelty then proposing a new recursive algorithm. This should be well explained in the manuscript. \n- The paper does not discuss the implications of the proposed algorithm. Is there any way to extend this to symbolic calculus, or to probabilistic programming? What are the obstacles for moving towards automatic causal inference with images (which would be a super exciting prospect).\n\nQUESTIONS:\n- Can you clarify whether the proposed ID-GEN algorithm is primarily an implementation of the existing ID algorithm with diffusion models substituting for probability tables, or does it introduce fundamentally new recursive methodologies? How does this distinction affect the perceived novelty of your contribution?\n- What are the potential extensions of your algorithm to areas like symbolic calculus or probabilistic programming?\n- Are there any specific obstacles that need to be overcome to advance towards automatic causal inference using high-dimensional data such as images? How feasible is this prospect in the near future?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an algorithm for sampling from intervention distributions under known causal models with hidden confounders, using conditional generative models.\nThis allows nonparametric distributional estimation of high-dimensional outcomes such as X-ray image generation, unlike existing methods.\nThe proposed method combines the existing ID algorithm with a generative model and inherits the ID algorithm's theoretical guarantees. That is, non-identifiable quantities are indicated to be non-identifiable, while all identifiable quantities can be estimated.\n\nSTRENGTHS:\n* They shed light on the new problem setting of causal simulation for outcomes with high-dimensional and multimodal distributions, such as image generation. This could open up new applications if well justified. Such a problem setting requires a very different approach to point estimation of expectations for low-dimensional, unimodal distributions.\n* The theoretical background is solid and well-explained. The method is proved to be able to estimate all identifiable quantities and otherwise outputs \"unidentifiable.\"\n\nWEAKNESSES:\n[W1] The motivation for high-dimensional distribution estimation is weak. For example, it does not seem very meaningful to me to generate synthetic X-ray images.\n\n[W2] In particular, is it important in cases where there are bidirectional edges due to the presence of hidden confounding factors, but where the causal orientations are all identified among variables? A clear comparison with similar methods would be beneficial for readers, e.g., comparison in assumptions and targets (e.g., parametric/nonparametric, latent confounder, distributional estimation, etc.).\n\n[W3] The base procedure seems to come from existing methods, such as the ID algorithm, and they just combine it with a generative model.\n\nQUESTIONS:\n[Q1] Related to W2, is the proposed method a non-trivial novelty in situations where there are no unobserved confounding variables, i.e. no bidirectional edges?\n\n[Q2] Related to W3, are there any non-trivial points in theoretical guarantees when the generative model is combined with the ID algorithm?\n\nLIMITATIONS:\nThe method is limited to the cases where the causal direction is known.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of sampling from an interventional distribution of high-dimensional data, where the causal relationships are described by a acyclic directed mixed graph (ADMG). Motivated by the ID algorithm that provides a recursive way of identifying any causal effect from a conditional probability table, the authors propose ID-GEN that follows the recursive steps of ID but instead trains deep generative models to fit the conditional distributions. The final sampling model is then obtained by connecting all the trained networks together in some suitable way. The authors prove that ID-GEN is sound and complete, and run extensive experiments on both synthetic and real dataset to demonstrate the effectiveness of their approach.\n\nSTRENGTHS:\n1. This paper studies the problem of sampling from an interventional distribution, an arguably important problem with broad applications. Current approaches, as the authors point out, are either restricted to simple causal graphs or face computational challenges.\n\n2. Most parts of the paper are clearly written, and sufficient explanations are provided for the key steps in the ID-GEN algorithm. Also, simple examples are provided that help with the understanding of the paper. The paper is also well-organized and the authors put most complicated details into the Appendix.\n\n3. The authors conduct extensive experiments to demonstrate the superior performance of their model, by comparing with other sampling models proposed by previous works.\n\nWEAKNESSES:\nI don't think this paper has obvious weaknesses. One thing that the authors may wish to improve is that the notations are a litle bit complex; and it would be better to more often remind the authors of their meanings.\n\nQUESTIONS:\n1. In some identification formula e.g. Eq.(1) in the paper, the probability on the denominator might be small. To what extent would this affect the stablity of the proposed algorithm.\n\n2. Can your algorithm be straightforwardly adapted to the case of imperfect interventions i.e. some conditional probabilities in the structural causal model are modified, but no causal edges are removed?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an algorithm for sampling from a causal interventional distribution using conditional generative models, building on Shpitser and Pearl's ID algorithm. They discuss how their algorithm, ID-GEN, can sample from any identifiable distribution given a causal graph, and handles the presence of unobserved confounders (when identifiable). Empirically, they demonstrate their method can work for measurement, evaluation and interpretability purposes in the challenging setting where both the target and treatment are high dimensional e.g. images.\n\nSTRENGTHS:\n- interesting work, the case of high-dimensional variables in a causal graph is a super important and under-discussed one\n- thorough theoretical treatment of the extension of the ID algorithm\n- experiments show a nice range of usages of the suggested ID-GEN approach\n\nWEAKNESSES:\n- I frankly got a bit lost in a few key parts of Section 3. I got the main ideas (I think) but missed a bunch of nuance. Some spots were: Example 4.1 (I don’t understand why ID fails in this case but ID-GEN succeeds - specifically the importance of merging is a bit lost on me), Step 4 of ID-GEN (again, merging), and Step 7 of ID-GEN (I think the logic around how training data is sampled, used and modified wrt the graph needs to be explained more clearly)\n- Step 1 of ID-GEN confuses me - I don’t see why we can’t just learn a model of P(y) directly in this case? Also the 2nd equality in 203 doesn’t make sense to me - how is the sum over values of v equal to P(y)?\n- in each experimental section I find I have at least a medium-sized point of confusion around the setup or evaluation - more care should be taken to explain empirical setup + results overall\n- In 5.1, the authors state that U_color makes W1 and Y correlated by color - however, X contains color information and is a direct ancestor of Y, so this unobserved confounding seems trivial\n- in 5.1, it seems like a better metric than a classifier (which may be unreliable and as you note isn’t useful for all possible images) would be something based specifically on the RGB values of the pixels themselves\n- in 5.2, I don’t quite see why Young & Male have unobserved confounding - are they not fully determined by the shared + observed parent I_1?\n- in 5.3, I don’t understand why the report is a causal ancestor in the graph - isn’t it generated upon viewing the X-ray?\n- in 5.3, I think the setup with the labeller can be made clearer - how good is this labeller? How is it structured? Additionally, is the bottom row intended to be a success or failure examples? (label says it should be right lung base but all inferences name the left lung)\n\n\nSmaller points:\n- L115: are unobserved confounders only allowed to affect 2 variables in this framework? Is that more limiting than general SCMs?\n\nQUESTIONS:\n- would be great to see clarification throughout Section 3, particularly in the highlighted areas and around merging and training data sampling\n- experimental setups all need clarifications\n- generally assuming I understood what's happening better I'd be happy to increase my score\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}