Venue: CLeaR 2022 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper presents a method that transforms images from one class to another based on diffusion models.
Main review: Summary
=======

I would characterize this work as an extension of generative image diffusion models where, instead of generating images from one class, we input an image of class A and want to generate an output image of class B. This seems (?) to be motivated by a desire to connect this class of models to counterfactual estimation and causal inference in structural causal models (SCM). However, to me the task appears more similar to what has previously been described as "counterfactual explanation" (CE) in this context (e.g. Schut et al, 2021): change an image in such a way that it retains features of the original image, remains interpretable and realistic, but will be accepted as a member of the new class. The authors show experimental results on MNIST that seem to indicate that this method outperforms Schut et al and another baseline (Van Looveren and Klaise 2021). 

The algorithm essentially operates in two steps:
(1) gradually add noise to an input image from class A using a diffusion model (forward diffusion) until we have reached the point where the image consists "only" of noise;
(2) apply a (reverse) diffusion model to the result of (1) to generate an output of class B. 

Interestingly, the "noise" produced in step (1) seems to retain some features of the original image, as the resulting image indeed appears to be "close" to the input class A image in many cases. This at least to me was somewhat unexpected, since I do not see where the method is particularly designed to retain input information in this way, but I found this very interesting to observe nevertheless. However, the qualitative results shown on the ImageNet data suggest that this technique no longer works reasonably in this more complex setting, as the resulting images would not be recognizable as members of the desired target class (they would be unrealistic counterfactual explanations). An alternative explanation of these results, which I would like to see the authors discuss, would be that step (1) is actually not successful in the sense that it does not achieve the stated goal of "recovering the exogenous noise".

Pros
===

Originality: I found this to be an interesting and thought-provoking idea, with some partly unexpected results. 
Quality: The algorithm is well developed and seems to be performing well at least on the MNIST data. This indicates that there is something to gain by employing diffusion models in this context of counterfactual explanation. 

Cons
====

Quality/Clarity: I have doubts about the stated connection to counterfactual prediction in structural causal models (SCMs). The introduction starts with some general ideas about how SCMs are defined, and how we could also use SCMs to represent (networks of) stochastic diffusion equations. While I can follow these general ideas, I do not see how they are connected to the algorithm presented in Algorithm 1, which appears to be much more close to the CE problem as discussed in Schut et al. It appears that Proposition 1 is where the authors are trying to make this connection. However I don't follow their reasoning at that critical point. How is it obvious that computing a counterfactual probability is equivalent to solving this reverse diffusion process? If this works for any structural causal model, as you seem to claim, then can you perhaps give a "toy problem" of a simple binary counterfactual quantity in a simple SCM, like one of the examples given in Pearl's "Primer"? The example below the proposition only adds to the confusion. Here the specific SCM x2 -> x1 is discussed where presumably, x2 represents the source class of an image and x1 the target class. In this specific case (which is the setting of the paper) we have P( x1 | do(x2) ) = P( x1 | x2 ), so there is no difference between the causal and observational distributions. Then why use causal language in the first place? Adding further to the confusion, the authors write "The causal effect of an intervention can, therefore, be written as p(x(1) | do(x(2) = x(2)))" -- why "therefore"? This is simply the notation that is used for causal effects, which would be valid in any model structure. Given these inaccuracies and omissions, I am currently not convinced that there is actually a connection between Algorithm 1 and counterfactuals in SCM. It rather appears that the authors are simply labelling their method as "counterfactual predictions" without really establishing carefully and in detail how these two concepts relate. To build their case for this connection between SCM and diffusion models, the authors also cite various papers e.g. by Mooij and Blom, but the specific connections made in those papers are very different from what is being done here and I don't see why this would be immediately applicable. I feel it would be important to clarify this, or alternatively revert to the much clearer concept of "counterfactual explanation" as used e.g. in Schut et al, which doesn't attempt (or need) to make the connection to SCMs.

Significance: Although it is reassuring that the method seems to slightly outperform the baselines, the theoretical foundations do not give me much confidence that we should expect this to be the case, given that the baseline methods are directly designed to solve the desired problem of counterfactual explanation. I can't therefore judge very well at this point how useful or influential this method will turn out to be. 

Other comments
============

In the empirical evaluation, the results you show for Schut et al. seem to be worse than what they show in their paper, why? 

It is unclear why you are mentioning conditional ignorability in this context. 

All endogenous variables in the training set are annotated but labels at inference time are not required” – could you clarify what you mean with the second part of that sentence?

p6 “...following Dhariwal and Nichol (2021).” This is a bit vague; what aspects of their work are you following? What did they do, and how is your work the same or different?

Results could use some more context. For example:
a. Could you explain briefly how CLD is different from IM1 and IM2?
b. Table 1 page 9: what are the subscripts?
c. What is the l1 distance (p10)?
d. Small note: abbreviation VAE is used in figure 3 before it is 
introduced on page 11.
e. P11: what is E_phi?
f. Could you put the results into context? E.g. the results in Figure 3 are better than other methods, but what currently are still the weaknesses that future versions should improve on?
g. Related: in Figure 1, the method is applied to ImageNet – but this is not described in the results at all. I get that previous methods could not handle this input so that no direct comparison is possible, but again, I would have liked to see a discussion of how well the method performs. For example, while do(cliff) seems to have worked reasonably well, the  same cannot be said for the espresso maker and the waffle 
iron.
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: Using recent advances in latent variable modeling for better counterfactual image generation.
Main review: The paper modifies the conditional sampling DDIM procedure of [Dhariwal and Nichol, 2021] to generate counterfactuals Pr(x=x_cf | do(y), x=x_f). The key addition over the DDIM+conditional sampling procedure is to extract a suitable latent encoding of x_f which is then shared with the generation of x_cf. Diffusion models run in the forward direction yield such an encoding, and conditional sampling (diffusion models run in reverse direction, along with a classifier to bias towards interventional distribution do(y)) yields impressive counterfactual image generation results on MNIST and ImageNet. 

The comments below focuses only on the weaknesses of the paper.

- Eqn1: Please clarify that there are several simplifications to reach the objective form in Eqn1. For instance, there is a weighting term that Ho et al 2020 empirically find to not be significant; there is a reparametrization trick in going from ||x_t - e_theta(...)|| to ||e_theta(...) - epsilon||, etc.

- How are e_theta and p_phi fitted to data? Does the training data need additional annotations (e.g., a suitable prior on the latent u^(k) to ensure separation of factual and counterfactual latent encodings)?

- Baselines: Dhariwal and Nichol, 2021 should be included as a baseline. The only change from Algorithm 1 is that they would set u^(k) from N(0,I) rather than the abduction procedure from x_F. 

- Additional experiment: Can you please clarify why Schut et al and van Looveren et al are sufficient baselines? At first glance, the approaches to controllable image generation, which learn disentangled latent spaces, seem to be immediately relevant. Rather than building an anti-causal predictor for interventions, they would map interventions to operations on the latent space. Why not compare to controllable generation via latent variable models that allow us to intervene on the latent space?

- Additional experiment: All the experiments are focusing on an image generation example (and a restricted 2 variable example). Illustrating the general approach of Algorithm 1 empirically on even a toy SCM will be very informative.

- Clarity: Section 3.2 begins with a general framing w.r.t SCMs but abruptly goes to the 2 variable case. It may be useful to include a discussion before/after Section 3.4 that discusses the requirements for running Diff-SCM. For instance, what should the data contain. What models need to be learned. How to set T, s, {beta_t}, etc. My understanding is as follows: Given an SCM, Diff-SCM will learn {e_theta(x)} for every node in the SCM x (if we pre-specify that we only care about counterfactual outcomes of a subset of nodes, then e_theta models will only be learned for those nodes). For every e_theta(x_k), Diff-SCM will learn an anti-causal predictor P(x_j | x_k) for every other node in the SCM (if we pre-specify that we only care about interventions on some subset of nodes, then only those anti-causal predictors are learned). This approach crucially needs to know the SCM structure (i.e., corresponding causal graph). There may also be some situations where this approach is warranted (even when we know the SCM) rather than modeling the SCM mechanisms f^(k) directly [e.g. pa(x_k) are unobserved in the data but x_j \in ancestor(x_k) is]. However, the paper does not discuss this, nor the computational and statistical issues that may be present in learning the many encoders and classifiers.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: Summary: In the paper « Diffusion Causal Models for Counterfactual Estimation », the authors propose a generative diffusion model making it possible to model the effects of counterfactual interventions on a target variable. The generative diffusion model is seen from the point of view of a functional causal model where each variable can be expressed as a function of its parents in the causal graph and a random noise. In order to model the intervention, an anti-causal model is learned. Once learned the gradient of this anti-causal model is integrated in the generation model in order to take into account the effects of the intervention. The influence of the intervention can be regulated with a hyperparameter s. In the experimental section, the authors study a causal model with two variables X1 and X2 and evaluate the response of the model when the value of the hyperparameter s varies. A distance called CLD is introduced to measure a trade-off between the sample reproduction of the factual class and the counterfactual class. This distance is measured as a Kullback–Leibler divergence in the latent space of an autoencoder. For the dataset MNIST, the authors show that a specific value of 0.7 can achieve the best tradeoff between the reproduction of the factual class while taking into account the counterfactual intervention.
Main review: 1) The paper is well written and fairly clear.

2) The generative diffusion model used in the article seems to be really powerful for modeling distributions. However, in this article it is not clear to me how this kind of conditional model could actually be used for real causal applications.

3) The review of the literature seems quite exhaustive with many interesting and recent references on the subject. However, I am concerned about the originality of the proposed approach as the model appears to be almost exactly the same as the model described in (Dhariwal and Nichol 2021), simply reinterpreting the conditional diffusion generative model with the language of causality. Is there anything different between the algorithm 1 of this paper and the algorithm proposed in (Dhariwal and Nichol 2021) in section 4, except that y is replaced by the parents of x in the causal graph ?

4) I understand however that the authors are proposing a more general model than (Dhariwal and Nichol 2021) because it can be extended to a complex causal model with more than two variables. However in the experimental section of this paper, the authors consider only a two variables causal model x2 → x1. So, is there something really new ? For example, the images shown in Figure 1 could also have been generated with the conditional diffusion generative model in Dhariwal and Nichol 2021, by setting y = cliff, y = expresso maker, or is there something that I don’t understand ?

5) Introducing the counterfactual latent divergence (CLD) distance seems interesting. The authors show that the factor s has an influence on the generated samples and allows to control the distance with the factual and counterfactual classes.

In general, although the paper is well written and well documented, I am not convinced that it really adds anything new from a theoretical and experimental point of view.

Référence :
Dhariwal, P., & Nichol, A. (2021). Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233.
review_title: null

Meta-review: The paper proposes to use a class of generative models (namely diffusion models) for counterfactual estimation from images. The difference from the existing diffusion models is that the proposed method allows to sample from the counterfactual distribution to allow the resulting image (i.e., result of the performing counterfactuals) to match the other aspects of the original image and hence only changing the target aspect. All the reviewers like the proposed idea. The paper validates the proposed idea by performing experiments on the MNIST dataset with two variables. Even though the results are on a toy dataset, the area chair believes that the underlying could be valuable to the causal community. I encourage the authors to include a thorough discussion of the limitations of the proposed method.
Decision: Accept (Poster)