Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes some metrics to evaluate counterfactual generation models, based on Galles and Pearl's axiomatization of counterfactuals that are interpreted according to (a certain class of) structural causal models. Corresponding to each of the three axioms -- effectiveness, composition, and reversibility -- a metric (family) is developed in the paper. Using these metrics, the authors examine the empirical behaviors and performance of some popular generative models, and demonstrate in experiments the potential of these metrics for indicating the quality of counterfactual generation. The extent to which simulated intervention can be used to mitigate confounding is also explored in the experiments. 
strength_and_weaknesses: Strengths:

1. The central idea of designing evaluation metrics based on Galles and Pearl's axioms is novel. 
2. The development of the metrics is well argued and explained.
3. The empirical studies are interesting and seem to provide good evidence for the potential usefulness of the proposed metrics.
4. The demonstration of the utility of simulated intervention is clever and fairly convincing.

Weaknesses:

1. Galles and Pearl's axioms are proposed to characterize a certain class of structural causal models, i.e., those models that imply that the endogenous variables are fully determined by the exogenous variables (including the exogenous noises). This class is on the one hand more general than the usually assumed recursive models and on the other hand quite special among non-recursive models. This point is only alluded to but not made sufficiently clear in the paper. It should be explicitly discussed to properly qualify the intended applications of the proposed metrics. In particular, if more general feedback mechanisms that allow multiple equilibria are of interest, composition and reversibility will need to be weakened to accommodate them. 
2. It is not clear what "counterfactual identifiability" exactly means in this paper. If it means identifiability of counterfactual queries from observational data (or perhaps also interventional data?), I don't see how the soundness and completeness results imply that "composition, effectiveness and reversibility are necessary and sufficient for counterfactual identifiability." It is possible to have a causal model that does not satisfy some of these properties (because it is outside of the class mentioned above), but still allows some counterfactuals to be identified, isn't it? Conversely, when data underdetermine the underlying causal model, even if all the possible candidates are assumed to satisfy these properties, counterfactuals may still not be identified. It seems to me that either the authors intend to use the term "counterfactual identifiability" in a very special sense, or there is some misunderstanding of what the completeness theorem amounts to.
3. The proposed metric for reversibility seems to be much more about the invertibility of a causal mechanism than about reversibility. In particular, I don't see exactly how the proposed metric derives from applying (something like) reversibility to "a twin network conception of the SCM." If it is really just a metric for invertibility, it is doubtful whether it is appropriate to require a counterfactual inference model to respect it (because invertibility may be too strong a requirement in many contexts). 
4. The proposed metric for effectiveness is apparently a metric to evaluate a combination of a counterfactual inference model and a pseudo-oracle function, rather than of the counterfactual inference model by itself. It is hence a little unclear in principle whether it is suitable for evaluating the latter. 
clarity,_quality,_novelty_and_reproducibility: I am quite impressed by the novelty and technical quality of the paper. Overall it is also clearly written, but clarity is a little lacking regarding those conceptual issues I listed above. I don't see any reason to doubt the reproducibility of the empirical results.
summary_of_the_review: This is a novel and interesting paper with quite a few merits. But I am also puzzled by several conceptual aspects of the paper, as indicated in my comments on the weaknesses. (I will raise the score if those comments prove to be my confusions in the subsequent discussions.)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes an approach to learn counterfactual functions based on the properties that the axiomatic definition of counterfactuals require (composition, reversibility, effectiveness). Evaluating the degree to which this properties are satisfied allows to assess the distance between an ideal counterfactual and its approximation.
strength_and_weaknesses: + The paper considers a relevant problem from an original perspective
+ The proposed framework is grounded in SEM theory
+ The framework is applied in representative experiments and datasets.
+ Experiments, ranging from simpler to more realistic examples are well designed, discussed and commented, providing an illustration of the application of the author's frameworks.

- It may have been interesting to have a short discussion (maybe in the Appendix) on the choice of metrics.
- If feasible, comparison with other counterfactual deep models would have been instructive.
clarity,_quality,_novelty_and_reproducibility: The paper is clear (explaining thoroughly the reasoning from the definition of conterfactuals to counterfactual functions), and of high quality. To the best of my knowledge, reliance on axiomatic properties for learning counterfactuals is novel. Experiments are well presented and documented in the appendix.
summary_of_the_review: The paper offers a good contribution which may be of interest for anyone working on causality and causal learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a method that evaluates counterfactuals.  The method does not focus on the algorithm (or models) for the inference of counterfactuals but on properties the inferred outputs must obey. Based on Pearl’s axiomatic definition of counterfactuals, the authors propose to check three properties: Effectiveness, Composition, and Reversibility. Experimental results demonstrate the utility of the proposed evaluation method. 
strength_and_weaknesses: Strong points:
As mentioned by the authors, there is little methodology available to assess the quality of inferred counterfactuals. The reviewer believes the assessment of counterfactuals is very important and methodologies of the counterfactuals assessment should be discussed more vigorously. The idea to focus on the properties the outputs must obey is novel and the list of the three properties spearheads research on counterfactual inference.


Weak points:
It should be more discussed on the properties the distance metric, dx, shown in Equations (1) and (2) should satisfy. It is sure that distance is equal to zero if the counterfactual function is ideal. In case the counterfactual function is not ideal, the distance would be larger than zero and the inference quality is measured by the distance metric quantitatively. The reviewer believes more discussion on the distance between given two images is needed.

For the check of Effectiveness, the method uses a pseudo-oracle function that infers the parents of a variable. An image generative model often generates artificial images that do not obey the probability density distribution of (real) images. When a generated counterfactual image deviates from the distribution of real images, the reviewer thinks, the pseudo-oracle function should not output any parents of a given image and should degrades the rating of Effectiveness.
clarity,_quality,_novelty_and_reproducibility: The paper is well organized and the proposition is novel.
summary_of_the_review: The reviewer recommends to accept this paper. The reviewer believes the evaluation of counterfactuals is an important research topic and is worth discussed in the conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Counterfactual modeling provides a conceptual basis for understanding image modifications. Given the degrees of freedom in such a task, it is relevant to go further on the idea, tapping not only on off-the-shelf causal concepts but specializing them for the task at hand. This paper provides conceptual advances on how to integrate ideas from deep generative models and counterfactual modeling.

Strengths: it provides original foundational advances on the problem of image generating by tapping into its own take of counterfactual modeling.

Weaknesses: there are still degrees of freedom to be decided, and further discussion to which extent just throwing more data at the problem would suffice to fill up some of these gaps.
justification_for_why_not_higher_score: There are still degrees of freedom to be decided, and further discussion to which extent just throwing more data at the problem would suffice to fill up some of these gaps.
justification_for_why_not_lower_score: The paper provides original foundational advances on the problem of image generating by tapping into its own take of counterfactual modeling. It's not only originality for its own sake, but the problem targeted is of particular relevance to ICLR.
Decision: Accept: poster