Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a counter-factual model with the assumption that the traditional exogenous variable is decomposed into independent categorical and continuous variable.
Here, only the categorical variable is governed by top-most exogenous variable, and due to the nature of categorical selection, the authors proposed clustering-based two-step counter-factual prediction model.


strengths_and_weaknesses: - The paper is well-written and well-organized. 
- Also, the quantitative results seems promising.
- The experiment is conducted two simple datasets and one real-world dataset. I would recommend conducting counter-factual image generation task on some other image dataset, perhaps CelebA, which can directly visualize and compare the effect and performance of proposed method against baselines.
- Setting $U_\eta$ as noise is a little bit unnatural. Regarding this, see questions below.
questions: - Is there any difference on the performance when using GMM instead of k-means clustering? 
- Could you explain about Figure 3 more precisely? For example, the digit in the second row & the third column is drawn by intervening the color, am I correct? What is $T'$? Considering Figure 1, $U_z$ and $T$ are not linked. However, it seems that other factors, such as rotation (and probably also the noisy-ness, but not perfectly sure due to the low-resolution of MNIST data itself), have changed after intervening the color of digit in Figure 3, which is not desirable. What causes such phenomenon? Probably, the size of $T'$?
- $U_\eta$ can be some other factors. For example, let $T$ be some other underlying natures of digit data (ex. thickness, slope), and set $U_\eta$ as rotation (since it is enforced at data generation stage). I think it also can be one example of the causality. (Please let me know if I'm missing something.) Will the intervention cause the change of rotation in this case? What was the intuition behind of setting $U_\eta$ as noise?
- With the reconstructed counter-factual images, how does the digit classification result change? (with simple classifier)
limitations: The authors have discussed the limitations and potential negative societal impact in the main paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors tackle the limitation of pre-existing neural network-based structual equation methods which only enable to model the factual distribution, not counterfactual, such that it often leads to incomplete ounterfactual distributions. The authors focus on the counterfactual inference of treatment effects on patient trjectories in healthcare setting. They validate the proposed model on three image and time-series datasets and the results show the model outperforms its beaseline in accuracy of counterfactual estimation.
strengths_and_weaknesses: The tackled goal is an interesting topic in the field of counterfactual inference and estimation. The proposed methodology is well described and the quality of writings is good.

However, it's a bit hard to understand the benefit of proposed methodology and, although interesting, it’s hard to say the introduced technique is quite new as the proposed method seems to be a simple extension from pre-existing method (structural causal model). Also, the paper lacks detailed explanation and proof for their argument that pre-existing structural causal equation models fail to provide theoretical guarantees for reconstruction of counterfacturals. Although the authors show some qualitative result (Figure 3) for supporting their argument, it's not quite convincing. It will be helpful if the authors are able to provide more results and explanations for this.

Furthermore, even though the authors argue that the proposed method provides a counterfactual reconstruction architecture on images and time series, the experimental results do not seem to be good enough to show benefits of their method. Especially for time-series experiment, more diverse qualitative and quantitative experiments should be performed.





questions: Q1) Based on the experimental results (Figure 3) with the authors' argument, it's a bit hard to understand that the lack of guarantees on pre-existing deep structural equation models tend to generate potential incorrect reconstructions. The shown results are relatively weak to support their argument. Can authors show more convincing resutls on that?

Q2) The authors need to describe how their assumption on strong ignorability works on time-series data. The authors said they perform counterfactual estimation of treatment on patient's trajectory in which the observed variable Y can be time-wisely influenced by multiple treatments or other confounding factors. Considering the fact that the patient in a hospitcal (a general ward/ICU) usually recieves multiple treatments, I wonder how the authors handle this problem.

For improvement, the authors need to describe how they address counterfactual estimation on time-series data setting.

Q3) It's hard to see the benefit of spliting Uε to a categorical variables and continuous variables. Can authors describe more on that in detail?

Q4) In the experiment on cardiovascular simulator, what kind of dataset does the authors use? They should describe more explanations about the dataset and explain  how they preprocess data when handling clinical records (such as time points for fluid intake, length of sequence) and how a training set of patients are selected from the entire clinical database (e.g., the selected patients show normal range of blood pressure? or what types of other clinical events do these patients have in their EHRs?). Also, more detailed explanations about qualitative analysis on cardiovascular experiments will be helpful for showing the effectiveness of their method. For example, how clinical meanful is the estimated patient trajectory from counterfactuals?
limitations: The authors addressed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper deals with the problem of counterfactual identification and estimation in treatment effect estimation problems from observational datasets. The task of counterfactual identification is challenging as we need access to the structural equations, which is a strong assumption in typical real-world datasets. The authors tackle these issues by making the assumption of discrete/categorical latent variables that affect the final outcome distribution and provide novel identification results under the ($\rho$, $\delta$) definition. They propose the approach CFQP, which takes inspiration from the discrete latent assumption and trains a mixture model using expectation maximization. They show the gains of the proposed approach against the existing works in the literature on semi-synthetic datasets; MNIST, harmonic oscillator, and cardiovascular simulator. 
strengths_and_weaknesses: **Strengths**

- Given that we would never observe structural causal models (SCM) for a majority of realistic datasets, the task of counterfactual identification without access to SCM is an extremely relevant problem. The paper provides identification results towards the same under the discrete latent variable assumption, which is novel to the best of my knowledge.

- The identification results are significant as the assumptions introduced in the paper for counterfactual identification are practically motivated. Many practical scenarios can have discrete latent variables affecting the outcome distribution, and additionally, there are no restrictions in terms of the dimensionality of treatment and outcome.

- The proposed approach (CFQP) has simplicity in the sense of being inspired from mixture models, and obtains substantial improvements over the prior approaches.

- The paper is overall well written with good explanations for all the assumptions and nice presentations of the empirical results.

**Weaknesses**

- The proposed approach (CFQP) required the knowledge of the true number of latent categories (K). While the authors present ablation studies when there is a mismatch with the true total latent categories, further empirical analysis might be needed. 

- The identification result holds for the ($\rho$, $\delta$) definition stated in the paper, which might not lead to perfect identification.

Please refer to the question section below for further details.

Note: I do think the paper makes good contributions and I am happy to raise my score if the authors answer my concerns with the evaluation studies.
questions: -  How does the proposed approach perform when there is a mismatch in the total number of latent categories for the harmonic oscillator and the cardiovascular dataset? The authors only show the effect of error in modeling the total latent categories for the MNIST dataset. Further, the authors should report results for the approach CFQP when they tune the total latent categories ($\hat{K}$) using validation data (instead of selecting $\hat{K}=K$) and include the same across different settings in Table 1. 

- A standard way to define counterfactual identification (https://causalai.net/r60.pdf) would be as follows: 'if two different SCM $M_1, M_2$ have the same causal graph and generate same observational/interventional distributions, then a counterfactual query Q is identifiable iff $M_1(Q) = M_2(Q)$'. How does the ($\rho$, $\delta$) definition considered by the authors compare with it? Does the identification result in the paper imply there can still be some SCMs that are not distinguishable from each other considering the proposed assumptions, but they yield different results on the counterfactual query?
limitations: I think the authors have adequately addressed the limitation and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a series of assumptions that permit identifiability in cases with categorical background variables. It further proposes CFQP a neural net based algorithm for counterfactual estimation based on the aforementioned assumptions and an EM method
strengths_and_weaknesses: - Originality : the paper appears to provide a novel and original methodology both in terms of identifiability results and NN methodology
- Clarity : adequate clarity
- Significance: Potential for impact in the wider causal ML community
- Related works : Not enough discussion of related works and how this work relates to it
- Experimentation: Only MSE and some images are shown - no accuracy for counterfactuals where ground truth exists - synthetic ones 
- Methodology: good but assumption of Gaussian distributions is not supported. See questions for more

Disclaimer: I did not check the validity of the proofs 
questions: - What is the justification of gaussian assumptions and not something like $g(u^{*}_{\eta})$ where is a learnable function and u* an approximation 
- What is the approximate complexity of the proposed NN method? How does the it scale as the number of treatments, outcomes and categories scale ? 
- What is the accuracy of the method in the cases where the ground truth is known ? in the image scenario what is the SSIM ? 
limitations: The authors have explicitly stated their assumptions and provided an ethical discussion
review_title: null

Meta-review: metareview: The authors develop a technique for estimating counterfactuals. Counterfactuals are generally not identifiable. This paper makes assumptions on the exogenous noise on the outcome to estimate counterfactuals. Namely, the paper assumes a purely exogenous continuous part and a discrete part. The reviewers were generally positive with the one negative reviewer noting in discussion that they still had concerns about novelty, but were swayed positive by the value the method could have on real world problems.

confidence: Certain
Decision: Accept