Venue: UAI 2022 Workshop CRL

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Summary**: This paper introduces deep twin networks, which allow for inferring certain counterfactual quantities by training a neural network fitted to a twin network structure. As shown in Sec. 3.2, counterfactual quantities are often not identifiable from observational and interventional data even in settings without unobserved confounders. This paper therefore considers a more restricted setting under the assumption of counterfactual ordering, which they prove is equivalent to monotonicity of the mechanisms. The model is tested empirically in the backdoor setting with ignorability conditions and is shown to have good performance under this assumption.

**Review**: This paper presents interesting ideas in introducing a deep learning approach to incorporate the well-studied properties of twin networks in counterfactual inference. The issue of identifiability, which dramatically limits the ability to perform counterfactual inference in practice, has clearly been considered, and additional assumptions in the form of monotonicity are clearly stated. The connection made between counterfactual ordering and monotonicity is quite subtle but is an interesting contribution of this work. While monotonicity is a property of the true functions/mechanisms, counterfactual ordering is a property of the induced distributions, and Thm. 2 proves that they are linked.

Still, there are a few aspects of the paper that could be clarified. First, although identifiability is considered, it is not clear why counterfactual ordering alleviates this issue. I would expect to see a theorem stating something along the lines of “Under the assumption that counterfactual ordering holds, queries of the form X are identifiable given causal diagram and data Y.” Second, although there is a paragraph explaining how monotonicity might be enforced, it is not explained how this is done in the experiments. Neither code nor pseudocode is provided to clarify this. Third, a line at the top of page 5 states that deep twin networks can “learn the underlying functions between causal variables”, but this seems to be a grandiose claim. What makes the trained deep twin network capable of modeling the underlying functions as opposed to simply matching the counterfactual distributions? Finally, there seems to be some additional proofreading required. For instance, I noticed that Thm. E is mentioned in several locations but does not exist (I am assuming it is referring to Thm. 2?).

For a full conference publication, I would expect the above issues to be clarified. Further, I would be very interested in seeing how the theoretical results on counterfactual ordering extend to other graphs with more variables (possibly with unobserved confounding). However, the ideas presented in this paper are certainly interesting and should be accepted for publication in this workshop.
review_title: An interesting study on the impact of monotonicity on induced counterfactual distributions.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the problem of identifying counterfactual probabilities from the combination of the observational and interventional data, and a causal diagram encoding structural constraints about the underlying SCM. The authors also assume the "counterfactual ordering", which is a generalization of the monotonicity for non-binary categorical domains. The proposed estimation method is relatively clear. It first trains a neural causal model that is capable of generating the provided observational and interventional data. The algorithm then computes the target counterfactual probability in the trained causal model, following the three-step method including abduction, action, and prediction.

The organization of the paper is relatively clear. The proposed "counterfactual ordering" assumption is interesting. However, I do have concerns with the soundness of the proposed algorithm. It appears to me that the proposed algorithm is able to recover the actual counterfactual probability if the target counterfactual is identifiable. It is unclear to me whether it is always the case in any causal diagram, provided with the counterfactual ordering. If it is not the case, there is not guaranteed that the output of the algorithm recovers any property of the underlying SCM, i.e., it is, at best, an educated guess. I think this paper could be most improved with an algorithm or a proposition showing whether/when the target counterfactual is identifiable. A sufficient identification condition is also encourged.
review_title: Review for Paper#3

final_score: rating: 7: Good paper, accept
review_content: Reviewer summary:
The authors propose a neural network architecture called deep twin networks to estimate counterfactuals based on the idea of twin network counterfactual inference, an alternative to the abduction-action-prediction framework. To avoid parametric assumptions, they introduce the idea of counterfactual ordering and equate it to a definition of monotonicity. Using these results, the neural network architecture is introduced. The neural networks respect the causal graph, learn a function from a simple distribution to the noise distribution of the variables in the causal graph, and output the potential outcomes. The counterfactual ordering condition is enforced via the loss function (given its equivalence with monotonicity). The authors test the proposed method in semi-synthetic and synthetic datasets.

Comments:
- Most parts of the paper are easy to follow, and the proposed architecture is intuitive. 
- The use of twin-network inference in conjunction with neural networks is original, as far as I know.
- Definition 1 (Counterfactual ordering) is difficult to parse. I believe you are missing an "if" in "[...]$\\{y_0, y_1, \dots, y_M\\}$ such that *if* $P(Y_{x_{i}}=y_k)$[...]". In any case, consider writing it differently to ease understanding.
- It seems the strength of the proposed architecture comes from the twin network type of inference. However, from my point of view, neural networks shine when using them in the context of high-dimensional data. I feel the proposed architecture is not suitable for such high-dimensional settings, given that one needs to enforce the causal graph into the architecture. For example, one could potentially use alternative models with similar approximation power in lower dimensions, such as regression trees, GPs, etc. and achieve similar results. The only high-dimensional scenario I can imagine where using NN and we enforce the causal graph architecture is one where we have high-dimensional controls, and we know there is no bad control there.

Potential extensions:
- What would it take to extend this work to have continuous $X$ or continuous $Y$? 

review_title: An interesting alternative of counterfactual estimation using twin-network inference

Meta-review: metareview: The reviewers generally agree that the paper is interesting. However, two reviewers have remarked that theoretical results formally showing identifiability of the target query are lacking and would be an important aspect to include if possible. Still, the overall reception was positive, and I thus recommend acceptance for poster presentation at the workshop.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)