Kati8/ICLR_2023_2025 huggingface dataset:
authors: Arvind Raghavan
authors: Elias Bareinboim
reviews: 
Comments: Inspired by (Bareinboim et al., 2015), the authors introduced a new action of "counterfactual randomization (ctf-randomization)" in structural causal models. One of the examples of how such action can be applied is given in Definition 2 (Counterfactual mediator). In particular, for some variable $X$ and its children $Y$, $W$ is the counterfactual mediator of $X$ if the mechanism generating $Y$ can retrieve the value of $X$ from $W$. The authors studied under what conditions on the causal structure and set of available actions (which might include counterfactual randomization), it is feasible to generate i.i.d samples from a counterfactual distribution. This study assumes that each unit undergoes each causal mechanism only once (Assumption 3.1). The authors provide graphical conditions along with an algorithm (CTF-REALIZE) to check the feasibility of sampling (which they call this problem "realizability"). Moreover, they characterized graphical conditions for the case that it is possible to perform ctf-randomization for each child of
each variable, and as a corollary, they obtained the case of "fundamental problem of causal inference". They explored the applications of counterfactual realizability in causal fair analysis and causal bandits. In particular, they showed in an example that the interventional fairness metric fails to preserve disparities
in outcomes across groups while a counterfactual one approach works in it. Moreover, in a specific causal bandit setup, for a specific "MAB template", they showed that the proposed $\mathcal{L}_3$-based strategy is optimal.
Strengths: Originality/Quality:
The definition of ctf-randomization and the algorithm for checking the counterfactual realizability are new in the field of causality. Moreover, the authors discussed the applications of $\mathcal{L}_3$-based methods in fairness analysis and bandit problems which might be interesting for people in other fields.

Clarity: The writing should be improved as some of the parts need more explanations.

Significance: Although the concept of counterfactual realizability is new in the field, it is not completely clear to what extent such ctf-randomization can be implemented in reality.
Weaknesses: The content of the paper is fairly dense and the authors did not have enough space to give a sketch of the proofs or even CTF-REALIZE algorithm is not described very well in the paper (For instance, it is not clear why there is a rejection sampling in line 18 of Algorithm 1). This also applies to the results in Section 4. Although the authors showed in some specific scenarios, $\mathcal{L}_3$-based methods perform better than $\mathcal{L}_2$-based methods, there is no intuition given in the paper why this is the case. One might wonder if the causal graphs or the causal mechanism are meticulously engineered to show the advantage of counterfactual reasoning.
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The authors provide necessary and sufficient conditions for a ground truth counterfactual quantity of an SCM to be sampled directly (''realized'') through a ''counterfactual experiment''. Such an experiment involves manipulating the original SCM through adding mediators through some direct causal effects---a process the authors motivate with intuitive real-world examples.
Strengths: The authors' contributions are original and significant. Their insights are laid out clearly and limitations and discussion of future work are provided. 

The authors show the advantages of counterfactual experiments with intuitive examples. The authors show that counterfactual fairness measures that are not identified through interventions can be realized (and thus identified) through counterfactual experiments. Furthermore, the authors show that bandit policies which allow for counterfactual mediators dominate policies from the classical causal bandits setting.
Weaknesses: I do have few substantial weaknesses to discuss. In the Questions section I voice a slight disagreement about the interpretation of ''physically possible'' experiments. Below are some minor comments.

(Lines 105--107) Presumably W_* is a tuple of potential outcomes and w is a vector over the support of W_*. These should be defined formally to avoid confusion. I suspect there is a typo on line 107 when defining P^M---perhaps taking the product over t = 1 to |W_*| would clear this up. 

(Lines 120--121) Read (V)^(i) does not ''measure'' the effect of f_V on V, these mechanisms are in general not identified. This should be reworded---perhaps the physical actions should be defined formally.

(Figure 2) Given the discussion in Example 1, it may be more natural to include latent confounding between X and Z in the DAGs.
Recommendation: 10
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The authors tackle the complex space of sampling from counterfactuals. 

Specifically, they propose a "formal definition of realizability, the ability to draw samples from a distribution, and then developing a complete algorithm to determine whether an arbitrary counterfactual distribution is realizable given fundamental physical constraints, such as the inability to go back in time and subject the same unit to a different experimental condition"

Both causal fairness and CRL are used as illustrative examples, suggesting that a counterfactual strategy dominates strategies based only on interventional and observational data.
Strengths: The paper tackles a relevant problem, and covers all key bases to provide its key contribution. 

Sampling from counterfactual distributions is naturally hard, and developing methods and tools to do so is important, as suggested in the paper.
Weaknesses: I am not an expert in this field, so take it with a grain of salt:

As written by the authors "representing counterfactual distri- butions, is believed to be inaccessible almost by definition."

References are provided for e.g. the 'counterfactual randomisation procedure', i.e. attempts at sampling from CFs, but there is not much discussion about why it is 'believed' why CFs are inaccessible by definition.

"inaccessible almost by definition" is not precise enough to be useful for a reader, "almost" needs to be refined and explicated, ideally both via intuition and formally, even just briefly.

In the conclusion, this rigour is missing, too, when the authors state: "Countering prevalent belief". A reader needs references to be able to follow the thoughts and verify by themselves, that indeed, the venture of CFs sampling is hard (or impossible?)

If CFs are not in fact ""inaccessible almost by definition"", then this paper of course would constitute a huge contribution to the field of causal inference in which case this should be highlighted more.
Recommendation: 6
Reviewer confidence: 2
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This paper considers a simple procedure that makes it possible to sample certain counterfactual distributions (counterfactual realizability). The key observation is that the perception of a variable to its children can be intervened upon in some cases. This procedure is called counterfactual randomization. It also proposes an algorithm that takes in a candidate counterfactual distirbution and a graph, and outputs a sample from the counterfactual query if and only if it is possible to sample from the input candidate. Finally, a graphical characterization for counterfactual realizability is given.
Strengths: As the paper points out, the counterfactual randomziation procedure in itself has been alluded to in previous work. But, the main contribution of this paper is a characterization of when a counterfactual distribution can be 'realized'. I liked that the paper operates under this structure for sampling distributions at different levels of the causal hierarchy by the notion of actions. Since counterfactuals are commonplace in many applications and a major hurdle is their non-identifiabilty, this work is certainly significant. The paper is well written and padded with many examples to illustrate the applicability and importance of the procedure.
Weaknesses: As the paper also mentions, I believe the idea for intervening on perceptions was known before and makes multiple informal appearances. A few other references are (S. Geneletti, A.P. David, Defining and identifying the effect of treatment on the treated, 2010). Also, the notion of SWIGs (Richardson and Robins, 2013) is quite similar. I believe a more thorough comparison with previous work is needed to make the paper more complete.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3