Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The purpose of this paper is to provide statistical method to handle the so-called non-targeted trials in which one can’t control the selection of treatment units. Therefore, “some (often most) treated subjects are healthy (p.2)”, and this greatly confounds the estimation of ATT. This paper provides asymptotically consistent algorithm for accurately estimating the treatment effect, and demonstrate its value numerically with synthetic data.
strength_and_weaknesses: - A stronger justification of the practical significance of non-targeted trials could be greatly beneficial. In particular, although self-selection is indeed a well-known in e.g. natural experiments on digital platforms, it is a bit difficult for me to image self-selection problem would occur in clinical trials, where experimental units are recruited through a careful and rigorous process where doctors are usually involved. Therefore, perhaps a concrete clinical situation should be carefully elaborated, or relevant literature should be cited in the paper to substantiate the importance of this somewhat non-canonical situation. It is also important to show the situation of non-targeted trial is sufficiently common, so that any new method being developed for this situation is worth much attention.
- The algorithm in Section 2 looks like a algorithm for estimating heterogeneous treatment effect, but HTE is fundamentally different from the clinical sick/healthy subjects setting because usually it is pretty clear whether a patient is sick or healthy.
- The algorithm is a combination of cluster analysis and off-the-shelf estimation of individual ITE. Both methods are well known, so synthetic analysis could’ve taken less proportion because it is unlikely that the two methods would go wrong, especially the simulation setting is simplistic with dimension 2. On the other hand, real data analysis is necessary in order to evidence the value of the algorithm in application.
clarity,_quality,_novelty_and_reproducibility: The writing could've been further polished, as many places use non-standard language, and others are somewhat confusing. It could be better to introduce the algorithm first before going through its theoretical properties. The proof can be differed to the appendix.
summary_of_the_review: The problem that this paper proposes to consider could have been better motivated by using a practical example, and the algorithm looks like a generic algorithm rather than one that centers at the proposed problem. A lack of a real data analysis also undermines the quality of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an algorithm (pre-cluster and merge) to better disambiguate heterogenous treatment effects in non-targeted clinical trials (where there is a hidden confounder of the patient of whether the patient was sick/healthy/other confounder variable-- with same observable patient properties/covariates otherwise. 
strength_and_weaknesses: Strengths
- Simple methods that works well.
- Proof of theorem 1 is methodical and clear.
- Robust to different clustering algorithms

Weaknesses
- Paper lacks refinement in exposition. For instance, I think a causal graph could have illustrative; defining groups (sick, health etc.) as a hidden confounder. I think cleaning this up and making this clearer would be helpful.
- The paper lacks a coherent related work section; would appreciate more related work for instance how this relates to estimating HTEs with hidden confounder.

Minor Comments
- Side effects doesn't seem like the appropriate nomenclature.
- ATT instead of ATE or CATE if conditional

Questions
- Experiments demonstrating performance in imbalanced dataset with respect to the groups would be useful
- Are there no baselines that are able to incorporate to incorporate using the HTE to identify groups?
clarity,_quality,_novelty_and_reproducibility: Paper is clear, seems like the PCM algorithm is relatively novel in this context (although the context of the entire fields it is difficult to discern given the lack of comprehensive related work section. No code to reproduce but should be easy.
summary_of_the_review: I am leaning to reject, given the few weaknesses I've noted which I think are significant-- but would recommend accept should these be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors study the problem of estimating treatment effect heterogeneity when the subgroup indicators are unknown (e.g. sick versus healthy). In particular, the researcher knows that treatment effect heterogeneity only depends on the latent subgroup indicators, which are functions of observed covariates, but does not know how many subgroups there are and which subgroup each individual belongs to. The goal is to recover the correct number of subgroups, the effect for each subgroup, and to assign each individual to the correct subgroup. The authors propose an algorithm based on clustering and prove that it asymptotically achieves these three goals under seven assumptions.
strength_and_weaknesses: See below
clarity,_quality,_novelty_and_reproducibility: Clarity

*The algorithm intuition is very clear

*Why is equation (2) about ATT instead of ATE? Relatedly, in the algorithm, isn’t the average of ITEs the ATE rather than the ATT? This was confusing

*The discussion of the difference between ATT and EFF used diction that is atypical in causal inference. A more typical discussion would say that under the stated identifying assumptions, ATT and EFF do not coincide. “rendering ATT useless” is a bit of a jarring expression.

*Some statements are too loose. “If unbiased counterfactual estimation is not possible, then any form of causal effect analysis is doomed.” Do the authors mean selection bias? Regularization bias? Either way, causal analysis is not doomed—there are entire literatures about how to correct for both biases. Please clarify which bias is being considered and fix the language.

*A figure would be helpful to visualize the different steps of the algorithm. 

*I found this sentence to be confusing: “the counterfactual estimates use a predictor learned from the independent control population”. Is that the untreated subpopulation? Something else? Please clarify

Quality

*I crudely checked the results, but they seem rigorous

*I was surprised that the results did not appeal to smoothness of h or compactness of X_c. Is this not required for inverting h?

*A6 seems to be a strong assumption. Please discuss or formally sketch why this assumption is plausible. It would be nice to comment further on how the curse of dimensionality appears in A6. e^d N_{impure}<… seems to be the constraint. Is that correct?

*Overall, the strongest assumption seems to be that the treatment effect heterogeneity only depends on C, an integer, and the analyst knows this fact. This is formally stated in A2, but the introduction somehow didn’t convey this point forcefully enough.

Novelty

*The authors introduce the language of effect and side effect, but the overall goal is closely tied to a known problem that goes by several names: sorted group average treatment effect, classification analysis, and endogenous stratification analysis. I would like the authors to compare their problem statement with the problem statement of sorted group average treatment effect in Chernozhukov-Demirer-Duflo-Fernandez Val (2018). This reference also provides a thorough discussion of related works that are relevant for this submission and worth citing.

*While the problem is related to existing problems, the solution seems to me to be innovative and interesting.
summary_of_the_review: I found the solution to be original and therefore significant. I will improve the score if the authors connect the problem statement with previous work (which will actually improve the significance).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the estimation of treatment effects in the non-targeted trials in which one cannot control the selection of treatment units. An asymptotically consistent effect estimation algorithm is proposed and the experiments on simulated data are conducted to verify the effectiveness of the proposed algorithm.

While the paper introduces a new and interesting problem, there are several concerns on the technical quality and experiments. First, the new problem setting needs more justifications to show it is a nontrival setting and has real applications. Second, the technical novelty of the proposed method needs further clarifications. In particular, the relations to existing individual ITE methods need to be discussed. Finally, experiments on real world data would significantly strengthen the paper. Given these concerns, I would not recommend acceptance of this paper in its current form.  Although we think the paper is not ready for ICLR in this round, we believe that the paper would be a good one if the concerns can be well addressed.

justification_for_why_not_higher_score: justification of the practical value of the proposed problem setting is needed

technical novelty needs further clarification, especially discussion on connection to related works is needed.
justification_for_why_not_lower_score: n/a
Decision: Reject