Venue: UAI 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations. 
review_content: Q1 Summary and contributions: This paper presents a feature selection framework and an algorithm to identify "distributional" treatment effect modifiers, by examining how a feature influence the discrepancy between potential outcome distributions. Most existing effect modifier detection methods are based on conditional average treatment effect (CATE), which only enables them to find features affecting the mean of treatment effect distribution, but not other distributional parameters (e.g. variance). 
Q3 Main strengths: 1. The problem proposed and studied by the paper is interesting and well motivated.
2. The proposed feature selection framework sounds reasonable, and the authors show that the empirical estimator of WCMMD can be converged to the true value in the limit of infinite samples. 
3. Attention is paid to improve the efficiency of the proposed estimator.
Q4 Main weakness: 1. Related work should be enhanced.
2. More based line methods should be include to strengthen the experimental evaluation.


Q5 Detailed comments to the authors: *Related work
The existing MMD-based metrics mentioned in the 2nd paragraph of Section 3.2 should be discussed in more detail, perhaps in the Related work section. More importantly, it is not clear why those metrics cannot be applied in the setting of this paper. Although it has been stated that these metrics are not designed for the conditional distributions conditioned on a single variable, would it be possible to adapt them for the setting "conditioned on a single variable", and is the difference between those metrics and the proposed one in this paper non-trivial, and why?

*Experiments
The paper is focused on distributional treatment effect modifiers, but the "mean" is an important functional of a treatment effect distribution, and features affecting the mean should be a target to be identified by the proposed method. Given this fact, it is necessary to compare the proposed method with existing mean-based methods to fully demonstrate the capability and performance of the proposed method.

Additionally, does the proposed method assume all covariates are pretreatment variables? How would confounders play a role/affect the functioning of the proposed method?
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: The problem studied by the paper is well motivated, and the proposed solution looks sound. As mentioned above, however, more details on the related work should be provided to clarify the originality/significance of the contribution. The evaluation of the method should be improved by including more baseline methods.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: The paper presents a method for selecting covariates that may influence the variance of treatment effects. The idea is novel and the authors show in simulated experiments that the method can indeed detect such covariates in simulated data. 
Q3 Main strengths: The idea is novel and interesting. 
The method is sound as far as I could check.
The experiments (albeit a bit limited) support the claims of the paper.
Q4 Main weakness: I do not think it is a major weakness, but it is a bit limiting that the method looks at the effect of single covariates. 
Q5 Detailed comments to the authors: Section 3.2. What is the challenge in using MMD measures designed for measuring discrepancy between potential outcomes distributions, since in Eq 2 you have conditioned on a single value of X?

Can you comment a bit on whether distributional effect modifiers would be detected using for example tests of independence to the outcome?

I would like some discussion or an example on how the method could be used to improve outcome prediction after these covariates are identified. 
Q2(4) Quality of experiments (Optional): 3: Good: The experimental evaluation is adequate, and the results convincingly support the main claims.
Q7 Justification for your score: I believe the paper presents a novel idea and method and it is well written and justified by the results.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept, e.g., novelty, outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: Q1 Summary and contributions: This paper proposes a feature selection framework for discovering the features related to distributional treatment effect heterogeneity. It proposes a feature importance measure based on the discrepancy between conditional potential outcome distributions and uses the multiple  hypothesis testing method to select the important features. It also provides a detailed algorithm for the proposed method. 
Q3 Main strengths: 1. The idea of the proposed method is creative. 

2. The introduction of the proposed method is detailed and clear.
Q4 Main weakness: 1. The computation of the proposed method is complex and heavy.

2. The true positive rate of the proposed method in the experiments is not so satisfying when the sample size is not too large.

3. The proposed method can not deal with the situations where the feature does not affect the discrepancy between conditional distributions  of potential outcomes but influence the joint distribution of potential outcomes. 
Q5 Detailed comments to the authors: 1. In the problem setup, Y is assumed to be a continuous-valued outcome. But in Example 1, Y is discrete. Why Y needs to be a continuous outcome in the proposed method?

2. Based on the comment in the last paragraph of Section 3.1, there are some situations that X_m does not affect the discrepancy between conditional distributions but influence the joint distribution. Then in some situations the false negative rate will be high. Can you show the results about the false negative rate in the experiments?

3. Can you explain more about the meaning and properties of characteristic kernels in Section 3.2?

4. In the experiments, it says that inferred propensity score and the encoders and decoders of CVAE are fitted using the neural networks. How to choose the number of neurons? Whether the results are sensitive to the number of neurons? 

5. Based on Figure 1, the TPRs based on  the naive methods are much higher than that based on the proposed method. Can you explain the reason for this?

6. Based on Figure 2, when the sample size is not too large (but still no smaller than 500), the TPR of the proposed method is not so satisfying. Note that in practice, the sample size of the real data can be much smaller than 500. 

Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: I make this score based on the main strengths,  weaknesses and my understanding of this paper. I think the strengths and weaknesses of this paper are comparable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area of AI or moderate to high impact on more than one area of AI, with good to excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations. 
review_content: Q1 Summary and contributions: This work describes a new method to discover the features that cause heterogeneous treatment effects across individuals. Instead of using the conventional mean-based method such as the conditional average treatment effect, this work proposes to use the variance as the measure. The key is to quantify how strongly the feature influences the discrepancy between potential outcome distributions. 
Q3 Main strengths: 1. A new research perspective to tackle a conventional task.
2. Well written and easy to follow.
3. Both theoretically and empirically sound.
Q4 Main weakness: 1. Some experimental results do not show the effectiveness of the proposed method.
2. Discussion of the different roles of X can be helpful.
Q5 Detailed comments to the authors: This paper addresses an important limitation of conventional approaches for finding treatment effect modifiers and introduces a novel perspective that can be potentially helpful for the research field.

Authors should consider the following to further improve the current version:
1. The only major concern I have is the results in Figure 1. The proposed method seems not to work well for Nonlin Mean and NonlinVar, i.e., it has a much lower TPR but a very similar FPR to the Naive method. This contradicts what the authors described in the text. A detailed explanation is needed.
2. It would be helpful to understand the problem and proposed method if the roles of X are discussed, e.g., confounder, mediator, or collider, in a causal graph. Would the method work in these different roles?
3. Would a method that considers both mean and variance outperform both mean-based and variance-based methods?
4. Some annotations are not described, e.g., what is B?
Q2(4) Quality of experiments (Optional): 2: Fair: The experimental evaluation is weak: important baselines are missing, or the results do not adequately support the main claims.
Q7 Justification for your score: Overall I like the idea of this paper and it is well written. I will give an Accept if the authors can further explain the results in Figure 1.
review_title: null

Meta-review: Meta Review: This work proposes a method for selecting features that are important for explaining treatment effect heterogeneity. Unlike most existing works, the proposed approach is able to detect features that drive differences in the conditional distributions of counterfactual outcomes across treatments, rather than only predicting the conditional average treatment effect.

All reviewers were positive on this work (accept, accept, weak accept, borderline accept). They praised the novelty of the proposed approach, the clarity of the writing, and the (albeit limited) experiments.

During the rebuttal period, the authors replied to the main comments of all reviewers. Two of the reviewers engaged in discussion with the authors, and both of these reviewers stated that they were satisfied with the authors' replies. The other two reviewers did not engage in the discussion, but in my view the authors adequately addressed their main concerns.
Decision: Accept (Poster)