Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper contributes to the domain of causal inference, in particular to the area of Heterogeneous Treatment Effect (HTE) estimation. Authors propose a methodology to identify the drivers of treatment effect heterogeneity along with a corresponding semi-synthetic benchmark and the new insights into the performance of existing methods on the identification task.
strengths: - The classification of features into predictive and prognostic in CATE estimation context are presented.
- Data generating process along with the metric for effect heterogeneity drivers identification are detailedly explained.
- Experimental results including variety of realistic setups are clearly identified.
weaknesses: - Simulations based on large-scale datasets are missed.
- Lack of hyperparameter tuning in experiments.
correctness: The evaluation methods and experiment design are appropriate and performed correctly.
clarity: Paper is clearly written.
relation to prior work: Related work in Section 1 clearly describes the difference of this work from previous contributions.
additional feedback: - Adding large-scale real datasets could help to better distinguish between the CATE estimators in terms of predictive features identification and bring the new insights into performance of existing methods.
- Line 342: In a light of your results, how to explain the CFRNet’s significant outperformance of the baselines in the original paper (Shalit, 2017), on the IHDP and Jobs datasets ? In particular, does it mean that the treatment assignment bias has a different nature from ones simulated in your work ?

Small typos:
- Line 249: should it be “high Attr_{prog}” instead of “low Attr_{prog}” ?
- Line 251: “does does”
review_title: Review for Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a benchmark called ITErpretability that focuses on post hoc feature importance methods for black-box predictive models. More specifically, the authors explore the estimation of conditional average treatment effects (CAGE) to identify important features (covariates) that a given Machine Learning (ML) model uses. A novel data generation strategy is used by the authors to distinguish between predictive and prognostic covariates. Finally, various experiments are explored by the authors to test various CAGE estimators on the basis of their effect on strength of predictive effects, in the nonlinear setting, and on the effect of compounding on them. 
strengths: 1. Estimation of post hoc feature importance in machine learning methods is an important and challenging task, especially in the healthcare domain.
2. Incorporation of CATE in machine learning models is a hot topic and the authors propose a novel benchmark that aims to identify predictive and prognostic covariates using CATE. This is important to push the field forward and therefore, this work is appreciated.
3. The motivation and the related works are explained clearly. 
4. The authors have implemented a novel data generation method for their task which is very relevant for the proposed benchmark.
5. The authors have explored various learners on several datasets as well as several tasks that show the completeness of this benchmark.
6. The hyperparameters used are made public ensuring reproducibility of their analysis. 
weaknesses: ### PRE-REBUTTAL
1. It is not clear why the authors used an 80%-20% split to train and test CATE estimators. Cross-validation would be a more robust strategy to apply in this setting. 
2. Figure 1 is not clear. A better resolution image would make it easier to read.
3. There are minor typing errors in the text - for example, on page 1 (Introduction), "an individuals potential outcomes" should be "an individual's potential outcomes".

### POST-REBUTTAL
The above issues have been resolved and appropriate justifications have been provided. 
correctness: Yes
clarity: Yes, although some concepts were difficult to understand based on the text and required external reading.
relation to prior work: Yes
additional feedback: -
review_title: Exciting work on interpretability of heterogeneous treatment effect models

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The estimation of conditional average treatment effect (CATEs) is the problem of predicting $\tau(\mathbf X)$, the difference between the expected outcomes of an individual/sub-group, characterized by covariates $\mathbf X$ under different treatments. A common assumption in the literature is that not all covariate features are relevant for the CATE. The authors propose to evaluate the ability of CATE estimators to distinguish between predictive and non-predictive covariates. For that, they propose semi-synthetic simulations based on existing covariate features. In their evaluations, they compare the ability of existing NN-based CATE estimators to identify the predictive covariates under varying strengths of predictive effects, nonlinearities, and confounding.  
strengths: * The paper is well-written, and the benchmark design looks sensible and is easy to follow
* The experiments reveal novel insights compared to an evaluation based solely on PEHE. Therefore, this contribution can become significant and relevant to the broader research community.

weaknesses: # Pre-Rebuttal
A common CATE meta-learning strategy is to incorporate a nuisance estimator for the prognostic effect $m(x) = E[Y | X=x]$. For example, see the works of [1] using non-neural-network models or [2] using NNs. The idea behind this strategy is to construct a pseudo-outcome in which the prognostic effect was removed, such that the CATE estimator can better identify the predictive covariates. 

I think adding at least one method that constructs a pseudo-outcome with removed (estimated) prognostic effect to the evaluation would be appropriate. Unfortunately, as far as I can tell, the authors have not evaluated such an estimator/strategy, which is precisely motivated for settings where there can be non-predictive covariates that we want to regularize against.   

Another minor weakness is that I think the motivation for such settings with naturally prevalent non-predictive covariates in the dataset could be better explained. It is not clear to me in which settings many non-predictive covariates naturally occur that a data scientist would not be able to filter out a priori. The above-mentioned works may be partly used as motivation, but I think introducing a novel benchmark and convincing future researchers to invest their time to use it requires a strong, watertight motivation for such settings. One way to explain this better can be to point out two or three concrete examples with many non-predictive covariates motivated through real-life scenarios. 

I don't find the cited drug discovery example mentioned in the introduction very clear. While it says that it is important that a model discovers the correct drivers of the underlying effect heterogeneity, it is not apparent to me why this is particularly important in drug discovery or why in these datasets, there may be many non-predictive covariates. 


[1] Nie et al, Quasi-Oracle Estimation of Heterogeneous Treatment Effects
[2] Kaddour et al, Causal Effect Inference for Structured Treatments

# Post-rebuttal
The above concerns have been addressed and I raised my score.
correctness: Yes, everything looks designed and performed correctly.
clarity: Yes, overall, the paper is indeed very well-written and easy to follow, besides the motivation for the problem
relation to prior work: The paper discusses previous benchmarks and issues thereof but does not mention works constructing a pseudo outcome with removed prognostic effect, see


[1] Nie et al, Quasi-Oracle Estimation of Heterogeneous Treatment Effects
[2] Kaddour et al, Causal Effect Inference for Structured Treatments

additional feedback: No additional feedback
review_title: Well-written and designed benchmark paper. Motivation and empirical evaluation could be improved.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a DGP that allows post-hoc feature importance, used to identify covariates important on CATE estimation. 

The feature important is especially useful if the CATE estimator is a 'black-box' (Neural network). 

The feature importance method was Integrated Gradients due to performance and efficiency (other methods available in Appendix).

The proposed DGP was evaluated empirically on several datasets, with several CATE estimators. 

 
strengths: 1. It addresses an important causal inference problem: the evaluation of treatment effect methods.      
2. The empirical study takes into consideration non-linearities and confounding.     
3. Interesting distinction between predictive and prognostic features. Several applications could benefit from such distinction between covariates.    
4. It allows the data simulation on top of existing datasets, which is a good and flexible characteristic.    


weaknesses: 1. The naming needs some improvement. It often mentions the 'discovery of covariates' which, considering the causality context, can be misinterpreted as a causal discovery problem. Important to point out/state the distinctions.  It also mentions confounding, but it is not clear how that would affect the identification (from the back-door perspective) of the treatment effects and feature importance. Are the confounders always observed?

2. The empirical study adopts a data generation process. While there are several plots to investigate if the methods work in these simulations, there is no analysis of how realistic they actually are on the real-world. 

3. Considering the problem of CATE evaluation: ground truth is not available in real-world applications. Are there any guarantees of how confident can I be about the 'most important prognostic feature' being indeed the most important prognostic feature? What settings can we know for sure, and in which settings might fail? 

correctness: The paper contains an empirical study to evaluate a data generation process on top of existing datasets. The goal is to estimate CATE. 

In the paper checklist, the authors claim that they didn't state all assumptions and theoretical results because their papers are purely empirical. However, I think some assumptions are required to offer some guarantees. Based on the experiments, the proposed method performs poorly if there are nonlinearities or confounding.  What does that mean in practice? If I have an application for such a method, and there are nonlinearities and confounding, can I apply the method and trust 100% the feature importance that it outputs? My intuition says that, while the empirical study worked on simple examples, that might not be true for other more complex applications, and results should take interpreted very carefully. Hence, assumptions regarding the covariates or a clear description of their limitations would be very welcoming. 

Hence, while not necessarily incorrect, I would say the paper is incomplete and lacks a proper discussion on limitations, assumptions, and interpretation of empirical results. 
clarity: The paper is very hard to read. I had to re-read several paragraphs to understand what the author was describing.

Some terms are now well defined. For example, what "discover drivers" means? Are these similar to causal discovery?              

A formal definition of prognostic and predictive covariates could also improve the paper. 

Figure 1 is very hard to read, the letters are very small. I have printed the paper, and I can not read anything. Other plots also have very small letters (but are more legible).         

The author use circle numbers several times, and it is confusing. Maybe keep to the most important numbered items, and find a better way to enumerate the other items with words.  

The discussion of why CATE is special is not clear. It seems the author is just describing how it is setting up the experiments again. 
relation to prior work: The author mentions some related works on feature importance, and approaches to interpreting CATE. 

Maybe the authors could have adopted some datasets that are more complex. What happens with the datasets being high-dimensional? Or if the covariates are images? If the method does work with these other datasets, it is one more indication that maybe some assumptions or a complete description of the limitations are missing. 
additional feedback: 1) I suggest renaming the predictive covariates for something else.  Readers can easily misunderstand the term, as it is often used in the context of predictions (predictive covariable = use to predict Y) versus predictive of effect heterogeneity (as used on the paper).

2) The use of 'discovery of predictive covariates' is also not ideal. Causal Discovery is a distinct and harder problem, and a similar name could also mislead readers.     

3)  5.1 Setup mentions alpha ~ Uniform(), but that is not shown in algorithm 1.       

4) Maybe standardize the Attr_pred and Att_prod. From my understanding, one is good if close to 1, and the other, if close to 0. It is easier to understand and interpret results if the ideal solution is on the same scale (close to 1 is good for both). 

5) You mentioned that PEHE is not directly comparable (lines 266-267). So maybe use a different plot, as that is not clear with the current line plots.       

6) The conclusion of 5.2 is not clearly stated. Does that mean the proposed approach fails if there are nonlinearities or confounding? If yes, that should be more clearly stated as a model limitation. 


For reference: I spent around 6h reading the paper, code, and appendix. 


POST-REBUTTAL
Several of my concerns were addressed or explained by the authors. I increased my score. 
I still think the writing could be improved, but most of my concerns regarding the method itself were clarified.
review_title: Post-hoc feature importance to identify covariates that are important on CATE estimation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors propose a framework to generate data that provides us with ground-truth access to predictive covariates. The main claim of the paper is to use the proposed framework as a benchmark to evaluate CATE estimators. The authors underline the importance of distinguishing between predictive and prognostic covariates and they delineate a method to assess the identification quality of predictive covariates by multiple techniques. 
They then show the results obtained on multiple dataset types (TCGA, Twins, News, ACIC2016) comparing the performance of five different methods. 


strengths: The idea is extremely interesting and quite simple in itself. I like the quality of the explanations and motivation given by the authors, who underline the importance of analysing separately predictive and prognostic covariates. Indeed, benchmarks as such can only help pushing forward the research on causation. 
weaknesses: The main weakness that I see in the paper is on the statement of the main contribution. It is unclear to me whether the authors are proposing a data-generation algorithm (i.e. Algorithm 1 on page 5), or if they are providing a benchmark of CATE learners that could be then extended to new learners and methods in the future. While the two things go hand in hand, I fear that the discussion on Algorithm 1 is not solid enough and that too much attention is given afterwards to the benchmarking results. 

correctness: I see the idea in principle as correct and I think it is quite simple and straightforward to implement. However, I do think that more attention should have been placed to the applicability of this method in high-risk domains such as the medical and biological domain. The authors do provide an example of a  bio-medical dataset with TCGA, but no discussion at all is given on the implications of their  arbitrary treatment assignments and outcomes in this context. TCGA contains several information on patient diagnostic, prognostic, and treatment. What aspect did the authors consider in the specific for this case? How do they think that their method could apply to similar datasets and what are the risks of using arbitrary associations to evaluate the CATE learners? This was my main concern during the entire read of the paper. What if the assigned outcome makes is really nonsense for the treatment that has been chosen? I think these kind of discussions must be in the paper. 
clarity: The paper is clearly written and I could follow the structure and understand how to recreate their dataset. 
relation to prior work: Yes, the authors discuss the previous work and they clarify that their method, differently from the existing ones, sits on top of already trained models, proposing a post-hoc approach. 
additional feedback: I would like to stress again the need for a stronger discussion on the implication of generating arbitrary associations. I feel that the authors tried to avoid this discussion point that is instead at the very core of their contribution. 
The authors could also introduce a small discussion on the limitations of Integrated Gradients (in particular with the baseline reference set to zeros). 
Finally DGPs on page 5 was not defined before in the text.
review_title: Generating ground truth for conditional average treatment effect (CATE)

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper investigates the feature importance estimation in conditional average treatment effect (CATE) tasks. Specifically, this paper argues that the prognostic and predictive features should be discriminated in CATE. To this aim, the authors propose a semi-synthetic dataset that provides ground truth PEHE and the identification of prognostic and predictive features. On top of it, experiments are performed to compare the existing meta-learners as well as model-specific methods in CATE on the feature importance allocated to predictive and prognostic features.
strengths: 1. Important question on how to interpret black-box CATE models, especially highlighting the importance of differentiating predictive and prognostic features in causal inference scenarios. 

2. Provide a practical pipeline for benchmarking existing CATE estimators in terms of interpretability.
weaknesses: It is still inconclusive from the results like which methods essentially perform the best. Several factors may influence the benchmark results significantly:

- the heterogeneity of the four datasets

- the chosen feature importance estimation methods

- the picked base models for meta-learners

- the mechanism that generates the semi-synthetic data

All of them narrow the scope of this benchmark for guidance in real-world practice.
correctness: I believe the methods and experiments of this paper are technically sound.
clarity: This paper is well written.
relation to prior work: This paper makes a thorough review of existing arts in CATE and model explainability.
additional feedback: If the feature importance estimated for predictive and prognostic features matters for the deployment of CATE methods, what is the best practice to evaluate CATE estimators? When one model gets high $Attr_{pred}$, low $Attr_{prog}$ but high PEHE, or when one model gets low $Attr_{pred}$ but low PEHE?
review_title: Review for "Benchmarking Heterogeneous Treatment Effect Models through the Lens of Interpretability"

Meta-review: metareview: This submission proposes a semi-synthetic dataset to estimate conditional average treatement estimation (CATE) and an empirical study of recovery of important parameters by CATE estimators.

The submission generated much discussion between the reviewers and the authors. The question and the empirical evidence was seen as interesting. Although it does not isolate factors of a causal-inference scenario that modulate the success of identifying important features for CATE models, it was seen as setting a promising line of research.
confidence: 2: The area chair is not sure
Decision: Accept