Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary and key claims**

This paper repurposes the balanced representation learning framework for estimating treatment effects, originally proposed in (Shalit et al., 2017), for the survival prediction setup. The proposed model deals with selection bias induced by confounded treatment variables, and in addition, deals with censoring bias and informative censoring.

*The key contributions claimed by the paper are:*
- Developing a loss function incorporating adjustments for informative censoring and selection bias.
- Developing a generative model for event times based on planar normalizing flows. 
- Proposal of survival-specific evaluation metrics, including a new nonparametric hazard ratio estimator.

**Originality and significance**

Overall, I think that the paper is a straightforward application of the balanced representation method in (Shalit et al., 2017) to survival outcomes. It does not seem like survival prediction is any different from the conventional ITE setup with respect to the representation learning aspect of the model, hence I don't think that the paper contributes methodologically to the problem of handling selection bias. Moreover, the censoring terms in the loss function are also very similar to those introduced previously in Chapfuwa et al ICML 2018 paper. Based on this, I think that the extent of technical contribution in the paper does not pass the acceptance threshold.

I was expecting some more analysis on the interplay between censoring bias, selection bias and the effect of both on causal identifiability. Unfortunately, it seems that authors chose to address each of these impairments/biases separately using existing solutions, and the resulting model is simply an amalgamation of existing ideas. 

The "generative modeling" part of the model is somehow alien to the original problem of estimating treatment effects on survival outcomes. It is not clear why planar flows were specifically used and why flows are needed at all since the complexity needed in modeling is in the relation between features and survival parameters and not the complexity of the survival distribution itself. The usage of normalizing flows on the output layer is fine but seems to me unnecessary; this reinforces my impression of the model being all over the place.

On the positive side, I think that the problem of estimating ITEs on survival is very important and rarely addressed. Most ML models for ITEs focus on real-valued targets, but this is rarely the relevant setup in practice as survival is often the measure of treatment efficacy in medicine. I also think that the idea of comparing the estimated HRs of RCTs with the ones recovered by the model is a very smart way to evaluate counterfactual inferences, and can be a useful evaluation metric for future papers.




review_title: Important problem but limited methodological contribution.

final_score: rating: 7: Good paper, accept
review_content: Disclosure: I found this paper online during review process https://arxiv.org/abs/2006.07756


This is a comprehensive paper with interesting application of counterfactual inference under survival analysis setting. Overall, my recommendation is to accept. 
•	It nice that the proposed nonparametric approach in this paper can adjusts for bias from confounding due to covariate dependent selection bias and censoring (informative or non-informative). 
•	Under three criterions [concordance index (C-Index) (Harrell Jr et al., 1984), mean coefficient of variation (COV) and calibration slope (C-slope) (Chapfuwa et al., 2019)] and three datasets [FRAMINGHAM, ACTG, semi synthetic ACTG], compared proposed method with 7 seven others, including survival Bayesian additive regression trees (Surv-BART) (Sparapani et al., 2016) [using nonparametric Kaplan-Meier based estimator] and Cox proportional hazard model (using real HR form, using three normalized weighting schemes). 
•	P6, equation (10), the nonparametric form is a natural adoption of KM estimator. We know S^{‘}=-f(t). Wondering the motivation of choosing a linear approximation to S, and curious would the cardiovascular and HIV data adopted happen to be with S not so curved? Could you shed light on these?
•	P3, assumption of “no unobserved confounders or ignorability” sounds strong. Understand the mathematical challenge if relaxing it. Maybe for future research.
•	The overall presentation is nice. The organization of a few places might be improved to help first time reader to follow, eg. ITE initially defined on P3 without example till two paragraphs below, h_{A} first mentioned with no prior definition and no explicit math relation to p(T|X), briefly specify “Do (A=a)” is for effect of intervention.  
•	Minor issues like align the symbols used across the paper, eg. add subscript when define S^{‘}_{i},  m_{i}, ... i=0, 1 to increase clarity. 

review_title: Under the survival analyses setting, this paper concentrates on counterfactual inference related to individualized treatment effect, especially hazard ratio. Bound proposed in Shalit et al., 2017 is adopted for model learning by minimizing the upper bound which consists of factual loss and integral probability metric. Proposed factual loss is similar to Chapfuwa et al. (2018) for non-informative censoring case, and adding extra loss terms for informative censoring case. Simulation result shown. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper provides an approach for causal inference in observational survival dataset in which the outcome is of time-to-event type with right-censored samples. The method consists of a representation learning component to reduce selection bias and a survival analysis component which is modeled with normalizing flows. 

Pros:
- The paper addresses an important and interesting question.
- The manuscript is well-written and easy to understand.

Cons:
- My main concern is the minimal originality and significance of this work. That is, the representation learning component is directly taken from (Shalit et al., 2017) and the objective function for the survival analysis component is directly taken from (Chapfuwa et al., 2018). Also, the use of normalizing flows for modelling time-to-event kind of targets is not well motivated. 
- The literature review only points to several publications but does not go in depth into why/how the proposed method differs from those. There are also some important references that are missing. For example, Miscouridou et al. (2018) also use normalizing flows for survival analysis and it is necessary that the authors discuss how their work differs from theirs.
Miscouridou, X., Perotte, A., Elhadad, N., & Ranganath, R. (2018). Deep survival analysis: Nonparametrics and missingness. In Machine Learning for Healthcare Conference
- the paper suffers from many inaccurate statements; a few examples follow:
1. In the third paragraph of the Introduction section, the authors mention that “the treatment assignment mechanism is not known a priori. Therefore, there may be variables, known as confounders, affecting both the treatment and survival time, which lead to selection bias”. This is wrong; even if we know the treatment assignment policy a priori, we still might have selection bias … these two are independent.
2. In the last sentence of paragraph four of Introduction, the authors say that the methods (cited above) that account for confounding bias (by re-weighting) lack a counterfactual prediction mechanism. This is wrong because trivially, all methods have a prediction mechanism in place, and the ones that do account for confounding bias can predict counterfactuals accurately as well.
3. The authors should note that representation learning does not **remove** confounding bias; it might only **reduce** it. Also, reweighting does not **remove** confounding bias either; it just **accounts for** it.

Minor:
- In section 2, under *Estimands of Interest*, the authors state that “$\lambda(t | x)$ is defined below” but it’s never defined.
- In section 3, under *Accounting for selection bias*, the authors state that we can go from Eq. (1) to (2) because identifiability holds (since “$X$ is a sufficient set from $A$ into $T$.”). Could you please explain how this is different from Ignorability? I.e., $\{ T_0, T1 \} \perp A | X$ ?
review_title: Concerned about the originality and significance; includes many inaccurate statements.

final_score: rating: 7: Good paper, accept
review_content: This paper is very well written. The motivation and formalism is also clear with every step in the argument properly justified. The extension of individualized treatment effects to survival data is in some sense straightforward as both areas are quite mature and can be unified with aggregated loss functions dealing with biases of different type. The proposed solution, metrics and datasets proposed for this problem are compelling though and I believe will serve as a benchmark for further studies on treatment effects and survival data.

One question I have is on Corollary 1. I don't see a meaningful difference between this statement and that given by (Shalit et al, 2017), nor a proof in the Appendix. Why the separate statement?

The experiments are somewhat underwhelming. Survival BART has also been developed for treatment effect estimation [1]. This benchmark is certainly more relevant. Similarly, Survival-based deep learning architectures have been developed which could have been considered as well [2]. At least considering these, perhaps modelling each treatment group separately or including treatment as an additional feature should be considered to understand where the source of gain comes from. As presented, since almost all benchmarks consider linear interaction between features I would guess that improvements come from non-linear modelling rather than bias reduction from censoring and selection bias. 

[1] Hu, Liangyuan, Jiayi Ji, and Fan Li. "Estimating Heterogeneous Survival Treatment Effect via Machine/Deep Learning Methods in Observational Studies." arXiv preprint arXiv:2008.07044 (2020).
[2] Lee, Changhee, et al. "DeepHit: A Deep Learning Approach to Survival Analysis With Competing Risks." AAAI. 2018.
review_title: Well-articulated paper extending individualized treatment effect estimation to time-to-event outcomes

Meta-review: Summary: This paper provides an approach for causal inference in observational survival dataset in which the outcome is of time-to-event type with right-censored samples.  To this end, the paper adapts the balanced representation learning approach proposed in (Shalit et al, 2017) to the context of survival analysis.
The paper adapts an approach that uses flexible models to learn nuisance models, common in machine learning.

The authors validated their approach via simulation study and a set of application datasets: a EHR-based cohort study of cardio-vascular health, an RCT dataset of HIV patients, and a semi-synthetic dataset.

The main concerns of reviewers were due to perceived lack of originality relative to the original proposal in (Shalit et al, 2017)

Decision: Reject