Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a latent variable model (SyncTwin) for estimating time-varying individualized (point-) treatment effects (ITE) given temporal irregularly sampled pre-treatment covariates and outcomes. Theoretical results justifying the proposed approach include upper bound errors on the learned representations and counterfactual predictions. Experimental results on synthetic and real-world datasets show SyncTwin outperforms baselines per mean absolute error (MAE) on ITE and recovering RCT findings, respectively.
main_review: Overall I enjoyed reading this paper; the reviewer appreciates the clarity and organization. 

**Strengths**
- The formulated latent variable model encodes interpretability by leveraging representation learning and a matching algorithm for counterfactual prediction.
- The theoretical analysis for upper bounds on the learned representations and counterfactual predictions justify the proposed formulation.
-  The paper shows SyncTwin recovers RCT findings on a challenging, large-scale real-world EHR dataset, something rarely done in most machine learning papers.

*Below are the limitations.*

**Technical limitations**
- The paper seems like a straightforward extension of Synthetic control (SC), albeit accounting for irregularly sampled covariates. Also, the performance gains over SC seem marginal given the complexity of the proposed approach.
- Unlike approaches such as CFRNet, SyncTwin is memory inefficient as it requires the storage of learned representations from the training datasets for the matching algorithm at test time.
- Given that the matching algorithm is key to SyncTwin, empirical/theoretical analysis of best or worse case scenarios would strengthen the paper.

**Weak experiments**
-  Experimental findings on the real-world dataset are missing baseline results from SC or I-NN?
-  Also, the paper should provide qualitative results from synthetic datasets showing predicted $y_t$ against ground truth.
- Comparisons with propensity weighted approaches would be interesting.

limitations_and_societal_impact: The discussion on limitations needs to be expanded.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a representation learning approach to model counterfactuals in longitudinal data with point interventions over time. Their approach, SyncTwin is thus suited to situations where an intervention is made on a temporal dataset at an instance of time or as a continuous exposure by treating all points post the intervention as the potential outcomes under treatment. 

They compare the performance of their model against several counterfactual estimation baselines originally proposed to model static observational data with interventions.
main_review: The problem of identifying the *Individual Treatment Effect* in temporal settings is a challenging one due to issues of identifiability of the potential outcome over time. In this paper, the authors assume a simple setting with a binary treatment or intervention, such that all points post the intervention are considered as sampled from the distribution under treatment. Furthermore, at each time step the treatment is considered the same (the current approach does not model the effect of time-varying interventions) 

Their main innovation is that at test time, the counterfactual outcome is determine by modelling it as mixture model over the observed time-steps (in the paper they refer to the weights of this mixture as $b_{ij}$ ). 

## Strengths

- Interesting problem of counterfactual inference with longitudinal data. 
- Some theoretical insights around the latent factor model maybe interesting to the readers. 
- Extensive experimental evaluation against multiple baselines on synthetic and real world datasets.

## Major Concerns / Weaknesses

- The authors frequently confuse the term, *Data Generation Process* (DGP) and the *Modeling Assumption*.(Ex. lines 135-136) The DGP that there is a latent factor $c_i$ that causes the outcome and covariates jointly, is different from the modeling assumption that the relationship is linear. The authors should reconcile this terminology and clearly differentiate between the modeling assumptions and the data generation process.

- The representation $c_i$ is frequently referred to as a *latent variable*. For $c_i$ to be a latent variable, there needs to be some explicit distributional assumption on $c_i$, which is not the case in the assumed model. The authors should thus instead call it a representation, not a latent variable. 

- The authors motivate the use of their mixing weights $b_i$ from an optimization perspective. In the DAG notation however, the $b_i$ s are mixing weights or probabilities. The authors should reconcile the proposed model clearly specifying the modeling assumptions via another DAG, (ideally separate from the DGP in figure 2) that includes $b_i$ as a *latent variable* over the time-steps. 

-  The final concern this reviewer has is that the current framework does not allow the model to represent the potential outcome $Y(1)$ or $Y(0)$ with a non-linear function of the representation $c_i$. The authors mention that this would be "inconsistent with the DGP" (ln 135)". Firstly, as noted above, this is a modelling assumption, not the DGP. Secondly, this reviewer believes that this assumption is not required per se under the proposed model, if the authors were to remodel $c_i$, factual outcomes $y_i$ and mixing weights $b_i$ as true random variables (with explicit probabilistic interpretation). The Matching Loss and Reconstruction Loss would thus be easily modified as a likelihood in that case. 

Overall this reviewer believes that although the proposed approach is interesting, in its current form the paper requires substantial revision to:
- present and motivate the approach in a formal probabilistic sense, including the mixing weights $b_i$ and the representation $c_i$
- De-confound the DGP from the Modeling Assumptions. 

## Other concerns: 

An important baseline that seems to be omitted when benchmarking performance is the Causal Effect-VAE (CEVAE) (citation [35] in the paper). I would recommend the authors to explicitly compare SyncTwin against CEVAE with an LSTM based encoder-decoder (to obtain the latent $c_i$)


limitations_and_societal_impact: I would encourage the authors to rephrase the Societal Impact section by focussing on how accurate counterfactual estimation can better help policy making for decision support in critical scenarios such as healthcare.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes an individual treatment effect estimation method called SyncTwin, which learns synthetic twins of target patients via the use of representation learning for counterfactual predictions. The proposed method is used on longitudinal data such as electronic health records. The paper is concerned with the setting of longitudinal and irregularly sampled data with point treatment. The paper describes the proposed method, offers some theoretical guarantees, and demonstrates the utility of the method on both synthetic and real-world data.
main_review: I think the paper makes efforts to address an important problem in causal inference, i.e. individual treatment effect estimation from longitudinal data. The proposed method is concerned with a specific, relatively simple setting (compared to dynamic treatment ) without unobserved confounders.

Originality: I think the proposed method is original, in terms of deriving solutions tailored to the specific setting proposed in the paper. The proposed method is also closely related to several lines of research in causal inference but different from those in various ways.

Quality: I think the proposed method presents solutions to deal with various important challenges under the specific setting. Theoretical characterization is provided in the paper, and the analysis of experimental results on both synthetic data and real-world data is detailed.  I am somewhat unconvinced by the interpretability provided by the method. While the linear combinations of contributors provide some convenience to interpretability, such contributors are learned latent representations, which potentially are difficult to interpret. While the proposed method is validated on real-world data, the authors use the fact that the average treatment effect learned by the method is close to the one reported in RCT as an important argument to justify the effectiveness of the proposed method. I am not sure if such a justification suffices. For one thing,  it is unclear whether the population in the observational study is comparable to that in the RCT (despite the authors used the admission criteria of the RCT for the observation study).  It is also unclear whether the observational study meets the LIP setting, satisfies the assumptions made in the paper, and whether all confounders are appropriately adjusted. Based on these concerns, I am somewhat skeptical about the reliability of the results from real-world data. 

Clarity: I think the paper is clear. Notation are well defined and used.

Significance: I believe the paper has certain significance based on the setting that it deals with and the difference it has compared to other related work. It's also a good attempt at tackling more challenging causal inference problems from longitudinal data.
limitations_and_societal_impact: I think the authors can discuss more the limitation of the proposed method and understand the failure mode of the proposed method better.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a method, SyncTwin, to estimate treatment effects in Longitudinal and Irregularly sampled data with Point treatment (LIP) setting. The idea is to create a synthetic twin from controls that closely matches a treated target in representation. By using the pretreatment time samples of the twin to estimate temporal trend, SyncTwin can predict the outcome of the target when no treatment is administered. The method is applied to simulated and real data with lower prediction error shown.

main_review: Originality
Not sure if the combined idea of creating a synthetic twin and using pretreatment outcome is novel since Synthetic Control at a high level seems to be doing that as well.

Quality
The method seems sound, and the results on both simulated and real data are solid. A couple of question comes to mind. First, are there studies where no matched control data are collected during post-treatment period (which is used as ground truth in the current paper)? Second is what happens if we directly use the outcome prediction without creating a twin, since SyncTwin provides such output?

Clarity
The paper is mostly clear with good motivation provided for the proposed method, i.e. better use of pretreatment outcomes. Also, dividing the paper into sections on problem statement, assumptions, etc. make the paper easy to read. The problem statement is clear, but would suggest labeling Figure 1 with variable names. Also, the sentence “the pre-treatment outcomes are arguably much more closely linked to the outcomes after treatment than the covariates— they hence deserve special considerations, e.g. modifying the architecture or loss function to reflect their importance” is a bit unclear without proper context in the Introduction.

Significance
Being able to predict treatment outcome is of broad interest. The method provides both outcome prediction and error estimates to inform clinicians. For others to adopt this method, the authors will need to describe in more details on what kind of data are applicable, i.e. this method would not be needed if we have data from matched controls taking placebo. 

Summary Post Author Response
I have read through the reviews and responses. I will keep my original score, which was mainly based on the results. My main concern was real world applications. It might be worth adding a bit more details on how to select a matched control group for executing the proposed method when data are not from well designed experiments, i.e. a random set of controls might not match well with all patients.
limitations_and_societal_impact: The authors mentioned extension to dynamic treatment settings and model outcomes in continuous time. The authors also mentioned, in practice, the clinician can accept the recommended treatment when the error bound is below a threshold and resort to expert knowledge otherwise. The question is how reliable/robust is the error bound to outlier subjects/measurements.
review_title: null

Meta-review: This paper proposes a new approach to estimating conditional average treatment effects based on longitudinal data. Overall, authors were responsive to criticism raised by reviewers. However, prior to publication two critical revisions should be made. First, explicitly state the key assumption that if the latent confounders are able to predict pretreatment then they will also be able to predict post treatment (this is somewhat implied by proposition 3, but never explicitly stated). Stating this assumption clearly is critical to ensuring that approach is not inappropriately applied in the future. Second, the authors deviate from the norm in this community reporting MAE in lieu of PEHE (or MSE), without providing a justification. In the revision, please include both MAE and MSE results, or provide a clear justification for why MSE results are omitted. 
Decision: Accept (Poster)