Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: ## Summary: 
The authors develop a method to estimate the effect of a static treatment on outcomes over time under temporal confounding, with an emphasis on making the method transparent. The authors define transparency as (1) the ability to explain the estimate for a given individual as a weighted compination of a small number of other individuals, and (2) trustworthiness meaning the model should be able to identify cases for which it cannot give a reliable estimate due to a violation of one of the main causality assumptions. 

## Main comments:
I would significantly increase the rating if the authors address the following comments. 
 
(1) The authors tackle an interesting challenge. I am a bit concerned though that it trivially reduces to ITE estimation with a high dimensional outcome. It seems that the main novelty here is that the authors use a specific architecture that allows for varying lengths of pre-treatment variables, and hence confounders. My concrete questions are (a) suppose for a second that the outcome is measured for a single time period, why does this problem not reduce to the typical ITE estimation? Conditional on all the time varying confounders would simply be equivalent to the typical ITE estimation with a high dimensional set of pre-treatment variables. 

(2) The authors only model y(0) for individuals who received the treatment, and define the ITE to be the difference between the observed outcome and \hat{y}(0), and similarly for control individuals. This means that in order to use this model i.e., in order to obtain the ITE estimate for a new test patient, we need to first observe the outcome under t = 1 or t = 0. This makes the model not useful for making treatment decisions: it would only be useful to validated that some chosen treatment choice was good/bad. Can the authors clarify what the intended use for this model would be?

(3) d_i^y is simply a test of whether or not we're able to model the outcomes under t = 0. I do not follow the logical jump from "there is a large estimation error" to there is additional hidden confounding or the data generating model is not right. There are several other things that could cause that (e.g., the model can be well specified, and no hidden confounding but errors arise due to finite sample analysis). To be more concrete, equation 15 in A.1.2 assumes that 
(a) the SEM is correctly defined. 
(b) Q, U are correct
Either of these might be violated in practice. In fact, they will likely be violated in practice. The flip side is true, one can get a small estimation error and still have hidden confounding. In general, the assumption of no hidden confounders is statistically untestable (meaning it is impossible to design a statistical test that will answer this question). Can the authors highlight what I am missing here?

## Minor comments: 
Addressing these would be helpful but not consequential for the rating
(1) The authors make a statement that in general the error in ITE should be bigger than the error in y --> this is not true, for example it does not hold in situations where there is a large bias in opposite directions for estimates in \hat{y}(1), and \hat{y}(0). My comment here assumes the authors are operating in the "usual" setting where we don't observe the treatment and hence the outcome under treatment for a new test case, and need to estimate the ITE by estimating both potential outcomes. 

(2) In eq 5 m_{is} is not defined anywhere

(3) The authors don't make a clear case (empirically or in the writing) as to why their chosen approach is transparent. This b vector can still be very high dimensional even with an imposed sparsity. In a sense, all what the authors do with this b vector can be done using a simple kernel approach. Would the authors be able to explicitly state how is their model uniquely equipped for transparency?

(4) It is somewhat odd to have a causal graph where there is no arrow between the treatment and the outcome. After doing a bit of mental gymnastics, I arrived at the conclusion that this graph is assuming do t= 0, which makes sense but is highly unconventional, and confusing. It would be helpful if the authors follow the conventional notation for causal graphs, which does not condition on a particular treatment assignment (e.g., see graphs in publications by Pearl, Eric Tchetgen Tchetgen, Ilya Shpitser, Tyler VanderWeele) 

review_title: Interesting concept, unclear use cases, and some technical confusions. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper provides an approach for treatment effect estimation when the observational data is longitudinal (with irregular time stamps) and consists of temporal confounding variables. The proposed method can be categorized under the matching methods, in which, in order to estimate the counterfactual outcomes, a subset of the subjects in the opposite treatment arm (i.e., contributors) is selected and weighted. The proposed method is designed such that it achieves explainability (by identifying a few contributors) and trustworthiness (by checking if the estimated outcome is reliable).

Pros:
- The paper presents a simple, yet reasonable solution to the task at hand. 
- The paper is easy to read and understand.
- Figures were very helpful in understanding the paper; especially Figure 2.

Cons:
- There are lots of design choices that are not motivated. Specifically, in section 4.1, the paragraph on Architecture includes too many design choices and heuristics that lack proper explanation and / or intuition. For example, it is unclear why there is need for a new time aware representation $o_{is}$ while we already do have one, i.e., $h_{is}$? Also, $o_{is}$ is mistakenly called a *projection* (in linear algebra terms).
- There are many missing details that are crucial for understanding the algorithm. For example, in Equation (5), it is not stated which parameters are optimized by each loss. I’m guessing $h$, $r$, and $Q$ are found by $\mathcal{L}_s$; and $\[k_0, k_1\]$, $\[s-0, s_1\]$, and $g$ are found by $\mathcal{L}_r$; right? And how are they optimized? Do you alternate between the two objectives? How about $\mathcal{L}_m$ in Equation (6)?
- The authors claim in section 3.1, paragraph 1, lines -6 to -4 that their model does not over-match however, $\mathcal{L}_r$ exactly does that.
-To achieve trustworthiness, the authors state that they minimize $d_i^y$ in Equation (4); however, this term does not show up in any of the objective functions --- see Equations (5) and (6). Also, there should be $M$ values of $d_i^y$; are you aggregating these values into one (e.g., via averaging) or …?


Minor:
- Section 2, paragraph 2, line 3: $t_{is} < 0$ is only for pre-treatment samples. Here, the discussion is on both pre- and post- treatment.
- Section 2, paragraph 5, line -4,-3: $v_i = 0$ and $v_i \neq 0$, did you mean empty set $\emptyset$ instead of $0$?
- Section 3.1, paragraph 1, line 1: Synthetic Control should be title-case.
- Equation (2), the subscript for $\widetilde{c}$ should have been $j$, not $i$.
- Section 4.3, paragraph 1, line -2: $M$ in $\mathcal{L}_M$ should be lower-case.

review_title: Some claimed contributions seem to be missing from the final proposed algorithm.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: I'm going to keep this review short because I thought this was a really well-motivated and executed paper. The paper builds on the 'synthetic control' approach that is popular in econometrics which uses a weighted sum of the outcomes of individuals in the control group as an estimate of the control potential outcome, which can then be compared to a given treatment outcome to estimate the individual treatment effect. The step is the choice of weights: Sync twin uses the same prediction approach, but constructs the weights for the weighted sum using hidden representations from a recurrent network. This lets them work with irregularly spaced observations & missing covariates in a natural manner. 

My only substantive complaint is in the framing of the paper: the introduction describes this temporal setting as more challenging than the static setting, but given that the paper is not about dynamic treatments (which are indeed far harder to deal with), the fact that you see multiple observations for any given individual makes the problem easier not harder... To be clear - this is not bad thing, but rather than saying, "there are many methods for the static setting, but few are able to address temporal confounding" (which isn't really true - these approaches could easily be adapted to the single treatment temporal setting using the appropriate recurrent architectures) - instead emphasize the fact that the static approaches don't take advantage of multiple observations from the same individuals. Multiple observations is a *stronger* assumption than the static setting---albeit a very natural one in this domain that you leverage for trustworthiness, etc.---so it shouldn't be presented as though its a weaker assumption / harder setting... 

review_title: Excellent paper on adapting synthetic control-style approaches to time-series data

final_score: rating: 4: Ok but not good enough - rejection
review_content: - pros
    - The problem is well motivated. Estimating the causal effect with time-series data is a practical and important problem. Transparency is a desirable property. 
    - The idea connects works in econometrics to modern ML. 
- cons
    - High level points
        - The paper proposes to use the number of donor units as a metric for explainability. It is not clear why this matric is good a priori. It is possible that the network selects an arbitrary set of units to construct the control.  
        - For trustworthiness, It is quite confusing how synctwin is supposed to identify individuals whose ITE cannot be reliably estimated. This paper did not provide any formal procedures or guarantees on when identification is possible.
        - Low rank is a key assumption for synthetic control. A discussion on whether this assumption is required for synthetic twins to work would be useful.
        - The experiment section could use more clarification. In the synthetic experiment, the paper compares against a number of benchmarks (including robust synthetic control ) for model performance. However, to show that the method uses fewer contributors, the paper only compared against synthetic control, even though robust synthetic control is a more natural comparator, as it explicitly isolates a small subset of contributors during its data pre-processing.
        - In the real world dataset, 1) while the train. test split is a common practice in ML, it doesn't generalize to inference. Namely, when the estimand is the ATT, we should not use just 1/3 of the data. 2) It is unclear how the ATT is measured.  Is it the average difference at the end of the trial? If so, it may not be a good metric to evaluate the time series model, since it's only the difference in the point estimate. 
    - minor details
        - in figure 2, is there a missing arrow between the treatment and the outcome?

review_title:  - This paper develops SyncTwin, a method for causal inference with time-series data. The main selling points of the paper are explainability, a small number of donors are used to construct the control, and trustworthiness, the method can identify individuals whose ITE cannot be reliably estimated. 

final_score: rating: 7: Good paper, accept
review_content: I enjoyed reading this paper and think it tackles an important problem. The paper works with a particular model of the data generation and propose to do synthetic control in representation space which ensures that the considered data generation/model are not inflexible.

I list a few issues here; they are mainly about the discussion of the assumptions.

There is very little discussion about identification guarantees in the paper. While A.1 provides some justification in this regard, I would've liked to see a paragraph about the identification guarantees for the proposed algorithm.

It seems to me that the functional linearity assumption of the model is what allows the method to not require sequential overlap. Could the authors expand on this?

I believe that the key requirement of the method is the linearity of the control outcome in the latent c_i even when it is a non-linear function of the covariates; I think this helps the method not face serious non-identification issues. Could the authors clarify this intuition?

I liked the idea of using the control outcome as a way to understand whether there is unobserved confounding as per the modelling assumption. The authors should however clearly state that no-unobserved-confounding is an assumption they also require to estimate effects; effect estimation is in generally impossible if v_i exists and is unobserved. 

I liked the experimental section also. The consistently better performance (even over the ablated models) provided some good evidence for the merit of the method. The real experiment (and the transperancy discussion) provided further strong evidence in favor of synctwin.

Finally, it would've been nice to see one more synthetic experiment but seeing that the data generation considered in the paper is a biologically informed model (not a toy one), I'm happy with them for now.

I'm happy to increase my score if the discussion around assumptions is sufficiently clarified.
review_title: interesting work on synthetic control on representations; requires some expanded discussion

Meta-review: 
The paper developed a method that estimates treatment effect with
longitudinal observational data under temporal confounding. It extends
the idea of the synthetic control method and offers flexibility and
ease of estimation. However, some major concerns remain after the
discussion among the reviewers. In particular, the proposed method
lacks a clear use case. Moreover, some arguments around
``trustworthiness`` (detecting unreliable ITE estimates) and ``avoid
over-matching`` need to be refined. The error bound for
``trustworthiness`` can not detect hidden confounding. For overlap
issues, the rejection of units with larger error could be overly
conservative because the error bound may often be too loose. Regarding
``avoid over-matching``, while SyncTwin uses a low-dimensional
representation as opposed to the whole x vector for matching, it is
unclear whether SyncTwin can avoid over-match. It is possible that
using low-dimensional representation makes it easier to find a match
in the data and may still over-match. Finally the paper would benefit
from proper causal identification results.
Decision: Reject