Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper considers the problem of optimally partitioning a panel into treatment and control groups, leveraging past observations to ensure a more balanced allocation between treatment & control.

Generally randomized-controlled trials are the standard in this instance, because particularly with large enough populations and samples the results will be robust to any issues in picking control groups. This paper strives to provide an approach when the sample size is not large enough to provide a suitably power RCT. The alternative approach is inspired by work in synthetic controls, where ex-post one may determine a set of weighted controls to use for a single potentially-unplanned treatment. 

The setting in this problem lies somewhere between the two - applicable for a small handful where perhaps tuning treatment/control can give additional experimental power, at some robustness cost. 

The approach shown draws heavily from the approaches in the synthetic control literature, formulating three approaches: 1) per-unit, wherein effectively synthetic controls are formulated for each treatment - or at least weightings are done accordingly, 2) two-way global, wherein weightings are only for each variable rather than the pairing - so one weighted treatment set, one weighted control set, and then 3) one-way global, with unweighed treatment and a weighted control. Across all, the experimenter is minimizing RMSE.

The paper formulates each as a mixed-integer optimization problem and does a simulated experiment on unemployment statistics across the 50 states (in 40 time periods), with either a fixed or linear additive treatment effect.

The paper does not include justifications for expected conditions on the randomness that would lead to this approach behaving better than an RCT (though it is cited as a future concern). 

main_review: The paper tackles a seemingly interesting middle ground between synthetic control and RCTs. The approach put forward derives its approach from a generalization of the synthetic control literature. I am not familiar enough with the applicable literature and area to characterize novelty among the most related papers.

Generally the paper is well written, clearly communicating the approach. Implementing the approach would seem not hard to follow. Optimally solving the IP is shown to be NP-hard. Since the approach seems like it is generally of use when the 

The justification for actually using the approach however is a little lacking. Implicit in the behavior is a reliance on the pre-treatment outcome data as a reliable predictor of correlations in treatment effect. Further justification of when this method should be chosen above RCTs would be helpful, either with a theoretical discussion of conditions on the relationships between the randomness. It would seem that the approach would be at its best when there is additional structure that is uncovered and leveraged. The experiment performed has some of that implicitly since there must be that uncovered structure in employment data, but it would be good to see further discussion. Additionally, implicit in some of the discussion is the reasoning that you would want to use this approach when panel size is small and thus an RCT may not have enough power. This tradeoff is not explored, and it would be great to see estimates of where that tradeoff is or a comparison. Since RCTs are chosen for intrinsic robustness, this may be challenging to quantify, but showing something about the sweetspot for this approach as the data size increases would be helpful.

limitations_and_societal_impact: Using this approach rather than RCTs does potentially expose experiments to a loss of robustness. I am sure that the authors would not advocate a large scale shift to these, but a little more discussion of what to be concerned about if interpreting the results from an experiment that was generated using this approach would be helpful.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose and analyze a new problem at the intersection of experiment design and synthetic control: in a panel data setting (many units i=1:N observed over time t=1:T), how should treatments be assigned and how should weights be computed at time T such that counterfactuals are imputed well at time T+1?

The authors interpret the question three ways, providing three mixed integer programming problems. The authors compare the quality of counterfactual implementation of these new experiment designs with difference in means (where neither treatment assignment nor weights are optimized) and synthetic control (where treatment assignment is not optimized).
main_review: Overall assessment

Originality: The problem studied by the authors appears to be original, and the results are quite cleanly derived. Posing and analyzing a new, interesting problem is the hallmark of a good paper. Can we confirm that this problem has not been previously studied?

Quality: I would like to see the permutation-based inference. Otherwise the paper is interesting but light on theory. It is also light on references to an exponentially growing SC literature.

Clarity: The paper is well written. Clarifying earlier on and more formally the difference between standard experiment design, SC, and this problem would be an improvement. How do the objectives compare to regularized SC objectives? Can the authors thereby draw on such results, or contribute to those existing literatures?

Significance: The paper is significant for proposing a new problem to study. Its significance within causal inference and policy evaluation would dramatically increase if it included the permutation inference result.

Suggestions to improve the paper

105: It would help to begin using the language of ATET earlier, since that is the causal estimand

108: This is an MSE conditional on treatment assignment and weights. Please make the conditioning explicit. Please also derive this result with a proof in the appendix (it is correct, but not easy on the reader)

113: empirical analogue in the pre-period on average

114: walk the reader through the logic. In the pre-period, Y_{it}=Y_{it}(0)=mu_{it}+e_{it}. Alternative, present the proof in the appendix

114: Strictly speaking, the latter term should be \sum_j (1-D_j) (w_j^i)^2, but the optimal w_j^i=0 if D_j=1 interpreting the former term as fit and the latter as a ridge penalty

114: Please compare this objective to the literature on regularized SC, e.g. as reviewed in Abadie and L’Hour (2018)

123: for the second term, index the sum by j

125: see note above about conditional MSE

127: Nice

141: See comment above. Strictly speaking, the sum in the second term would be over D_i=0

151: Please summarize these constraints in the main text

164: How does this relaxation affect the geometry? C.f. Delauney tessellation

165: Please maintain the parallelism of the main text: per unit, then two-way, then one-way

180: Nice

210: Nice. It would help to mention this comparision of difference in means and SC to the problem of interest earlier on, ideally as different optimization problems in Section 2

287: This is the obvious way to improve this paper. It seems the authors even know how to do it. Including this result would make this paper excellent
limitations_and_societal_impact: What assumptions could be violated, and how would that impact conclusions and policy recommendations?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies an approach for designing experiments in panel data settings. Based on data from T time periods, the proposed approach selects weights and treatment assignments for a subsequent time period to minimize an empirical estimate of the mean-squared error of an individual-level or weighted average treatment effect. 
main_review: Originality: As far as I'm aware, the proposed approach is novel.


Quality:

1. The objective functions that you focus on are inspired by the mean-squared errors (MSEs) of the form $\mathbb{E}[(\hat{\tau}_i-\tau_i)^2]$ or $\mathbb{E}[(\hat{\tau}-\tau)^2]$. However, unless I'm missing it, no theoretical guarantees are provided for your proposed objective function. Can you show that the solution to your empirical objective function is (nearly) as good as the solution to the true MSE $\mathbb{E}[(\hat{\tau}_i-\tau_i)^2]$ or $\mathbb{E}[(\hat{\tau}-\tau)^2]$? Presumably you end up with guarantees as $T\rightarrow\infty$? What about if $T$ does not grow but $N$ does? What happens if neither of these quantities grows? Such a guarantee would go a long way in supporting the use of your objective functions.

2. As you've pointed out, the weighted average treatment effect on the treated (wATET) can be a weird estimand to look at when you're deciding the weights based on some MSE-type criterion. The oddity arises because, when the wATET is of interest, the weights that minimize this criterion decide the estimand, and so even if the chosen weights result in an easier estimation problem (in terms of MSE), it's possible that this estimation problem may not actually be of interest. You also pointed out that this problem goes away when treatment effects are homogeneous and, when they're not, you suggested instead solving a different, one-way global optimization problem. Though this problem seems to resolve the problem of the estimand being decided by the choice of weights, it seems to me that the (unweighted) estimand pursued by this problem is still influenced by the chosen solution of the problem. In particular, the solution to the two-way optimization problem selects individuals to treat, and the average treatment effect on the treated (ATET) estimand is clearly influenced by who receives treatment. Therefore, unless I'm misunderstanding something, it seems like the so-called optimal design still leads to a challenging interpretation wherein the estimand is chosen based on a criterion that has nothing to do with subject-matter expertise, but instead based on making the resulting estimation problem as easy as problem. If this is correct, then, at a minimum, some discussion of the consequences of doing this would be warranted.

3. Related to the above, the experiment section says that:

> We either assign each treated unit the additive treatment effect of 0:05 (homogeneous treatment case) or assume that the first—as determined by the row number—selected treatment unit receives the additive treatment effect of 0, the last unit receives the additive treatment effect of 0.1, and the remaining treatment units receive the treatment effects spread linearly from 0 to 0.1 according to the order determined by their row numbers (heterogeneous treatment case). The average treatment effect on the treated remains 0.05 in both cases.

This text appears to suggest that the choice of estimand does not, in fact, depend on the treatment allocation. However, I fail to see how the above actually fits with the sampling scheme described in Section 2. It seems that the treatment effects should be decided before individuals are treated, making the above-described mechanism impossible --- but perhaps there's something that I'm not understanding.

E.g., we could have a population of 6 people with potential outcomes on control always equal to zero and potential outcomes on treatment (and, also, individual-level treatment effects) equal to 0, 0, 0.05, 0.05, 0.1, 0.1. If we treat 3 people, we could choose individuals with effects (0,0,0.05), leading to an average treatment effect among the treated (ATET) of $0.05/3\approx 0.017$, or individuals with effects (0,0.05,0.1) leading to an ATET of $0.05$, or we could choose individuals with effects (0.05,0.1,0.1), leading to an ATET of $0.067$.
In contrast, the way you've described your data-generating mechanism here, it seems like the three treated individuals will always have treatment effects (0,0.05,0.1), and therefore always having an ATET of 0.05. 


Clarity: 

1. At the beginning of Section 2, you should immediately make clear whether $Y$ will be treated as random. From what's written at the top of page 3, it looks like the answer is "yes", and furthermore that the noise is homoscedastic. I think it's important to mention this upfront, since otherwise it's unclear what, exactly, $Y$ is.

2. It seems that most of the paper focuses on the case where $S=T+1$. This makes me wonder whether it was worth introducing notation for $S$ at all, or whether, to reduce notation, you could have just focused on this special case from the start.

3. In the top paragraph of page 3, the text first says that $i=N$ and then later in that paragraph says that $\hat{Y}\_{N,T+1}=\sum_{i=1}^N w_i Y_{i,T+1}$, leading to an overload on $i$. The latter $i$'s should be replaced by $j$, which would avoid the overload and also be more consistent with the notation used in the following paragraph.

4. Another subsection should be added to Section 2, or the \subsection tag should be removed above Section 2.1.
limitations_and_societal_impact: yes
review_title: null

Meta-review: The expert reviewers for the most part appreciated the paper and were guardedly positive. The paper is commended for posing an interesting new question. At the same time there were concerns about the relevance of the estimand targeted as well as formal guarantees. The authors suggested possible ways to address this that should be incorporated into the paper. Crucially, the permutation inference result would add an important aspect to the paper that would merit its acceptance. Moreover, the authors should give a detailed discussion about their ATET estimand and explain carefully its limitations in the absence of homogeneity, which is the practically common setting, and how possibly slight violations might affect the interpretation of the results.
Decision: Accept (Poster)