Synthetic Design: An Optimization Approach to Experimental Design with Synthetic Controls
Authors: Nick Doudchenko
Khashayar Khosravi
Jean Pouget-Abadie
Sebastien Lahaie
Miles Lubin
Vahab Mirrokni
Jann Spiess
Guido Imbens
Venue: NeurIPS 2021 Poster (2021)
Keywords: experimental design
synthetic control
causal inference

Abstract: We investigate the optimal design of experimental studies that have pre-treatment outcome data available.  The average treatment effect is estimated as the difference between the weighted average outcomes of the treated and control units. A number of commonly used approaches fit this formulation, including the difference-in-means estimator and a variety of synthetic-control techniques. We propose several methods for choosing the set of treated units in conjunction with the weights. Observing the NP-hardness of the problem, we introduce a mixed-integer programming formulation which selects both the treatment and control sets and unit weightings. We prove that these proposed approaches lead to qualitatively different experimental units being selected for treatment. We use simulations based on publicly available data from the US Bureau of Labor Statistics that show improvements in terms of mean squared error and statistical power when compared to simple and commonly used alternatives such as randomized trials.

Synthetic Design: An Optimization Approach to Experimental Design with Synthetic Controls
Nick Doudchenko" Google Research New York, NY 10011 n nikolayd@google.coom com
Khashayar Khosravi" Google Research New York, NY 10011
Jean Pouget-Abadie Google Research New York, NY 10011 jeanpaggoogle.c com
thosravi@google.com
Sebastien Lahaie
Miles Lubin
Vahab Mirrokni
Google Research New York, NY 10011
Google Research New York, NY 10011 ml mlubin@google. com
Google Research New York, NY 10011
sl slahaie@google com
mirrokni@google com
Jann Spiess Stanford GSB
Guido Imbens Stanford GSB Stanford, CA 94305
Stanford, CA 94305
jspiess@stanford. edu
imbens@stanffrdd edu
Abstract
We investigate the optimal design of experimental studies that have pre-treatment outcome data available. The average treatment effect is estimated as the difference between the weighted average outcomes of the treated and control units. A number of commonly used approaches fit this formulation, including the difference-in. means estimator and a variety of synthetic-control techniques. We propose several methods for choosing the set of treated units in conjunction with the weights. Observing the NP-hardness of the problem, we introduce a mixed-integer pro- gramming formulation which selects both the treatment and control sets and unit weightings. We prove that these proposed approaches lead to qualitatively different experimental units being selected for treatment. We use simulations based on pub- licly available data from the US Bureau of Labor Statistics that show improvements in terms of mean squared error and statistical power when compared to simple and commonly used alternatives such as randomized trials.
Introduction
Randomized experiments have long been a staple of applied causal inference. In his seminal paper, Rubin 1974) suggests that "given a choice between the data from a randomized experiment and an cquivalent nonrandomized study, one should choose the data from the experiment, especially in the social sciences where much of the variability is often unassigned to particular causes." Using the language of Rubin's s pntential-outcome status is independent of the potential outcomes and that simple and intuitive estimator that compares the average outcomes of the treatment and control units is an unbiased estimator of the average treatment effect (ATE). If both the treatment and control samples are sufficiently large, the hope is that this lifference-in-ment estimate is close to the population mean of the treatment effect.
framework, randomization guarantees that the treatment
Equal contributions.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Another crucial property of randomized experimental designs is their robustness to alternative assumptions about the data generating process-4 a completely randomized experiment does not take into account any features of the observed data. Perhaps not surprisingly, when the researchers are willing to incorporate additional probabílistic assumptions in their design decisions, they can improve on the statistical properties of the average treatment effect estimators (see, for example. |aasy, (016)) These improvements, however, do not come for free and the performance of the estimators may suffer if the incorporated assumptions are violated (Banerjee et al. 2020).
For these reasons randomized experiments are widely used in academic and clinical settings, as well as industrial applications. However, not every practically important question can be easily answered using an experiment on large sample of experimental units. For instance, the evaluation of major policies targeting large geographic areas has long been of interest in social and political sciences. One of the traditional approaches to compare the affected unit-such as a metropolitan area or state to the average across sample of carefully picked control units which are deemed to be suitable comparisons (Card] 1990p. more recent alternative, called synthetic control, first introduced by AAbadie and Gardeazabal (2003) and later developed in Abadie et al. (2010) and AAbadie et al. (2015), compares the unit of interest to a weighted average of the units unaffected by the treatment, where the weights are selected in way that achieves good fit on pre-treatment outcome variables as well as potentially other observed covariates.
While originally developed by academics for evaluating the effects of policies, approaches similar to the synthetic- control methodology have recently gained popularity industry as well in cases when applied researchers decide to run experiments targeting larger units often representing geographic areas. This decision may be justified when more granular experiments are either unavailable (for example, television advertising can only be targeted at the media-marke level) or are unlikely to capture the relevant effects due to interference or equilibrium concerns (Sobel, 2006 Rosenbaum 2007 Hudgens and Halloran 2008). For instance, company like Uber may want to evaluate some of the possible treatments at the market level rather than at the driver level the treatment in question is likely to affect the driver supply. Moreover, launching an experiment targeting even single unit may be so expensive that the researchers try to minimize the required number of treated units. Privacy and fairness concerns may also make treatment assignment
more granular level problematic.
Synthetic control and similar approaches may be attractive as estimation procedures in those cases, but they fail to address the equally if not more important aspect of the optimal choice of experimental units (see, for example, Rubin et al., 2008). We attempt to narrow this gap the current paper. We consider a panel-data setting in which the researcher observes the outcome metric of interest for number of units in a number of time periods and has to decide: (i) which units to experiment on and (ii) how to estimate the treatment effects after collecting the outcome data in the experimental time periods. The main difference between this setting compared to a typical synthetic-control study is that the treated units are not fixed, but rather chosen by the researcher. Proving that the underlying optimization problem is NP-hard, we rule out the possibility of designing polynomial moooial-timee algorithms for the problem under P*P.. Therefore, we formulate this combined design- resign-and-and and-analysis problem as a mixed-ini -integern program (MIP). Depending on the particular estimands of interest, we propose one of the several formulations and discuss their advantages and drawbacks. We motivate the choice of the optimization objectives and discuss the selection of experimental units each of the objectives leads to. The MIP formulation allows for an easy inclusion of additional constraints long as those are linear. For instance, it is easy to restrict the overall number of treated units, exclude specific units from treatment, or enforce a budget constraint if there a varying cost treat different units.
Using publicly available state-level unemployment data from the US Bureau of Labor Statistics, we compare the proposed methodology to randomized design that utilizes either the conventional difference-in -n-msans estimator or the symthetic-control approach. We estimate the average as well as individual state-level treatment effects in a simulated experiment and find that our approach substantially reduces the root mean squared error (RMSE) of the estimates. We show that our MIP-based aansal-snd-anallsis procedures consistently outperform the more traditional baselines regardless of whether the treatment effects are homogeneous or heterogeneous and whether the number of units selected for treatment is small or large relative the total number of units in the sample. We also suggest a permutation-bast -basdd inference procedure that follows Chernozhukov et al. (202T). We verify in our simulations that this procedure leads to correct test sizes and improved statistical power for testing the sharp null hypothesis of zero treatment affects across all treated units,

---

when used in conjunction with the proposed estimators. We provide theoretical guarantees albeit ensure that the proposed tests have proper sizes.
under rather strong assumptions
To our knowledge, this is one of the first papers to study experimental design in the context of synthetic control and adjacent estimation techniques. [Doudchenko et al. (2019) consider the case when the underlying effects are homogeneous and only a single unit can be treated; they suggest searching for the unit that delivers the highest statistical power when testing the hypothesis of zero treatment effect with an artificially applied treatment. In an independent work. AAaaie and Zhao (202T) also consider optimal design in settings where synthetic-come control estimators are used for estimation. However, there are few important aspects that differentiate the current paper from theirs. First, Abbsii and Zhao (202I) use both pre- treatment unit-level covariates as well as the outcomes from some but not necessarily all-of of the pre- pre-treatment periods whereas we focus exclusively on the outcome variable and utilize all of the pre-treatment outcome data. As result, the inference procedure proposed by Abadie and Zhao (202T) does not apply to our approach. Second, we discuss formal hardness results. Finally-and, perhaps, most importantly (Abadie and Zhao (202T) choose an objective function that fixes a priori the target average treatment effect, as either the population average treatment effect or weighted version thereof. In contrast, the objective function we use focuses on the average treatment effect only for the units we choose for the treatment, making the target estimand stochastic, but potentially easier to estimate and requiring different assumptions.
The rest of the paper is organized as follows. Section 2 introduces the setting and the proposed estimation approaches.
the mixed-integer formulation of the suggested estimators. Section presents some of the theoretical unit-selection results and the intuition behind them. Section 5 reports the empirical results obtained through simulations. Section [ discusses some of the practical consideration that should accompany applied work that uses the proposed methodology. Section 7 concludes and outlines the directions of future research.
2 Setting
Let the researcher observe the outcome metric of interest, Y, for N units during T time periods, such that the observed data can be represented as an x T matrix of values Ya. At time = the researcher decides-based on the data observed up until that point-which units should be treated and assigns a binary treatment described by variables D; E {0, 1}.
I...., N. The outcomes then observed for additional S - T time periods / T + 1, ..... and the treatment-efffect estimates are constructed. Each unit i = 1,..., N in each time period / = T +++1, 1,..., ,S is associated with two potential outcomes (Y (0), (1)) which are considered random. The potential outcome (1)()) is realized if D. = 0 and (1) is realized if D. = so that the observed outcome is Yu Yu (D.) = (0)(L - D.) + (I)
Recall that, given a setting where a single unit i = N has received the treatment in a single time period t = T + 1 = SS the synthetic-control literature (Abadie et al.. 2010, among others) suggests constructing a counterfactual estimate for unit N as a weighted average the other units' observed outcomes: (0) = N W,Y,,44- In the previous equation, the w; 's are weights learned from the data observed during the pre-treatment periods = 1,2,..., T, often by minimizing E.= - N-1 w: )² under some constraints on the weights. Assuming that the treatment effect, TN, for this unit is additive, it can then be estimated as îN- = YN,T+1 - Yx.r+1(0).
We now consider more general setting where K units have received the treatment, with outcomes given by
(o)( - Pat + Ea and Ya Ya) + D
with homoscedastic noise Ext that has mean zero and variance x² and additive treatment effects T;. In order to estimate the treatment effect in this more general setting, we can apply a separate synthetic control method to each treated unit i, learning an appropriate set of weights [w]) for each treated unit
i individually: (0)) Ej D, w/ y,T+1 The treatment effect for unit is then estimated as f, = -YYT++1 (0).
3

---

Rather than considering each reight-fitting optimization problem separately, we can express the (conditional) mean squared error (MSE) of the resulting estimator of the treatment effect as:
2 + +
(f: - -1) m
P1,T+1
Σ 3: D;=0
1+
Σ J: D;=0
(w/)²
A proof is included in the supplementary materials. The synthetic-control literature often operates in settings where the treated units are given. Here, we allow the experimenter to select which units should receive the treatment. The mean-squared-error formula above leads us consider the following optimization problem over the weights {w and the treatment variables {D: }
under some appropriate constraints on the weights {w] J.y -i The objective below can be seen the empirical analog of the right-hand side the population equation above in the pre-treatment period averaged across time and across the treated units.
N
T Dj
N
min (D.1w))ALE.
Yu
Σ j=l
w1 (1-1)))Kyr
D
(w))²
KT
(per-unit)
Note that we can safely sum across all units, not just the control ones, in the second term since the optimal values w for units j with D, will be equal zero in an optimal solution.
In essence, this optimization problem attempts to minimize the discrepancy between the pre-treatment outcomes of the units chosen for treatment and the weighted averages of the outcomes of the units left as controls. At the same time, due to the second term, the objective attempts to balance the unit weights themselves.
The term o² unlikely to known to the researcher and should be chosen based on the observed data. One possible way is to set it equal to the sample variance of the observed outcomes. We further discuss selection of the penalty parameter Section [] Penalizing the weights is not uncommon in the synthetic-contrr literature. For example. Doudehenko and Imbens (016)) the elastic-net penalty on the weights and Abbadi and L H'Hour (2021) introduce a lasso-style penalty in which the nonnegative weight, w is multiplied by the squared distance between the vectors of the covariates used for matching the units. This way, depending on the magnitude of the penalty hyperparameter, they can balance between the synthetic-control fit and the nearest-neighbor fit that puts all the weight on unit j closest to i in terms of the observed covariates.
In many applications, the researcher may be interested in estimating some weighted average of the unit-level treatment effects on the treated units. Rather than considering each treated unit separately and then computing the weighted average of the estimated individual treatment effects, practitioners may wish to construct a muutheti--oonrol-yye estimate for the weighed average of the treated units directly.
N Consider set of treatment assignments [D,) and weights { on outcomes time T + and assume that we wish to estimate the weighted average treatment effect on the treated (wATET),
T= E.: D:=1 WyT; = = Drw,TE, difference in weighted means: + -= - D.=1 wi W.YYTT+ En D:=0 wy Yi,++1 Then, under the same outcomes model presented above, the (conditional) mean squared error of the cenpreenee-iinnwiwweeeeee means estimator is
N E [(( ((f--)* 1)1-7))1(1; wi |==1
Σ D;=1
Σ D:=0
WiIGT+11
Willi
+
w
As before, our setting allows the experimenter to optimally select which units should receive the treatment as well as which particular weighting scheme should used. This is especially appropriate when the treatment effects are homogeneous and т; 7 for all i = 1,..., N. In that case, any weighted average of the unit-level treatment effects is equal to 7 and the weights can be chosen in way that minimizes the mean squared error.

---

The population equation above suggests solving the following optimization problem on the weights NN and the treatment variables {Di) based on the data observed in periods 1...., [w
min (Diws.)
1 T
Σ Y t=1 D:=1
Σ waYa D,=0
(two-way global)
So far, we have not considered specific constraints on the weights, {w; {wi If > 0, w; =0 for all i I,...,,N the unique optimal solution to the two-way global problem above if the weights are not constrained in any way. In order to avoid this clearly undesirable solution, we assume that the weights { [w;]] J are normalized: Σ:; D.=1 W/ = =E; D,=0 W; 1. While not strictly required, another reasonable set of constraints motivated by estimating a proper weighted average is requiring all weights to be nonnegative, w, 2 0 for all i I....,..
Similar constraints can be imposed in the context of the per-unit problem: waf 2 0 for all i,j
1. I.....N N and Σj D, wj 1 for all ...... N such that D, 1.
Finally, the weights on the treated units may be fixed if specific weighted average for example, simple average with equal weighting-neee to be estimated. This constraint is particularly justified when the treatment effects are hetcrogeneous and different weighting schemes lead to different estimands. For that reason, we formulate another variation of the global problem:
min (D.,w,
T
Σ
W.Y,
Σ wiYK
+
(one-way global)
D,=1
D,=0
subject to an additional set of constraint that require wy = Wj for i.j = -....,,A N such that
D, D, =
is possible to obtain other design- design-and-estie and-mmtimttionn approaches as special cases of these problems- something that we utilize later in the paper. For instance, the standard difference-in-ment approach under randomized design can be viewed as the special case of the one-way global problem with the treatment indicators, D, set randomly and the weights on the control units restricted to be equal, uy = Wy for all i.j ...... N such that D; = Dj = Likewise, the synthetic-control estimator can be viewed as the per-unit problem with the treatment indicators set according to the experimental design of choice-for example, randomly.
3 Mixed-integer formulation
The three optimization problems introduced in the first part of Section 2 the per-unit the two-way global and the Jone-way global problems can all be formulated as mixed-integer programs. We now describe the specific optimization problems we use in the empirical section of the paper.
The per-uuitt is formulated as:
2
N D.
T
N Yu Σ
N ΣΣ
N
min IN HDriug
(1
D.
(w²)²
KT
1=1
3=1
s.t.
wj 2 0,
D, E (0, 1} for i,j = =.....,
D K, Σ (1 D;)= for i = 1,.... N such that D, 1.
The two-way global problem can be formulated as:
T
N
N wi(1 Di
2
N +
min
Σ
Σ
w.D.Ya
w.
Σ
(D
T
1=1
1=1
1=1
1=1
s.t.
w; 2 0,
D, E (0, 1} for i = -,...,,N
N
N
N Σ
ΣD = K, Σ
W.D, =
w/ (1 D.) =
1=1
i=
5

---

This becomes a one-way global problem we impose an additional set of constraints requiring that w; 1/K for all 1,..,, N such that D, = 1. When the unit weights are optimized (see the supplementary materials for the derivation), the optimal values of the objectives can be written in and the oon-way global problem is simply the two-way global problem with an additional set of constraints: w, = w for all i,j L...., N' such that D; DD
All three problems require additional auxiliary variables representing the products of the weights and the treatment indicators as well as additional constraints in order to have representation with a quadratic objective and only linear constraints so that they can be easier to solve by one of the academic or commercial MIP solvers See the supplementary materials for the exact formulations.
The term that is used in all three objectives a nonnegative penalty factor. Its selection discussed in Section
Note that the global versions of the problem can be solved without the constraint on the number of treated units, Σ D; = K, but the per-unit problem requires it-without the constraint the per-unit problem will tend to select fewer treatment units unless the objective is divided by Σ D; which introduces a nonlinearity. See the supplementary materials where we introduce an alternative formulation that allows to circumvent this by imposing additional quadratic constraint.
4 Design
The three optimization problems introduced in Section _ the one-way and two-way global and the per-unit problems- tend to select certain treatment umits in terms of their location within the distribution of the observed outcome data. this section we illustrate that behavior using simple example motivated by the objectives from Section 2 To simplify the analysis, we let 7 = 1 and denote a, Yı-- We also not restrict the unit weights to be nonnegative allow for simpler closed form solutions Specifically, we consider the following per-unit problem:
2
N Σ
min (Date IN
1 K Dj
Σ
a: Σ F=1
w1(l w
ΣV
Di(w))?
D; E {0, 1} i. = =L, I,...,N,
DD. D. K, (1 Dj) for I,...,N.
similarly simplified two-way global problem can be written
N
N
min (Demss
Daa;
Σ
w:(1
Di)a;
+
W
1
1=1
1=1
s.t
Dj E (0,1)
for i, = =I...., N,
N Σ D.= Σ Di 1,
Σ
w: D.) 11
1=1
closed-form as functions of the set of treated units, T.
Theorem 1. Let denote the set of treated units and =1,....,, NY. denote the set of control units. az)²
Let āj Eiet a:/\T be the average outcome within the treatment group. V? - quantity proportional to the sample variance of the outcomes within the treatment group, and the corresponding quantities for set I defined similarly. After the unit weights are optimized away, the per-unil objective can be written as:
(2(N~))) Jper-umie (I) = N N-K" K
(ar (ar-a7) K K- o² ++12
We use SCIP Gamrath 2020 when generating the empirical results in Sections and 5 'In some cases the nonnegativity constraint will not be binding. while in other cases the opumal weights without the constraint may actually turn out to be negative. This implies that only subset of all units will have strictly positive weights in the optimal constrained solution.

---

the two-way global objective can be written as!
(
1 + N-
(a: - a1)²
22 (1) =0² K and the one-way global objective can be written as:
K
o2 V7 V?
1 J1-way (1) = 0² ( N K
[ - kT) + 0² + V?
is evident from the objectives that all problems aim to select units in such a way that averages of the observed outcomes for the treatment and control groups similar- squared difference between the average outcomes within each group appears in the numerators of all three objectives. The problems differ in terms of how they approach the sample variances of the two groups. The per-unit problem maximizes the sample variance of the control units while minimizing that of the treated units-the term V appears in the denominator while the term V? appears in the numerator. The intuition for the latter is that the per-unit objective tries to simultaneously model each treated unit with combination of control units, while keeping weight variance small, so it is best to keep treated units as homogeneous as possible. The two-way problem attempts to maximize both (taking into account that they are, of course, interdependent), the sample variances of the outcomes of the control and treated groups both quantities appear in the denominator. The one-way problem maximizes the sample variance of the outcomes of the control units only, as the weights for the treated units are fixed-only the term V appears in the denominator.
To illustrate the same patterns visually, we solve per-unit problem and a two-way global problem for a simulated dataset with N = 25 units, T' = 2 pre-treatment periods and the outcomes Y; drawn from a standard normal distribution independently across and t. This allows 2-d plotting of the units in the space of observed pre- treatment outcomes (Yn, Y12). We select either K = 3 or K = 25 - 3 = 22 treated units and variance across all units). Figure ssoows the units selected by each of the problems.
À = En (Yn Y22 /2((N) (which is the average two-period sample
The results align with the intuition presented using the simple model with T 1 above. Specifically, the per-unit problem maximizes the spread of the control units which is particularly apparent from the plot corresponding to K = 22 while keeping the treatment units as close to each other as possible (see the plot for K = 3). Since the control and the treatment units have symmetric roles in the two-way global objective, the three treatment units selected by the problem when K 3 are the same as the three control units selected when = 22. The control and treatment groups have similar average outcomes and both groups are relatively spread out.
5 Empirical results
To evaluate the performance of the methods proposed in Section 2 we compare several design-and- estimation procedures: (i) the per-unit problem, (ii) the two-way global problem, (iii) the one-way global problem, (iv) the per-unit problem with randomly chosen treatment units, and (v) the standard randomized experiment which randomly assigns the treatment and estimates the average treatment effect on the treated as the difference in means between the two groups. It is important to note that approach (iv) is equivalent to using the caathetic-controy method for each randomly chosen treatment unit separately and then either averaging the unit-level treatment effect estimates or using the individual estimates directly. Taking this into account, comparing (i) (iv) amounts evaluating the role of optimal design in synthetic-control study, while comparing (iv) to (v) evaluates the synthetic-control approach relative to the difference-in means estimator.
To run a number of simulated experiments, we take publicly available data from the US Bureau of Labor Statistics (BLS) which contain unemployment rates of 50 states 40 consecutive months We run 500 simulations such that each simulation utilizes a 10-by-10 matrix sampled from the original
*The only difference compared to the traditional synthetic-control methodology used, for example, in Abadie al. (2010) is that no additional covariates are used and the weights obtained using the outcome data alone- the approach similar to the one taken in, for instance, Doudchenko and Imbens (2016).
'The data are available from the BLS website, but the specific dataset we use is taken from https:// www.o/ow///otttt.ooeo.e wwwpeeeppeeeeeeeeeeeeeeeeeeeeeee wttt////aaaaaaaatteeeeee httpe/urateeppppee csv
github. ctw../cco/eaaeee wwt.cooo/cc/eeeeee coot/sytth-aoe
7

---

Yz2
(a) per-unit problem, =
(b) per-unit problem, K 22
Y2
two-way global problem, K
(d) two-way global problem, 22
Note: The units are plotted in the space of observed pre-treatment outcome data, (Yn1, Yi2); "o' denote the selected control units and "+" denote the selected treatment units.
Figure 1: Units selected by the two-way global and per-unit problems.
50-by-40 dataset. Specifically, randomly select 10 units and the first time period. The remaining 9 time periods are the consecutive months that follow. In each simulation we treat units (equal to 3 in
of simulations and 7 in another) which are chosen based the data in the first periods-or chosen randomly in cases (iv) and (v)-and the treatment is applied in the last of the 10 periods. We either assign each treated unit the additive treatment effect of 0.05 (the homogeneous treatment case) or assume that the treatment effects increase linearly from 0 to 0.1 from the first unit in the (randomly selected) 10-by-10 10 matrix the last one (the heterogeneous treatment case). This implies that the true value of the ATET changes depending on the identity of the units selected for treatment. However, the (overall) ATE remains 0.05.
We estimate the average treatment effect on the treated as well as the unit-level treatment effects. Only the per-umit problem, (i), and the synthetic control, (iv), allow for nontrivial estimation of heterogeneous treatment effects while the remaining approaches estimate all unit-level effects as being equal the estimate of the average treatment effect the treated.
We then compare approaches (i)-(v in terms of the not-meen-square error (RMSE), where the squared differences between the true values of the treatment effects and the respective estimates are computed for each treatment period (and each treatment unit in case of the unit-level effects) and averaged. The square roots of these quantities are the RMSE's in question. Table the RMSE's averaged across all simulations.
The specific improvements in terms of the RMSE over the baselines (iv) and (v) the randomized synthetic control and the randomized difference-in-mene means-depend on the data and the true treatment effects. The main takeaways, however, are more general. The per-unit problem consistently outper- forms the other methods when the underlying treatment effects are heterogeneous. For homogeneous treatment effects the difference between the per-unit and the global problems either vanishes, the

---

Table 1: Root-mean eoot-mean-squre errors of the average and unit-level treatment effect estimates
Homogeneous trealment
K =3
K =
ATET RMSE
Unit-level RMSE
ATET RMSE
Unit-level RMSE
Per-unit
8.5
13.9
8.3
16.0
Two-way global
8.4
8.4
8.4
8.4
(iii)
One-way global
8.5
8.5
8.5
8.5
(iv) Synthetic control
9.7
15.9
10.3
19.0
(random treat.)
(v) Diff-in-means
12.1
12.1
11.5
11.5
(random treat.)
Heterogeneous treatment
K 7
ATET RMSE
Unit-level RMSE
ATET RMSE
Unit-level RMSE
(i) Per-unit
8.5
13.9
8.3
16.0
Two-way global
8.6
27.6
8.9
32.5
(iii) One-way global
8.5
27.6
8.5
32.5
(iv) Synthetic control
9.7
15.9
10.3
19.0
(random treat.)
(v) Diff-in-means
12.1
29.7
11.5
33.6
(random treat.)
Note: The reported RMSE's multiplied by 10 for readability. The values in bold are the lowest in the respective columns and correspond to the methods that perform best.
two-way approach starts outperforming the alternatives since any weighting scheme leads to the same value of the ATET. Moreover, in the homogeneous treatment case the global problems always outperform the baselines when estimating the average treatment effect the treated.
For the particular sinulations we run, the one-way and two-way global objectives provide improve- ments over the baselines that vary from 12% to 31% in the homogeneous treatment case and from 11% to 30% the heterogeneous treatment case when estimating the average treatment effect on the treated. The per-unit approach, (i), performs well across the board while being particularly effective when estimating the unit-level effects in the heterogeneous treatment case and providing improvement of over 13% relative to the synthetic control approach, (iv), when the number of treated units is small and 16% when the number treated units is large. is not surprising that the per-unit problem provides smaller improvement that case the donor pool of units that are used for comparison with the treated units is large relative the overall number of units and the probability that will not be able to find good synthetic comparison is relatively low. The situation different when we only have control units that are used for constructing synthetic outcomes for the remaining 7 treatment units. Section 4 provides an additional discussion of the optimal design when the number of treatment units changes. Neither is surprising that the per-unit approach performs poorly relative to the global problems and even the randomized difference-in means estimator (but not the synthetic control) when estimating unit-level effects the homogeneous treatment case. Since all treatment effects are the same, efficient to pool all the data together and estimate the constant effect on the full sample rather than estimating the same quantity for each unit individually. What important though, is the robustness of the per-unit approach which provides similar performance in the case of either homogeneous or heterogeneous treatment effects while the global problems and the randomized difference-in-nee perform poorly when estimating unit-level effects in the heterogeneous treatment case.
the synthetic control when K because in
more

---

6 Practice
There are number of practical considerations that need to be addressed when using the proposed design and analysis approaches.
Formulating the mixed-inteeer programs. Both the per-unit problem and the global problems can be formulated as mixed-integer programs with quadratic objectives and linear constraints. As discussed in Section 2 the per-unit problem requires either an additional linear constraint that fixes the number of treated units, K, or an additional quadratic constraint that allows optimizing over the number of treated units. In addition to that, the per-unit problem uses more variables that need to be optimized since sets of weights vary across treatment units allowing separate estimation of every unit-level treatment effect. This implies that the per-unit problem is generally harder to solve and is only tractable for a smaller number of experimental units, N, compared to the global problems.
Choosing the penalty factor. The penalty factor, X, used by all of the optimization problems can be chosen using cross-validation. Specifically, the pre-treatment time periods can be split into the consecutive training and validation time periods and the value of can be chosen by minimizing the RMSE over the validation period in simulated experiment that is similar to the one we conduct in Section [ An alternative approach motivated by the setting Section 2 Juses an estimate of the variance of the outcome variable. For example, the approach take in Sections 4 and computes the sample variances for every unit across pre-treatment time periods t = I...., T and then uses the average of those quantities across all units the penalty factor, X.
Quantifying the uncertainty. Most applied settings require evaluating the uncertainty in the obtained estimates of the treatment effects. We suggest the permutation-basee approach for testing the sharp null hypothesis of zero treatment effects across all treated units proposed by Chernozhukov ct al. (202T). See the supplementary materials for the detailed description of the proposed inference procedure as well theoretical result that guarantees its validity- albeit under rather strong assumptions-and the power curves constructed using a simulated setting similar to that from Section When the proposed procedure used in conjunction with the per-unit global problems it provides the correct (or conservative) test sizes and improves the power relative to the synthetic- control and difference-in-mees approaches.
Solving the global and the per-unit problems in their mixed-integer formulations becomes computationally burdensome as the total number of units increases, especially if the exact optimal solutions are required. In our simulations we were able solve problems for N = 50 units-which is meaningful threshold corresponding to the number of states, a typical experimental unit in wyttheic-conntoo--yye studies-ss on a single machine within hours. However, we prove that the underlying optimization problem is NP-hard (by providing reduction to the partitioning problem; for the exact proof see the supplementary materials), and therefore exact solutions to substantially larger problems are unlikely.
Computational complexity.
Conclusion
In this paper we evaluate the role of optimal experimental design in panel-data settings where traditionally the average treatment effect on the treated might be estimated using randomized design and the difference-ing iiteerence-i--mmaaee in-nn estimator. We propose several design-and-ang and-analysis procedures that can be solved as mixed-integer programs. Our empirical evaluations show that these procedures lead to substantial improvement in terms of the root-mean-squat error relative to the randomized difference- innee-ini-mmnnss as well as the randomized synthetic-contre approaches.
We discuss the roles that underlying assumptions about the nature of the treatment effects, the estimands of interest, and the computational considerations play when deciding which approach should be used. We propose permutation-based inference procedure that is shown to deliver the correct test sizes simulations. We also discuss practical considerations when applying this methodology well as its current limitations.
10

---

References
Abadic, A., A. Diamond, and J. Hainmueller (2010), Synthetic control methods for comparative case studies: Estimating the effect of california's tobacco control program. Journal of the American statistical Association 105(490), 493-505.
Abadie, A., A. Diamond, and J. Hainmueller (2015). Comparative politics and the synthetic control method. American Journal of Political Science 59(2), 495-510.
Abadie, A. and J. Gardeazabal (2003). The economic costs of conflict: case study of the basque country. American Economic Review 93(1), 113-132.
Abadie, A. and J. L'Hour (2021). A penalized synthetic control estimator for disaggregated data. Journal of the American Statistical Association (just-accepted), 1-34.
Abadic, A. and J. Zhao (2021). Synthetic controls for experimental design. arXiv preprint Stti:2208..21996
Banerjee, A. V., S. Chassang, S. Montero, and E. Snowberg (2020). A theory of experimenters: Robustness, randomization, and balance. American Economic Review 10(4), 1206-30.
Bottmer, L.,, G. Imbens. J. Spiess, and M. Warnick (2021). A design-based perspective on synthetic control methods. arXiv preprint arXiv:2101.09398
Card, D. (1990). The impact of the mariel boatlift on the miami labor market. ILR Review 43(2), 245-257.
Chernozhukov, V., K. Wüthrich, and Y. Zhu (2021). An exact and robust conformal inference method for counterfactual and synthetic controls. Journal of the American Statistical Association, 1-16.
Cieliebak, M., S. J. Eidenbenz, A. Pagourtzis, and K. Schlude (2008). On the complexity of variations of equal sum subsets. Nord. Comput. 14(3), 151-172.
Doudehenko, N., Gilinson, Taylor, and N. Wernerfelt (2019). Designing experiments with synthetic controls. Working paper.
Doudchenko, N. and G. W. Imbens (2016). Balancing, regression, differeeneeiinddiffeeecce and synthetic control methods: synthesis. Technical report, National Bureau of Economic Research.
Gamrath, G., D. Anderson, K. Bestuzheva, W.-K. Chen, L. Eifler, M. Gasse, P. Gemander, A. Gleixner, L. Gottwald, K. Halbig. G. Hendel, C. Hojny, T. Koch, P. Le Bodic, S. J. Maher, F. Matter, M. Miltenberger, E. Mühmer, B. Müller, M. E. Pfetsch, F. Schlösser, F. Serrano, Y. Shinano, C. Tawfik, S. Vigerske, F. Wegscheider, D. Weninger, and J. Witzig (2020, March). The SCIP Optimization Suite 7.0. Technical report, Optimization Online.
Hudgens, M. G. and M. E. Halloran (2008). Toward causal inference with interference. Journal of the American Statistical Association 103(482), 832-842.
Kasy, M. (2016). Why experimenters might not always want to randomize, and what they could do instead. Political Analysis 24, 338-338.
Rosenbaum, P. R. (2007). Interference between units in randomized experiments. Journal of the American Statistical Association 102(477), 191-200.
Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66(5), 688.
Rubin, D. B. al. (2008). For objective causal inference, design trumps analysis. Annals of Applied Statistics 2(3), 808-840.
Sobel, M. E. (2006). What do randomized studies of housing mobility demonstrate? causal inference in the face of interference. Journal of the American Statistical Association 10/(47 1398 1407.


Reviews:
[{"review_id": "tvA3iSzPR39", "reviewer": "Reviewer_aWJt", "summary": "The authors propose and analyze a new problem at the intersection of experiment design and synthetic control: in a panel data setting (many units i=1:N observed over time t=1:T), how should treatments be assigned and how should weights be computed at time T such that counterfactuals are imputed well at time T+1?\n\nThe authors interpret the question three ways, providing three mixed integer programming problems. The authors compare the quality of counterfactual implementation of these new experiment designs with difference in means (where neither treatment assignment nor weights are optimized) and synthetic control (where treatment assignment is not optimized).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall assessment\n\nOriginality: The problem studied by the authors appears to be original, and the results are quite cleanly derived. Posing and analyzing a new, interesting problem is the hallmark of a good paper. Can we confirm that this problem has not been previously studied?\n\nQuality: I would like to see the permutation-based inference. Otherwise the paper is interesting but light on theory. It is also light on references to an exponentially growing SC literature.\n\nClarity: The paper is well written. Clarifying earlier on and more formally the difference between standard experiment design, SC, and this problem would be an improvement. How do the objectives compare to regularized SC objectives? Can the authors thereby draw on such results, or contribute to those existing literatures?\n\nSignificance: The paper is significant for proposing a new problem to study. Its significance within causal inference and policy evaluation would dramatically increase if it included the permutation inference result.\n\nSuggestions to improve the paper\n\n105: It would help to begin using the language of ATET earlier, since that is the causal estimand\n\n108: This is an MSE conditional on treatment assignment and weights. Please make the conditioning explicit. Please also derive this result with a proof in the appendix (it is correct, but not easy on the reader)\n\n113: empirical analogue in the pre-period on average\n\n114: walk the reader through the logic. In the pre-period, Y_{it}=Y_{it}(0)=mu_{it}+e_{it}. Alternative, present the proof in the appendix\n\n114: Strictly speaking, the latter term should be \\sum_j (1-D_j) (w_j^i)^2, but the optimal w_j^i=0 if D_j=1 interpreting the former term as fit and the latter as a ridge penalty\n\n114: Please compare this objective to the literature on regularized SC, e.g. as reviewed in Abadie and L’Hour (2018)\n\n123: for the second term, index the sum by j\n\n125: see note above about conditional MSE\n\n127: Nice\n\n141: See comment above. Strictly speaking, the sum in the second term would be over D_i=0\n\n151: Please summarize these constraints in the main text\n\n164: How does this relaxation affect the geometry? C.f. Delauney tessellation\n\n165: Please maintain the parallelism of the main text: per unit, then two-way, then one-way\n\n180: Nice\n\n210: Nice. It would help to mention this comparision of difference in means and SC to the problem of interest earlier on, ideally as different optimization problems in Section 2\n\n287: This is the obvious way to improve this paper. It seems the authors even know how to do it. Including this result would make this paper excellent", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "What assumptions could be violated, and how would that impact conclusions and policy recommendations?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Yq7Cr0neuk8", "reviewer": "Reviewer_Cxg6", "summary": "This work studies an approach for designing experiments in panel data settings. Based on data from T time periods, the proposed approach selects weights and treatment assignments for a subsequent time period to minimize an empirical estimate of the mean-squared error of an individual-level or weighted average treatment effect. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: As far as I'm aware, the proposed approach is novel.\n\n\nQuality:\n\n1. The objective functions that you focus on are inspired by the mean-squared errors (MSEs) of the form $\\mathbb{E}[(\\hat{\\tau}_i-\\tau_i)^2]$ or $\\mathbb{E}[(\\hat{\\tau}-\\tau)^2]$. However, unless I'm missing it, no theoretical guarantees are provided for your proposed objective function. Can you show that the solution to your empirical objective function is (nearly) as good as the solution to the true MSE $\\mathbb{E}[(\\hat{\\tau}_i-\\tau_i)^2]$ or $\\mathbb{E}[(\\hat{\\tau}-\\tau)^2]$? Presumably you end up with guarantees as $T\\rightarrow\\infty$? What about if $T$ does not grow but $N$ does? What happens if neither of these quantities grows? Such a guarantee would go a long way in supporting the use of your objective functions.\n\n2. As you've pointed out, the weighted average treatment effect on the treated (wATET) can be a weird estimand to look at when you're deciding the weights based on some MSE-type criterion. The oddity arises because, when the wATET is of interest, the weights that minimize this criterion decide the estimand, and so even if the chosen weights result in an easier estimation problem (in terms of MSE), it's possible that this estimation problem may not actually be of interest. You also pointed out that this problem goes away when treatment effects are homogeneous and, when they're not, you suggested instead solving a different, one-way global optimization problem. Though this problem seems to resolve the problem of the estimand being decided by the choice of weights, it seems to me that the (unweighted) estimand pursued by this problem is still influenced by the chosen solution of the problem. In particular, the solution to the two-way optimization problem selects individuals to treat, and the average treatment effect on the treated (ATET) estimand is clearly influenced by who receives treatment. Therefore, unless I'm misunderstanding something, it seems like the so-called optimal design still leads to a challenging interpretation wherein the estimand is chosen based on a criterion that has nothing to do with subject-matter expertise, but instead based on making the resulting estimation problem as easy as problem. If this is correct, then, at a minimum, some discussion of the consequences of doing this would be warranted.\n\n3. Related to the above, the experiment section says that:\n\n> We either assign each treated unit the additive treatment effect of 0:05 (homogeneous treatment case) or assume that the first—as determined by the row number—selected treatment unit receives the additive treatment effect of 0, the last unit receives the additive treatment effect of 0.1, and the remaining treatment units receive the treatment effects spread linearly from 0 to 0.1 according to the order determined by their row numbers (heterogeneous treatment case). The average treatment effect on the treated remains 0.05 in both cases.\n\nThis text appears to suggest that the choice of estimand does not, in fact, depend on the treatment allocation. However, I fail to see how the above actually fits with the sampling scheme described in Section 2. It seems that the treatment effects should be decided before individuals are treated, making the above-described mechanism impossible --- but perhaps there's something that I'm not understanding.\n\nE.g., we could have a population of 6 people with potential outcomes on control always equal to zero and potential outcomes on treatment (and, also, individual-level treatment effects) equal to 0, 0, 0.05, 0.05, 0.1, 0.1. If we treat 3 people, we could choose individuals with effects (0,0,0.05), leading to an average treatment effect among the treated (ATET) of $0.05/3\\approx 0.017$, or individuals with effects (0,0.05,0.1) leading to an ATET of $0.05$, or we could choose individuals with effects (0.05,0.1,0.1), leading to an ATET of $0.067$.\nIn contrast, the way you've described your data-generating mechanism here, it seems like the three treated individuals will always have treatment effects (0,0.05,0.1), and therefore always having an ATET of 0.05. \n\n\nClarity: \n\n1. At the beginning of Section 2, you should immediately make clear whether $Y$ will be treated as random. From what's written at the top of page 3, it looks like the answer is \"yes\", and furthermore that the noise is homoscedastic. I think it's important to mention this upfront, since otherwise it's unclear what, exactly, $Y$ is.\n\n2. It seems that most of the paper focuses on the case where $S=T+1$. This makes me wonder whether it was worth introducing notation for $S$ at all, or whether, to reduce notation, you could have just focused on this special case from the start.\n\n3. In the top paragraph of page 3, the text first says that $i=N$ and then later in that paragraph says that $\\hat{Y}\\_{N,T+1}=\\sum_{i=1}^N w_i Y_{i,T+1}$, leading to an overload on $i$. The latter $i$'s should be replaced by $j$, which would avoid the overload and also be more consistent with the notation used in the following paragraph.\n\n4. Another subsection should be added to Section 2, or the \\subsection tag should be removed above Section 2.1.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Xp0sND331VD", "reviewer": "Reviewer_Qhdf", "summary": "The paper considers the problem of optimally partitioning a panel into treatment and control groups, leveraging past observations to ensure a more balanced allocation between treatment & control.\n\nGenerally randomized-controlled trials are the standard in this instance, because particularly with large enough populations and samples the results will be robust to any issues in picking control groups. This paper strives to provide an approach when the sample size is not large enough to provide a suitably power RCT. The alternative approach is inspired by work in synthetic controls, where ex-post one may determine a set of weighted controls to use for a single potentially-unplanned treatment. \n\nThe setting in this problem lies somewhere between the two - applicable for a small handful where perhaps tuning treatment/control can give additional experimental power, at some robustness cost. \n\nThe approach shown draws heavily from the approaches in the synthetic control literature, formulating three approaches: 1) per-unit, wherein effectively synthetic controls are formulated for each treatment - or at least weightings are done accordingly, 2) two-way global, wherein weightings are only for each variable rather than the pairing - so one weighted treatment set, one weighted control set, and then 3) one-way global, with unweighed treatment and a weighted control. Across all, the experimenter is minimizing RMSE.\n\nThe paper formulates each as a mixed-integer optimization problem and does a simulated experiment on unemployment statistics across the 50 states (in 40 time periods), with either a fixed or linear additive treatment effect.\n\nThe paper does not include justifications for expected conditions on the randomness that would lead to this approach behaving better than an RCT (though it is cited as a future concern). \n", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles a seemingly interesting middle ground between synthetic control and RCTs. The approach put forward derives its approach from a generalization of the synthetic control literature. I am not familiar enough with the applicable literature and area to characterize novelty among the most related papers.\n\nGenerally the paper is well written, clearly communicating the approach. Implementing the approach would seem not hard to follow. Optimally solving the IP is shown to be NP-hard. Since the approach seems like it is generally of use when the \n\nThe justification for actually using the approach however is a little lacking. Implicit in the behavior is a reliance on the pre-treatment outcome data as a reliable predictor of correlations in treatment effect. Further justification of when this method should be chosen above RCTs would be helpful, either with a theoretical discussion of conditions on the relationships between the randomness. It would seem that the approach would be at its best when there is additional structure that is uncovered and leveraged. The experiment performed has some of that implicitly since there must be that uncovered structure in employment data, but it would be good to see further discussion. Additionally, implicit in some of the discussion is the reasoning that you would want to use this approach when panel size is small and thus an RCT may not have enough power. This tradeoff is not explored, and it would be great to see estimates of where that tradeoff is or a comparison. Since RCTs are chosen for intrinsic robustness, this may be challenging to quantify, but showing something about the sweetspot for this approach as the data size increases would be helpful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Using this approach rather than RCTs does potentially expose experiments to a loss of robustness. I am sure that the authors would not advocate a large scale shift to these, but a little more discussion of what to be concerned about if interpreting the results from an experiment that was generated using this approach would be helpful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The expert reviewers for the most part appreciated the paper and were guardedly positive. The paper is commended for posing an interesting new question. At the same time there were concerns about the relevance of the estimand targeted as well as formal guarantees. The authors suggested possible ways to address this that should be incorporated into the paper. Crucially, the permutation inference result would add an important aspect to the paper that would merit its acceptance. Moreover, the authors should give a detailed discussion about their ATET estimand and explain carefully its limitations in the absence of homogeneity, which is the practically common setting, and how possibly slight violations might affect the interpretation of the results.
Decision: Accept (Poster)