Synthetic Design: An Optimization Approach to Experimental Design with Synthetic Controls

1 Introduction . Randomized experiments have long been a staple of applied causal inference . In his seminal paper , Rubin ( 1974 ) suggests that “ given a choice between the data from a randomized experiment and an equivalent nonrandomized study , one should choose the data from the experiment , especially in the social sciences where much of the variability is often unassigned to particular causes. ” Using the language of Rubin ’ s potential-outcomes framework , randomization guarantees that the treatment status is independent of the potential outcomes and that a simple and intuitive estimator that compares the average outcomes of the treatment and control units is an unbiased estimator of the average treatment effect ( ATE ) . If both the treatment and control samples are sufficiently large , the hope is that this difference-in-means estimate is close to the population mean of the treatment effect . ∗Equal contributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Another crucial property of randomized experimental designs is their robustness to alternative assumptions about the data generating process—a completely randomized experiment does not take into account any features of the observed data . Perhaps not surprisingly , when the researchers are willing to incorporate additional probabilistic assumptions in their design decisions , they can improve on the statistical properties of the average treatment effect estimators ( see , for example , Kasy , 2016 ) . These improvements , however , do not come for free and the performance of the estimators may suffer if the incorporated assumptions are violated ( Banerjee et al. , 2020 ) . For these reasons randomized experiments are widely used in academic and clinical settings , as well as industrial applications . However , not every practically important question can be easily answered using an experiment on a large sample of experimental units . For instance , the evaluation of major policies targeting large geographic areas has long been of interest in social and political sciences . One of the traditional approaches is to compare the affected unit—such as a metropolitan area or a state—to the average across a sample of carefully picked control units which are deemed to be suitable comparisons ( Card , 1990 ) . A more recent alternative , called synthetic control , first introduced by Abadie and Gardeazabal ( 2003 ) and later developed in Abadie et al . ( 2010 ) and Abadie et al . ( 2015 ) , compares the unit of interest to a weighted average of the units unaffected by the treatment , where the weights are selected in a way that achieves a good fit on pre-treatment outcome variables as well as potentially other observed covariates . While originally developed by academics for evaluating the effects of policies , approaches similar to the synthetic-control methodology have recently gained popularity in industry as well in cases when applied researchers decide to run experiments targeting larger units often representing geographic areas . This decision may be justified when more granular experiments are either unavailable ( for example , television advertising can only be targeted at the media-market level ) or are unlikely to capture the relevant effects due to interference or equilibrium concerns ( Sobel , 2006 ; Rosenbaum , 2007 ; Hudgens and Halloran , 2008 ) . For instance , a company like Uber may want to evaluate some of the possible treatments at the market level rather than at the driver level if the treatment in question is likely to affect the driver supply . Moreover , launching an experiment targeting even a single unit may be so expensive that the researchers try to minimize the required number of treated units . Privacy and fairness concerns may also make treatment assignment at a more granular level problematic . Synthetic control and similar approaches may be attractive as estimation procedures in those cases , but they fail to address the equally if not more important aspect of the optimal choice of experimental units ( see , for example , Rubin et al. , 2008 ) . We attempt to narrow this gap in the current paper . We consider a panel-data setting in which the researcher observes the outcome metric of interest for a number of units in a number of time periods and has to decide : ( i ) which units to experiment on and ( ii ) how to estimate the treatment effects after collecting the outcome data in the experimental time periods . The main difference between this setting compared to a typical synthetic-control study is that the treated units are not fixed , but rather chosen by the researcher . Proving that the underlying optimization problem is NP-hard , we rule out the possibility of designing polynomial-time algorithms for the problem under P 6=NP . Therefore , we formulate this combined design-and-analysis problem as a mixed-integer program ( MIP ) . Depending on the particular estimands of interest , we propose one of the several formulations and discuss their advantages and drawbacks . We motivate the choice of the optimization objectives and discuss the selection of experimental units each of the objectives leads to . The MIP formulation allows for an easy inclusion of additional constraints as long as those are linear . For instance , it is easy to restrict the overall number of treated units , exclude specific units from treatment , or enforce a budget constraint if there is a varying cost to treat different units . Using publicly available state-level unemployment data from the US Bureau of Labor Statistics , we compare the proposed methodology to a randomized design that utilizes either the conventional difference-in-means estimator or the synthetic-control approach . We estimate the average as well as individual state-level treatment effects in a simulated experiment and find that our approach substantially reduces the root mean squared error ( RMSE ) of the estimates . We show that our MIP-based design-and-analysis procedures consistently outperform the more traditional baselines regardless of whether the treatment effects are homogeneous or heterogeneous and whether the number of units selected for treatment is small or large relative to the total number of units in the sample . We also suggest a permutation-based inference procedure that follows Chernozhukov et al . ( 2021 ) . We verify in our simulations that this procedure leads to correct test sizes and improved statistical power for testing the sharp null hypothesis of zero treatment affects across all treated units , when used in conjunction with the proposed estimators . We provide theoretical guarantees—albeit under rather strong assumptions—which ensure that the proposed tests have proper sizes . To our knowledge , this is one of the first papers to study experimental design in the context of synthetic control and adjacent estimation techniques . Doudchenko et al . ( 2019 ) consider the case when the underlying effects are homogeneous and only a single unit can be treated ; they suggest searching for the unit that delivers the highest statistical power when testing the hypothesis of zero treatment effect with an artificially applied treatment . In an independent work , Abadie and Zhao ( 2021 ) also consider optimal design in settings where synthetic-control estimators are used for estimation . However , there are a few important aspects that differentiate the current paper from theirs . First , Abadie and Zhao ( 2021 ) use both pre-treatment unit-level covariates as well as the outcomes from some—but not necessarily all—of the pre-treatment periods whereas we focus exclusively on the outcome variable and utilize all of the pre-treatment outcome data . As a result , the inference procedure proposed by Abadie and Zhao ( 2021 ) does not apply to our approach . Second , we discuss formal hardness results . Finally—and , perhaps , most importantly—Abadie and Zhao ( 2021 ) choose an objective function that fixes a priori the target average treatment effect , as either the population average treatment effect or a weighted version thereof . In contrast , the objective function we use focuses on the average treatment effect only for the units we choose for the treatment , making the target estimand stochastic , but potentially easier to estimate and requiring different assumptions . The rest of the paper is organized as follows . Section 2 introduces the setting and the proposed estimation approaches . Section 3 introduces the mixed-integer formulation of the suggested estimators . Section 4 presents some of the theoretical unit-selection results and the intuition behind them . Section 5 reports the empirical results obtained through simulations . Section 6 discusses some of the practical consideration that should accompany applied work that uses the proposed methodology . Section 7 concludes and outlines the directions of future research . 2 Setting . Let the researcher observe the outcome metric of interest , Y , for N units during T time periods , such that the observed data can be represented as an N × T matrix of values Yit . At time t = T the researcher decides—based on the data observed up until that point—which units should be treated and assigns a binary treatment described by variables Di ∈ { 0 , 1 } , i = 1 , . . . , N . The outcomes are then observed for an additional S − T time periods t = T + 1 , . . . , S and the treatment-effect estimates are constructed . Each unit i = 1 , . . . , N in each time period t = T + 1 , . . . , S is associated with two potential outcomes ( Yit ( 0 ) , Yit ( 1 ) ) which are considered random . The potential outcome Yit ( 0 ) is realized if Di = 0 and Yit ( 1 ) is realized if Di = 1 so that the observed outcome is Yit = Yit ( Di ) = Yit ( 0 ) ( 1−Di ) + Yit ( 1 ) Di . Recall that , given a setting where a single unit i = N has received the treatment in a single time period t = T + 1 = S , the synthetic-control literature ( Abadie et al. , 2010 , among others ) suggests constructing a counterfactual estimate for unit N as a weighted average of the other units ’ observed outcomes : ŶN , T+1 ( 0 ) = ∑N−1 i=1 wiYi , T+1 . In the previous equation , the wi ’ s are weights learned from the data observed during the pre-treatment periods t = 1 , 2 , . . . , T , often by minimizing∑T t=1 ( YNt − ∑N−1 i=1 wiYit ) 2 under some constraints on the weights . Assuming that the treatment effect , τN , for this unit is additive , it can then be estimated as τ̂N = YN , T+1 − ŶN , T+1 ( 0 ) . We now consider a more general setting where K units have received the treatment , with outcomes given by Yit ( 0 ) = µit + εit and Yit = Yit ( 0 ) +Diτi with homoscedastic noise εit that has mean zero and variance σ2 and additive treatment effects τi . In order to estimate the treatment effect in this more general setting , we can apply a separate synthetic control method to each treated unit i , learning an appropriate set of weights { wij } for each treated unit i individually : Ŷi , T+1 ( 0 ) = ∑ j : Dj=0 wijYj , T+1 . The treatment effect for unit i is then estimated as τ̂i = Yi , T+1 − Ŷi , T+1 ( 0 ) . Rather than considering each weight-fitting optimization problem separately , we can express the ( conditional ) mean squared error ( MSE ) of the resulting estimator of the treatment effect as : E [ ( τ̂i − τi ) 2 ∣∣ { Dj , wij } Nj=1 ] = µi , T+1 − ∑ j : Dj=0 wijµj , T+1 2 + σ2 1 + ∑ j : Dj=0 ( wij ) 2 . A proof is included in the supplementary materials . The synthetic-control literature often operates in settings where the treated units are given . Here , we allow the experimenter to select which units should receive the treatment . The mean-squared-error formula above leads us to consider the following optimization problem over the weights { wij } Ni , j=1 and the treatment variables { Di } Ni=1 under some appropriate constraints on the weights { wij } Ni , j=1 . The objective below can be seen as the empirical analog of the right-hand side of the population equation above in the pre-treatment period averaged across time and across the treated units . min { Di , { wij } Nj=1 } Ni=1 1 KT N∑ i=1 T∑ t=1 Di Yit − N∑ j=1 wij ( 1−Dj ) Yjt 2 + σ2 K N∑ i=1 N∑ j=1 Di ( wij ) 2 ( per-unit ) Note that we can safely sum across all units , not just the control ones , in the second term since the optimal values wij for units j with Dj = 1 will be equal to zero in an optimal solution . In essence , this optimization problem attempts to minimize the discrepancy between the pre-treatment outcomes of the units chosen for treatment and the weighted averages of the outcomes of the units left as controls . At the same time , due to the second term , the objective attempts to balance the unit weights themselves . The term σ2 is unlikely to be known to the researcher and should be chosen based on the observed data . One possible way is to set it equal to the sample variance of the observed outcomes . We further discuss the selection of the penalty parameter in Section 6 . Penalizing the weights is not uncommon in the synthetic-control literature . For example , Doudchenko and Imbens ( 2016 ) use the elastic-net penalty on the weights and Abadie and L ’ Hour ( 2021 ) introduce a lasso-style penalty in which the nonnegative weight , wij , is multiplied by the squared distance between the vectors of the covariates used for matching the units . This way , depending on the magnitude of the penalty hyperparameter , they can balance between the synthetic-control fit and the nearest-neighbor fit that puts all the weight on unit j closest to i in terms of the observed covariates . In many applications , the researcher may be interested in estimating some weighted average of the unit-level treatment effects on the treated units . Rather than considering each treated unit separately and then computing the weighted average of the estimated individual treatment effects , practitioners may wish to construct a synthetic-control-type estimate for the weighed average of the treated units directly . Consider a set of treatment assignments { Di } Ni=1 and weights { wi } Ni=1 on outcomes at time T + 1 , and assume that we wish to estimate the weighted average treatment effect on the treated ( wATET ) , τ = ∑ i : Di=1 wiτi = ∑N i=1Diwiτi , as a difference in weighted means : τ̂ = ∑ i : Di=1 wiYi , T+1−∑ i : Di=0 wiYi , T+1 . Then , under the same outcomes model as presented above , the ( conditional ) mean squared error of the difference-in- ( weighted ) -means estimator is E [ ( τ̂ − τ ) 2 ∣∣ { Di , wi } Ni=1 ] = ( ∑ i : Di=1 wiµi , T+1 − ∑ i : Di=0 wiµi , T+1 ) 2 + σ2 N∑ i=1 w2i . As before , our setting allows the experimenter to optimally select which units should receive the treatment as well as which particular weighting scheme should be used . This is especially appropriate when the treatment effects are homogeneous and τi = τ for all i = 1 , . . . , N . In that case , any weighted average of the unit-level treatment effects is equal to τ and the weights can be chosen in a way that minimizes the mean squared error . The population equation above suggests solving the following optimization problem on the weights { wi } Ni=1 and the treatment variables { Di } Ni=1 based on the data observed in periods t = 1 , . . . , T : min { Di , wi } Ni=1 1 T T∑ t=1 ( ∑ i : Di=1 wiYit − ∑ i : Di=0 wiYit ) 2 + σ2 N∑ i=1 w2i . ( two-way global ) So far , we have not considered specific constraints on the weights , { wi } Ni=1 . If σ2 > 0 , wi = 0 for all i = 1 , . . . , N is the unique optimal solution to the two-way global problem above if the weights are not constrained in any way . In order to avoid this clearly undesirable solution , we assume that the weights { wi } Ni=1 are normalized : ∑ i : Di=1 wi = ∑ i : Di=0 wi = 1 . While not strictly required , another reasonable set of constraints motivated by estimating a proper weighted average is requiring all weights to be nonnegative , wi ≥ 0 for all i = 1 , . . . , N . Similar constraints can be imposed in the context of the per-unit problem : wij ≥ 0 for all i , j = 1 , . . . , N and ∑ j : Dj=0 wij = 1 for all i = 1 , . . . , N such that Di = 1 . Finally , the weights on the treated units may be fixed if a specific weighted average—for example , a simple average with equal weighting—needs to be estimated . This constraint is particularly justified when the treatment effects are heterogeneous and different weighting schemes lead to different estimands . For that reason , we formulate another variation of the global problem : min { Di , wi } Ni=1 1 T T∑ t=1 ( ∑ i : Di=1 wiYit − ∑ i : Di=0 wiYit ) 2 + σ2 N∑ i=1 w2i ( one-way global ) subject to an additional set of constraint that require wi = wj for all i , j = 1 , . . . , N such that Di = Dj = 1 . It is possible to obtain other design-and-estimation approaches as special cases of these problems— something that we utilize later in the paper . For instance , the standard difference-in-means approach under randomized design can be viewed as the special case of the one-way global problem with the treatment indicators , Di , set randomly and the weights on the control units restricted to be equal , wi = wj for all i , j = 1 , . . . , N such that Di = Dj = 0 . Likewise , the synthetic-control estimator can be viewed as the per-unit problem with the treatment indicators set according to the experimental design of choice—for example , randomly .

Summary: The paper considers the problem of optimally partitioning a panel into treatment and control groups, leveraging past observations to ensure a more balanced allocation between treatment & control.  Generally randomized-controlled trials are the standard in this instance, because particularly with large enough populations and samples the results will be robust to any issues in picking control groups. This paper strives to provide an approach when the sample size is not large enough to provide a suitably power RCT. The alternative approach is inspired by work in synthetic controls, where ex-post one may determine a set of weighted controls to use for a single potentially-unplanned treatment.   The setting in this problem lies somewhere between the two - applicable for a small handful where perhaps tuning treatment/control can give additional experimental power, at some robustness cost.   The approach shown draws heavily from the approaches in the synthetic control literature, formulating three approaches: 1) per-unit, wherein effectively synthetic controls are formulated for each treatment - or at least weightings are done accordingly, 2) two-way global, wherein weightings are only for each variable rather than the pairing - so one weighted treatment set, one weighted control set, and then 3) one-way global, with unweighed treatment and a weighted control. Across all, the experimenter is minimizing RMSE.  The paper formulates each as a mixed-integer optimization problem and does a simulated experiment on unemployment statistics across the 50 states (in 40 time periods), with either a fixed or linear additive treatment effect.  The paper does not include justifications for expected conditions on the randomness that would lead to this approach behaving better than an RCT (though it is cited as a future concern).