Title: Double and Single Descent in Causal Inference with an Application to High-Dimensional Synthetic Control
Venue: NeurIPS 2023
Accepted: yes
Keywords: Double descent

interpolating regression

synthetic control

causal inference
Abstract: Motivated by a recent literature on the double-descent phenomenon in machine learning, we consider highly over-parameterized models in causal inference, including synthetic control with many control units. In such models, there may be so many free parameters that the model fits the training data perfectly. We first investigate high-dimensional linear regression for imputing wage data and estimating average treatment effects, where we find that models with many more covariates than sample size can outperform simple ones. We then document the performance of high-dimensional synthetic control estimators with many control units. We find that adding control units can help improve imputation performance even beyond the point where the pre-treatment fit is perfect. We provide a unified theoretical perspective on the performance of these high-dimensional models. Specifically, we show that more complex models can be interpreted as model-averaging estimators over simpler ones, which we link to an improvement in average performance. This perspective yields concrete insights into the use of synthetic control when control units are many relative to the number of pre-treatment periods.

Double and Single Descent in Causal Inference with an Application to High-Dimensional Synthetic Control

Abstract

Motivated by a recent literature on the double-descent phenomenon in machine learning, we consider highly over-parameterized models in causal inference, including synthetic control with many control units. In such models, there may be so many free parameters that the model fits the training data perfectly. We first investigate high-dimensional linear regression for imputing wage data and estimating average treatment effects, where we find that models with many more covariates than sample size can outperform simple ones. We then document the performance of high-dimensional synthetic control estimators with many control units. We find that adding control units can help improve imputation performance even beyond the point where the pre-treatment fit is perfect. We provide a unified theoretical perspective on the performance of these high-dimensional models. Specifically, we show that more complex models can be interpreted as model-averaging estimators over simpler ones, which we link to an improvement in average performance. This perspective yields concrete insights into the use of synthetic control when control units are many relative to the number of pre-treatment periods.

1 Introduction

Motivated by a recent literature on the double-decent phenomenon in machine learning, we investigate the properties of common econometric estimators when we increase their complexity. For highdimensional linear regression and for synthetic control with many control units, we document empirical applications where extremely over-parameterized models impute missing out-of-sample and out-of-time outcomes well. We then provide a common explanation for the returns to complexity in high-dimensional linear regression and synthetic control in terms of a simple model-averaging property, which we link to an improvement in imputation performance.

We often conceptualize the effect of complexity on econometric models in terms of a bias-variance trade-off: Adding complexity makes models more expressive and reduces bias, while increased overfitting leads to additional variance. In this view, an overly complex model fits overly well in the training sample, but fails to recover true parameters or generalize to new data. For example, linear regression with too many variables or synthetic control with many control units may perform poorly because of overfitting and excess variance. Consistency results for high-dimensional econometric models therefore usually assume that the expressiveness of models is limited relative to the available sample size, and practical advice often highlights choosing simple models or regularizing complex ones, in a way that balances bias and variance optimally to achieve good estimation and prediction performance.

A recent literature in statistics and machine learning has highlighted the surprisingly good prediction performance of extremely high-dimensional models that fit the data perfectly. That literature has documented a so-called double-descent phenomenon for deep neural networks and other highdimensional regression models: increasing complexity beyond the interpolation threshold at which the training sample error is zero can lead to a gradual reduction in variance and improvement in out-of-sample performance. In such cases, there are often two complexity regimes: below the interpolation threshold, the usual bias-variance trade-off leads to a decrease (first descent) and then an increase in out-of-sample loss, while beyond the interpolation threshold, out-of-sample loss decreases again (second descent).

In order to investigate the properties of highly over-parameterized models in causal inference, we first demonstrate a double-descent curve for linear regression in imputing wage outcomes and estimating average-treatment effects. In the LaLonde (1986) sample of the Current Population Survey (CPS) drawn from Dehejia and Wahba (1999, 2002), we generate over 8,000 variables from binning and interacting the original eight demographic and employment-related variables. We then fit a linear regression model on 3,000 training units and an increasing, randomly chosen subset of these variables, choosing the norm-minimizing solution once the model fits perfectly. Evaluated on the LaLonde (1986) control sample from the National Supported Work Demonstration (NSW) experiment, we observe the usual bias-variance trade-off for a low and moderate number of included covariates, where performance first increases slightly, before deteriorating substantially when approaching the interpolation threshold. Beyond the interpolation threshold, however, out-ofsample performance increases again. Ultimately, an extremely over-parameterized model on over 8,000 variables even outperforms less complex regressions with a randomly chosen set of covariates and achieves performance comparable to a linear regression on the original, unmanipulated set of covariates.

Having demonstrated the returns to complexity in high-dimensional linear regression, we document the performance of synthetic control estimators with many control units. In the California smoking data (Abadie et al., 2010), we impute missing smoking rates based on a small number of pre-treatment periods and an increasingly large number of control states. As in the case of linear regression, we see returns to increasing model complexity, even beyond the point where some of the synthetic-control models fit the training data perfectly. However, for synthetic control we do not observe an initial trade-off between bias and variance: in our empirical example, the performance on future periods is always better for the more complex models, with no intermittent increase in errors. Unlike the doubledescent relationship for linear regression, the performance of synthetic control in our application yields a single-descent curve that only improves with complexity, no matter whether there are a few or many control states.

We then connect the returns to complexity in high-dimensional linear regression and in synthetic control in terms of a simple model-averaging property. While both estimators are constructed differently and represent different regressions, they both share a common feature: More complex models can be represented as convex averages over simpler models. We show that this modelaveraging property applies to linear regression in the interpolation regime, as well as to synthetic control in general. The property holds purely mechanically and does not depend on the training or target distributions. It relates to other model-agnostic properties of linear regression, for which we also show a reduction in (conditional) variance for minimal-norm least-squares estimators beyond the interpolation threshold.

Having established a model-averaging property for interpolation linear regression and for synthetic control, we provide high-level assumptions under which this property translates to better imputation performance. A direct consequence of model averaging is that the (convex) prediction loss of a more complex model cannot be worse than the corresponding average prediction loss of simpler models, when the same weights are used to average. When the complex model on average also outperforms comparable models of the same complexity, then we show that the model-averaging property translates into a reduction in average out-of-sample error relative to a randomly selected simpler model. These results only put high-level, largely model-agnostic assumptions on the data-generating process, and are driven by mechanical properties of the underlying estimators.

Our results have practical implications for the use of synthetic control with many control units. Conventional wisdom may indicate that synthetic control with a very large number of donor units relative to the number of pre-treatment periods is problematic, and that selecting among many, ex-ante

exchangeable units in this case may represent a challenge. However, our results imply that this is not the case: as long as ties are broken by a suitable regularization procedure (such as the minimum-norm solution in our case), making an ex-ante choice among many control units is not necessary. That said, if ex-ante information is available about which units are particularly suitable as controls, then using this information can still be helpful and improve imputation.

We build upon results on over-parameterized regression in machine learning and statistics, where double-descent curves for kernel and linear regression and the performance of norm-minimizing interpolating solutions have been studied extensively (including Liang and Rakhlin, 2020; Liang et al., 2020; Liang and Recht, 2023; Bartlett et al., 2020; Hastie et al., 2022) as part of a broader literature on double descent and interpolation in deep learning (Zhang et al., 2016; Belkin et al., 2018; Belkin, 2021; Mei and Montanari, 2022). Kelly et al. (2022) documents the benefits of overparameterized models in asset return prediction both in theory and empirically. Kato and Imaizumi (2022) provides results on the estimation of conditional average causal effects using over-parametrized linear regression. Relative to this work on interpolating regression, we show connections to synthetic control based on simple mechanical properties that are largely agnostic about the true data-generating process. Thereby, we also relate to work on high-dimensional and regularized synthetic control (Doudchenko and Imbens, 2016; Abadie and L'Hour, 2021; Ben-Michael et al., 2021) and the connections between synthetic control and linear regression (Athey et al., 2021; Agarwal et al., 2021; Shen et al., 2022; Bruns-Smith et al., 2023). 1 Finally, we connect to a literature on model averaging in econometrics and statistics (e.g. Hansen, 2007; Claeskens and Hjort, 2008). More specifically, Wilson and Izmailov (2020) consider Bayesian model averaging in deep learning, and discuss relationships to double descent. While our linear-regression solutions are closely related to available results on norm-minimizing regression, we are not aware that the results on synthetic control were noted previously.

The remaining note is structured as follows. Section 2 provides an empirical example of double descent for linear regression and discusses some properties of the norm-minimizing linear-regression estimator in the interpolating case. Section 3 discusses the relationship of complexity and imputation performance of synthetic control in an empirical example, and makes connections to the properties of interpolating linear regression. Section 4 discusses high-level consequences of the model-averaging property of interpolating linear regression and synthetic control. Section 5 concludes by discussing some limitations and open questions.

2 Double Descent for Linear Regression

In this section, we consider imputation by high-dimensional linear regression as an illustration of the performance of highly over-parameterized models. We start with an empirical illustration of wage imputation in CPS data with a varying number of randomly ordered covariates, which we evaluate in terms of its ability to estimate an average treatment effect. We then discuss theoretical properties of the interpolating linear-regression estimator, followed by a graphical illustration. These results are closely related to prior work on interpolating linear and kernel regression (including Bartlett et al., 2020; Liang et al., 2020; Hastie et al., 2022; Kato and Imaizumi, 2022).

2.1 Setup and Estimator

We consider a linear-regression estimator in data ( y i , x i ) n i =1 ∈ R × R k from n observations of a scalar outcome and k scalar covariates. We write X = ( x ′ i ) n i =1 ∈ R n × k and Y = ( y i ) n i =1 ∈ R n . For a subset J ⊆ { 1 , . . . , k } of the covariates, we denote by B J = arg min β ∈ R k ; β j =0 ∀ j / ∈ J ∑ n i =1 ( y i -x ′ i β ) 2 the set of all least-squares linear-regression estimates on J . Among these, we choose the norm-minimizing estimate ˆ β J = argmin β ∈B J ∥ β ∥ . Throughout, we assume that X J = ( x ij ) i ∈{ 1 ,...,n } ,j ∈ J ∈ R n ×| J | is of full row rank. Hence, there are multiple least-squares solutions, |B J | > 1 , if and only if | J | > n . In that case, Y = X ˆ β J and ˆ β J is the minimal-norm interpolating solution (cf. Liang et al., 2020).

2.2 Empirical Motivation

We impute wages in the non-experimental LaLonde (1986) sample of the Current Population Survey (CPS) using linear regression on a large number of covariates, based on data drawn from Dehejia and Wahba (1999, 2002). From the original eight covariates, we obtain k = 8408 explanatory variables by binning and interacting the available features. We train minimal-norm least-squares regression models on a training sample of n = 3000 randomly chosen observations, and evaluate their mean-squared error in imputing average wages for subsets (of various sizes) of the 260 LaLonde (1986) National Supported Work Demonstration (NSW) experimental controls as provided through Dehejia and Wahba (1999, 2002). To these covariates we add a small amount of iid Gaussian noise to avoid rank deficiency when estimating linear regression. When fitting wage imputation models, we vary the set of covariates included in the regression. Specifically, we order all features randomly. For a complexity ℓ ∈ { 1 , . . . , k } , we then choose the first ℓ covariates, and obtain the estimate ˆ β J for J = { 1 , . . . , ℓ } . Our presented results reflect an average over five such random orderings of covariates.

In order to assess the viability of this approach for applications to average treatment effect (ATE) estimation, we assess each model by its ability to accurately predict the average outcome on a new dataset. In particular, we consider various subsets of size m of the NSW experimental control set. For each m , we draw 1000 samples of size m without replacement from the NSW experimental control dataset. For each sample, we average the m observation-level outcome predictions and compare the result to the true mean outcome of the given sample. We then take the RMSE across the 1000 draws of size m to obtain our evaluation metric. 2

In Figure 1a, we consider CPS data and plot the average root-mean-squared error (RMSE) for pointwise prediction of the wages, reporting the in-sample performance as well. The vertical dashed line denotes the interpolation threshold where ℓ = n . Figure 1b shows the ATE RMSE metric, once again averaged over five random orderings of the covariates, for various subset sizes. The horizontal dashed lines show the RMSE of a simple linear regression on the original, unmanipulated (low-dimensional) features, colored according to corresponding sample size m . A zoomed-in version of Figure 1b focusing on the highly overparametrized regime can be found in Figure 7.

0

2000

4000

6000

8000

Number of Covariates

0

10

20

30

40

50

RMSE (thousands $)

Out-of-sample

In-sample

(a) Average out-of-sample (blue) and in-sample (orange) pointwise RMSE for CPS data.

0

2000

4000

6000

8000

Number of covariates

0

5

10

15

20

25

30

ATE RMSE (thousands $)

m = 1

m = 5

m = 50

m = 100

(b) Average counterfactual prediction RMSE for varying subset sizes on NSW controls.

The out-of-sample losses in these illustrations show a descent in loss right of the interpolation threshold (denoted by the vertical dashed line), at which point in-sample loss is zero. For NSW experimental controls, out-of-sample error initially decreases (first descent), while for CPS nonexperimental controls it remains initially flat. For both out-of-sample datasets, loss then goes on to increase and peaks sharply at ℓ = n , at which point the linear models start to fit perfectly. As complexity increases further, loss decreases again (second descent). In both cases, loss continues to decrease throughout. For NSW experimental controls, loss ultimately reaches a minimum that is below the lowest error achieved left of the interpolation threshold, while the loss achieved in the CPS case is similar between the right tails and the minimum on the left. Furthermore, as the sample size m increases in Figure 1b, the highly complex interpolating models outperform a simple model based

on the original set of provided features; the m = 50 and m = 100 curves have minima below their corresponding dashed lines.

In this setting, highly over-parameterized linear models constructed via discretizing and interacting available features exhibit performance improvements over a linear model fitted on the original features. This result appears remarkable, since the heavily over-parameterized models do not include the original non-binary covariates, but only indicators obtained from quantile binning (see Section B.1 for details). Further improvement could be achieved by always including the original covariates in the model.

We note that our illustration is extreme in that it artificially creates a large number of low-signal covariates, for which the best model only slightly outperforms a small, hand-curated model based on the original covariates. Nevertheless, this simple empirical exercise demonstrates the nonmonotonicity in the relationship of complexity to variance and loss that has been studied by the literature on interpolating regression and double descent, and extends it to causal target parameters like the average treatment effect.

2.3 Theoretical Properties and Geometric Illustration

Motivated by the empirical illustration, we note some theoretical properties that are direct consequences of the structure of the norm-minimizing linear least-squares estimator, and will later serve as a comparison point for synthetic control. We note that formal results on the bias-variance properties in terms of more primitive properties of the data-generating process are available, including in Bartlett et al. (2020); Liang et al. (2020); Hastie et al. (2022). Here, we focus on illustrating properties that follow mechanically from the construction of the estimator. Our focus is on comparing more complex models, with covariates J , to slightly simpler (more sparse) ones, with covariates J \ { j } , where the j -th covariate is dropped. Throughout, we assume that the covariate matrices are of full row rank for the more and less complex models

Assumption 1 (Full rank) . The covariate matrix X J ∈ R n ×| J | with columns J as well as the covariate matrices X J \{ j } ∈ R n × ( | J |-1) for all j ∈ J are of full row rank.

We first consider the geometry of interpolating solutions, for which we note that more complex model can be expressed by an average over simpler models.

Proposition 1 (Model averaging for interpolating linear least-squares regression) . For every X and J with | J | > n such that Assumption 1 holds there exists

We can choose the weights in Proposition 1 as a function of the covariate matrix X J only. Specifically, weights can be chosen as λ j = 1 -X ′ j ( X J X ′ J ) -1 X j | J |-n , where X j is the j -th column of X and X ′ j ( X J X ′ J ) -1 X j can be seen as the 'leverage' of feature j ∈ J , analogously to the leverage of an observation in the usual (low-dimensional) linear regression case. 3

β

1

β

2

ˆ

β

{

1

,

2

}

ˆ

β

{

2

}

ˆ

β

{

1

}

(a) Non-interpolating case with

n

= 2

.

β

1

β

2

ˆ

β

{

2

}

ˆ

β

{

1

}

ˆ

β

{

1

,

2

}

(b) Interpolating case with

n

= 1

.

The model averaging property of interpolating linear regression is illustrated in the right panel of Figure 2 for the simple case of two covariates ( J = { 1 , 2 } ) and a single data point ( n = 1 ). Here, the models β ∈ R J that for the model perfectly lie on the line through ˆ β { 1 } and ˆ β { 2 } , with the normminimizing solution ˆ β { 1 , 2 } lies between the two. In contrast, if ˆ β { 1 } and ˆ β { 2 } are not interpolating (such as in the case of the left panel of Figure 2, where n = 2 ), then the more complex model ˆ β { 1 , 2 } does not generally lie in the convex hull of the simpler ones.

As an immediate consequence, any interpolating model can in this case be expressed as a convex average of simpler interpolating models, ˆ β J = ∑ L ⊆ J ; | L | = ℓ λ j ˆ β L for all ℓ ∈ { n, . . . , | J |} (provided all X L are of full row rank). A particularly interesting special case is ℓ = n , for which we express ˆ β J as a model average of just-interpolating models ˆ β L with ˆ β L L = X -1 L Y .

In addition to the model-averaging property, the geometry of interpolating solutions also implies that the variance of the norm-minimizing linear least-squares estimator generically decreases in the interpolation regime | J | > n , for which the complex estimator ˆ β J along with the simpler estimator ˆ β J \{ j } both fit the training data perfectly.

Proposition 2 (Variance reduction for linear least-squares regression) . Suppose Assumption 1 and that Y has a second moment. If | J | > n then a.s. tr Var ( ˆ β J | X ) ≤ min j ∈ J tr Var ( ˆ β J \{ j } | X ) .

The reduction in variance is a direct consequence of a more general property of the least-squares solution. Specifically, the next proposition shows that if we redraw new outcome data in the interpolation regime, then the distance between more complex solutions is smaller than the distance between less complex models.

Proposition 3 (Variation hierarchy for linear least-squares regression) . For fixed X and J for which Assumption 1 holds consider two draws Y A and Y B yielding minimal-norm least-squares estimates ˆ β J A , ˆ β J \{ j } A and ˆ β J B , ˆ β J \{ j } B , respectively.

Here, we write ∥ β ∥ M = √ β ′ Mβ for some positive semi-definite symmetric matrix M ∈ R k × k .

In words, the variation of models across draws of the outcome data increases with complexity on the left of the interpolation threshold, while it decreases on the right. Here, the choice of norm is essential for these results to hold uniformly across simpler models. 4

Figure 3 depicts this graphically. In Figure 3a, for the non-interpolating case we observe that the norm of the difference between the model coefficient vectors making use of both covariates ( ˆ β { 1 , 2 } , ˜ β { 1 , 2 } ) is larger than the norm of the differences between the coefficient vectors taking into consideration just a single covariate at a time. In Figure 3b for the interpolating case, we observe that the reverse is true; in this case, the norm of the difference between the complex models is smaller than the norms of the differences between the simpler models.

In practice, we may care about model properties beyond variance, and consider imputation loss beyond the above norms in the parameters. In Section 4, we will leverage the model-averaging properties from Proposition 1 to establish such bounds on more general imputation errors.

We believe that these variance and geometric properties of linear regression are well understood in the literature and likely not new, although we are not aware of an explicit statement of the model-averaging connection between more and less complex interpolating linear-regression models.

3 Single Descent for Synthetic Control

We next consider imputation using synthetic control with many control units. As in the case of linear regression, we start with an empirical illustration. In the Abadie et al. (2010) California smoking

√

β

1

β

2

ˆ

β

{

1

,

2

}

ˆ

β

{

2

}

ˆ

β

{

1

}

˜

β

{

1

,

2

}

˜

β

{

2

}

˜

β

{

1

}

(a) Non-interpolating case with

n

= 2

.

β

1

β

2

ˆ

β

{

1

,

2

}

ˆ

β

{

2

}

ˆ

β

{

1

}

˜

β

{

2

}

˜

β

{

1

}

˜

β

{

1

,

2

}

(b) Interpolating case with

n

= 1

.

dataset, we impute smoking rates for a target state for a varying number of control states. We then discuss theoretical properties of the synthetic control estimator, which we connect to its imputation quality in the following Section 4.

3.1 Setup and Estimator

We consider a panel of N +1 units observed over T time periods, Y = ( y it ) i ∈{ 0 ,...,N } ,t ∈{ 1 ,...,T } ∈ R ( N +1) × T , where i = 0 denotes the target unit. Our goal is to impute y 0 t for t ∈ { T +1 , . . . , T + S } given y it for i ∈ { 1 , . . . , N } , t ∈ { T + 1 , . . . , T + S } by the synthetic-control estimator ˆ y 0 t = ∑ N i =1 ˆ w i y it with convex weights ˆ w ∈ W = { w ∈ [0 , 1] N ; ∑ n i =1 w i = 1 } . Specifically, for a subset J ⊆ { 1 , . . . , N } of control units, we consider the synthetic control weights

Here, we choose the (unique) norm-minimizing synthetic control weights whenever there is more than one empirical risk minimizer. We can also interpret this solution as the limit ˆ w J of a ridge penalized synthetic control estimator ˆ w J η ,

where ˆ w J η puts a penalty on the Euclidean norm ∥ w ∥ 2 of the weights, multiplied by a factor η > 0 . This form of the penalized synthetic-control estimator is also considered by Shen et al. (2022).

We note that, unlike in the linear-regression case, we can now end up with non-interpolating solutions even in the case of high model complexity (many control units). The reason is that the convexity restriction ˆ w ∈ W allows for interpolation only if the target outcomes are in the convex hull of the control outcomes.

3.2 Empirical Motivation

To illustrate some properties of the synthetic control estimator, we impute California smoking rates in the Abadie et al. (2010) dataset. In that dataset, California experiences the introduction of smoking legislation in 1989, for which Abadie et al. (2010) provides a causal effect estimate by imputing counterfactual smoking rates for those years when the legislation is in effect. We instead consider only the time before the legislation is introduced, giving us access to observed control outcomes in all years, even for California. Specifically, we fit synthetic control models for California on T = 3 years of data (1984-1986), and evaluate their imputation performance on the following S = 2 years (1987-1988) in terms of mean-squared error.

When fitting synthetic control models, we vary how many of the N = 20 control states we include in the estimation process. Specifically, for a given complexity ℓ ∈ { 1 , . . . , N } , we average out-of-time

root-mean squared error (RMSE) across all ( N ℓ ) combinations of control states. We report results in Figure 4.

0

5

10

15

20

Number of control states

0

5

10

15

20

RMSE (daily cigarette packs)

Out-of-time

Training

Unlike the linear-regression case, the loss in the synthetic-control case changes monotonically: as we increase the number of control units, average RMSE decreases. We therefore observe a single-descent curve in the relationship of complexity and loss, with no notable change in regimes when the number of control states surpasses the number T = 3 of training periods.

As before, our illustration is extreme: by using only three training periods and a random selection of control states, we can provide a stark illustration of the difference in behavior between the synthetic control and linear-regression estimators.

3.3 Theoretical Properties and Graphical Illustration

While linear regression and synthetic control behave differently in terms of their double- vs singledescent behavior, we note that both exhibit continuing returns to increasing complexity, with no limit. In our empirical illustration, that return to complexity kicks in in the interpolation regime for linear regression and throughout for synthetic control. We now connect this commonality in returns to complexity to a corresponding theoretical connection.

Proposition 4 (Model averaging for synthetic control) . For all J with | J | > 1 and data Y there exists

Hence, synthetic control has the same model-averaging property as interpolating linear regression (Proposition 1). Now, however, the model-averaging property also holds without interpolation, and is instead driven by the convexity of synthetic control weights. Furthermore, the weights can depend on outcome data, although only for pre-treatment outcomes. We further note that the result extends to penalized synthetic control with some fixed penalty parameter η .

We illustrate the model-averaging property for synthetic control in Figure 5, where we show that synthetic California with | J | = 2 (left) and | J | = 3 (right) control states is a convex combination of synthetic California with fewer control states. Here, we focus on the fit in the training data, and do not explicitly show the underlying synthetic-control weights. We note that Figure 5 covers both cases where the synthetic estimator for California has perfect training fit (right) and cases where it does not (left).

Unlike in the case of linear regression (Proposition 3) we do not, however, obtain an immediate bound on the variation of synthetic control models. Indeed, as the case of | J | = 2 controls in Figure 5 clarifies, the complex model can have more variance in weights than the simple ones (which here do not vary at all), despite the model-averaging property. In the next section, we will therefore connect model averaging directly to improvements of imputation quality, without relying on explicit variation results.

MA

IL

y

i

1

y

i

2

CA

ˆ

y

{

MA

,

IL

}

CA

ˆ

y

{

MA

}

CA

ˆ

y

{

IL

}

CA

(a) Non-interpolating case, | J | = 2 .

MA

IL

NJ

y

i

1

y

i

2

CA

ˆ

y

{

MA

,

IL

}

CA

ˆ

y

{

IL

,

NJ

}

CA

ˆ

y

{

NJ

,

MA

}

CA

ˆ

y

{

MA

,

IL

,

NJ

}

CA

(b) Interpolating case,

|

J

|

= 3

.

4 Model-Averaging Based Risk Bounds

Above, we argued that more complex models are model averages over simpler models in two cases that are relevant to causal inference: interpolating linear regression and synthetic control. In this section, we discuss conditions under which model averaging leads to better imputation quality.

To unify the above cases, we now consider generic estimated functions ˆ f ∗ : X → R that can be related to simpler models ˆ f j : X → R for an index set j ∈ J by the model-averaging property

We see this model-averaging property as a purely mechanical property of estimators, which we applies to interpolating linear regression (Proposition 1) and to synthetic control (Proposition 4) by our results above

Model averaging provides some insurance against excess loss. Intuitively, being able to represent a more complex model ˆ f ∗ in terms of a model average over simpler models diversifies the risk of a bad imputation fit. When considering convex loss functions, we can make this intuition precise via a simple application of Jensen's inequality, which yields for the case of squared error that

for any target point ( y, x ) ∈ R ×X . Hence, imputation loss using the complex model is at most a weighted average over the loss of simpler models. The relationship also extends directly to estimating averages of outcomes y by averages of predictions ˆ f ( x ) , as in the case of average treatment effects in Section 2.

In principle, the simple portfolio bound in (3) leaves open the possibility that the more complex model ˆ f ∗ performs as poorly as the worst of the simpler models ˆ f j . However, for this to occur, the weights would have to be positively correlated with bad performance. A condition on imputation quality we can therefore consider imposing is that the selection of weights is not, on average, working against imputation quality. The following result formalizes this idea on a (very) high level.

Proposition 5 (Model-agnostic risk bound) . Assume that (MA) holds and that for some distribution over training and target data we have that for all permutations π : J → J

In words, if the model chosen by the data on average over some distribution is not worse than a model where we mix up weights, then the imputation performance of the complex model is not worse than the average of the imputation performances of simple models, leading to observations like those in Figures 1b and 4 where increased model complexity leads to improved imputation quality for randomly-ordered models.

We make two comments on the formal conditions of the proposition. First, it is enough to assume that (P) holds on average over permutations chosen uniformly at random. Second, the distribution behind the expectation E can incorporate prior distributions over underlying parameters, in which case the resulting bound holds on average over the same distribution.

In the examples of linear regression and synthetic control, Figure 6 illustrates the respective permuted models. In those cases, the permutation assumption (P) assumes that, on average, the model chosen by the data is not worse than a model where we mix up the weights (in gray).

β

1

β

2

ˆ

β

{

2

}

ˆ

β

{

1

}

ˆ

β

{

1

,

2

}

ˆ

β

{

1

,

2

}

π

(a) Interpolating linear regression

MA

IL

y

i

1

y

i

2

CA

ˆ

y

{

MA

,

IL

}

CA

ˆ

y

{

MA

,

IL

}

π,

CA

ˆ

y

{

MA

}

CA

ˆ

y

{

IL

}

CA

(b) Synthetic control

While more primitive conditions may be helpful to judge when a condition like (P) holds, we note two attractive properties. First, the assumption complements the model-averaging property (MA) in an important way: While model averaging relates more complex to less complex models, the permutation property only compares models of comparable complexity (assuming that there are no systematic ex-ante differences between the ˆ f j ). To violate this property would thus amount to assuming that selection among models with comparable complexity is disadvantageous, which may be unreasonable to expect on average. Second, we can formulate this condition on the level of estimators, without explicit reference to the underlying data-generating process.

5 Conclusion

We study the imputation performance of interpolating linear regression and synthetic control, and provide a unified perspective on returns to complexity in both cases: More complex models can be expressed as model averages over simpler ones. While we provide some high-level assumptions on when this model-averaging property improves average imputation risk, more work is needed to establish primitive sufficient conditions. This includes, in particular, studying how the bias changes as models become more complex. In addition, we limit our analysis to comparing more complex to simpler models when features or control units are randomly ordered, but in practice, we may have knowledge about which simple models are more plausible than others. However, our results show that highly over-parameterized models that achieve perfect in-sample fit can yield measurable performance improvements over non-random simple models in causal settings. Future research could explore conditions under which this phenomenon holds, namely where complex models can beat non-random simple ones.

Acknowledgments and Disclosure of Funding

We thank Tengyuan Liang, Sendhil Mullainathan, Ashesh Rambachan, audiences at Emory University, the 2023 'Econometrics in the Era of Machine Learning' conference at the University of Chicago, the 2023 'HDMetrics: Big Data, High-Dimensional Methods, and Machine Learning' workshop at the University of Illinois at Urbana-Champaign, the 2023 CEME Conference for Young Econometricians at Georgetown University, and the 2023 California Econometrics Conference at the University of Washington, as well as four anonymous reviewers for helpful comments and discussions.

Computational support was provided by the Data, Analytics, and Research Computing (DARC) group at the Stanford Graduate School of Business (RRID:SCR_022938).

References

Abadie, Alberto, Alexis Diamond, and Jens Hainmueller (2010). Synthetic control methods for comparative case studies: Estimating the effect of california's tobacco control program. Journal of the American statistical Association , 105(490):493-505. (Cited on pages 2, 6, 7, and 17.)

Abadie, Alberto and Jérémy L'Hour (2021). A penalized synthetic control estimator for disaggregated data. Journal of the American Statistical Association , 116(536):1817-1834. (Cited on page 3.)

Agarwal, Anish, Munther Dahleh, Devavrat Shah, and Dennis Shen (2021). Causal matrix completion. arXiv preprint arXiv:2109.15154 . (Cited on page 3.)

Athey, Susan, Mohsen Bayati, Nikolay Doudchenko, Guido Imbens, and Khashayar Khosravi (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association , 116(536):1716-1730. (Cited on page 3.)

Bartlett, Peter L, Philip M Long, Gábor Lugosi, and Alexander Tsigler (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences of the United States of America , 117(48):30063-30070. (Cited on pages 3 and 5.)

Belkin, Mikhail (2021). Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248. (Cited on page 3.)

Belkin, Mikhail, Siyuan Ma, and Soumik Mandal (2018). To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning , pages 541-549. PMLR. (Cited on page 3.)

Ben-Michael, Eli, Avi Feller, and Jesse Rothstein (2021). The augmented synthetic control method. Journal of the American Statistical Association , 116(536):1789-1803. (Cited on page 3.)

Bruns-Smith, David, Oliver Dukes, Avi Feller, and Elizabeth L Ogburn (2023). Augmented balancing weights as undersmoothed regressions. (Cited on page 3.)

Claeskens, Gerda and Nils Lid Hjort (2008). Model selection and model averaging . (Cited on page 3.)

Dehejia, Rajeev H and Sadek Wahba (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American statistical Association , 94(448):10531062. (Cited on pages 2, 4, and 16.)

Dehejia, Rajeev H and Sadek Wahba (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and statistics , 84(1):151-161. (Cited on pages 2, 4, and 16.)

Doudchenko, Nikolay and Guido W Imbens (2016). Balancing, regression, difference-in-differences and synthetic control methods: A synthesis. Technical report, National Bureau of Economic Research. (Cited on page 3.)

Hansen, Bruce E (2007). Least squares model averaging. Econometrica , 75(4):1175-1189. (Cited on page 3.)

Hastie, Trevor, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani (2022). Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986. (Cited on pages 3 and 5.)

Kato, Masahiro and Masaaki Imaizumi (2022). Benign-overfitting in conditional average treatment effect prediction with linear regression. arXiv preprint arXiv:2202.05245 . (Cited on page 3.)

Kelly, Bryan T, Semyon Malamud, and Kangying Zhou (2022). The virtue of complexity in return prediction. Technical report, National Bureau of Economic Research. (Cited on page 3.)

LaLonde, Robert J (1986). Evaluating the econometric evaluations of training programs with experimental data. The American economic review , pages 604-620. (Cited on pages 2, 4, and 16.)

Liang, Tengyuan and Alexander Rakhlin (2020). Just interpolate: Kernel 'Ridgeless' regression can generalize. The Annals of Statistics , 48(3):1329-1347. (Cited on page 3.)

Liang, Tengyuan, Alexander Rakhlin, and Xiyu Zhai (2020). On the Multiple Descent of MinimumNorm Interpolants and Restricted Lower Isometry of Kernels. In Abernethy, Jacob and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 2683-2711. PMLR. (Cited on pages 3 and 5.)

Liang, Tengyuan and Benjamin Recht (2023). Interpolating Classifiers Make Few Mistakes. Journal of machine learning research: JMLR . (Cited on page 3.)

Mei, Song and Andrea Montanari (2022). The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75(4):667-766. (Cited on page 3.)

Shen, Dennis, Peng Ding, Jasjeet Sekhon, and Bin Yu (2022). Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data. (Cited on pages 3 and 7.)

Wilson, Andrew G and Pavel Izmailov (2020). Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems , 33:4697-4708. (Cited on page 3.)

Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals (2016). Understanding deep learning requires rethinking generalization. (Cited on page 3.)

A Proofs

Proof of Proposition 1. We provide a direct proof for the choice λ j = 1 -X ′ j ( X J X ′ J ) -1 X j | J |-n via the Sherman-Morrison-Woodbury formula. We first note that, for k ≥ n , A ∈ R n × k of full row rank n , a ∈ R n , and ˆ α = arg min α ∈ R k ; Aα = a ∥ α ∥ we have that ˆ α = A ′ ( AA ′ ) -1 a. Indeed, A ˆ α = a , and for any α ∈ R k with Aα = a and α = ˆ α , for Π = A ′ ( AA ′ ) -1 A we have that

̸

We next write X J ∈ R n × k for the matrix with columns X J j = X j for j ∈ J and X J j = 0 for j / ∈ J . Applying the above result to ˆ β J and ˆ β J \{ j } for all j ∈ J , we find

Using that X J \{ j } X ′ J \{ j } = X J X ′ J -X j X ′ j , which is invertible by the assumption that X J \{ j } is of full row rank, we find by the Sherman-Morrison-Woodbury that

̸

Since ∑ j ∈ J X j X ′ j = X J X ′ J and ∑ j ∈ J X { j } = X J , we have that

Finally, X ′ j ( X J X ′ J ) -1 X j ≥ 0 since X J X ′ J positive definite, X ′ j ( X J X ′ J ) -1 X j ≤ 1 since X J \{ j } X ′ J \{ j } = X J X ′ J -X j X ′ j ⪯ X J X ′ J in (4), and ∑ j ∈ J X ′ j ( X J X ′ J ) -1 X j = tr ( ∑ j ∈ J X j X ′ j X J X ′ J ) = n , so λ j ∈ [0 , 1] for all j ∈ J and ∑ J j =1 λ j = 1 .

Proof of Proposition 2. The result follows from Proposition 3 by noting that, for two indpendent draws Y A and Y B for fixed X , we have that

(and the same for J \ { j } ), and thus

Proof of Proposition 3. Consider first the case | J | ≤ n . Under Assumption 1, we define the projection matrices

Since Π J \{ j } = Π J \{ j } Π J , we have that X ˆ β J \{ j } = Π J \{ j } Y = Π J \{ j } Π J Y = Π J \{ j } ˆ β J . Therefore,

Consider now the case | J | > n . Using the notation from the proof of Proposition 1, under Assumption 1 we have that X J ˆ β J = X ˆ β J = Y = X ˆ β J \{ j } = X J ˆ β J \{ j } and thus Π ˆ β J = Π ˆ β J \{ j } (as well as ( I -Π) ˆ β J = 0 ) for the projection matrix Π = X J ′ ( X J X ′ J ) -1 X J ∈ R k × k . As a consequence,

Proof of Proposition 4. Building upon the notation from Section 3.1, for J ⊂ { 1 , . . . , N } write W J = { w ∈ W ; w j = 0 for all j / ∈ J } (where W = { w ∈ [0 , 1] N ; ∑ N i =1 w i = 1 } is the N -1 -simplex) and let ∂ W J = ⋃ j ∈ J W J \{ j } ⊆ W J be the boundary of W J . For outcomes, it will also be convenient to write X = ( y it ) t ∈{ 1 ,...,T } ,i ∈{ 1 ,...,N } ∈ R T × N for the pre-treatment outcomes of the control units (with columns representing units), and y = ( y 0 t ) t ∈{ 1 ,...,T } ∈ R T for the pre-treatment outcomes of the treated unit.

As the first step, we note that we can express the quality of synthetic control weights w ∈ W J as

in terms of the fitted values y J = Xw J for the solution w J to a relaxed problem that drops the non-negativity constraint. That solution with weights in W ∗ = { w ∈ R N ; ∑ n i =1 w i = 1 } is defined, analogously to the synthetic-control solution in (1), as

̸

For this solution, we note that ∥ Xw -y ∥ 2 = ∥ X ( w -w J ) ∥ 2 + 2( w -w J ) ′ X ′ ( Xw J -y ) + ∥ Xw J -y ∥ 2 . Assume now that ( w -w J ) ′ X ′ ( Xw J -y ) = 0 . Then there is some ε = 0 such that w J ( ε ) = w J -( w -w J ) ε ∈ W ∗ with w j = 0 for j / ∈ J fulfills ∥ Xw J ( ε ) -y ∥ < ∥ Xw J -y ∥ , contradicting the choice of w J . Hence we must have that ∥ Xw -y ∥ 2 = ∥ X ( w -w J ) ∥ 2 + ∥ Xw J -y ∥ 2 = ∥ Xw -y J ∥ 2 + ∥ y J -y ∥ 2 , which establishes (5).

̸

As the second step, we note that we can therefore define the synthetic control solution in (1) in terms of the fitted values y J of the relaxed solution as

This follows immediately from (5) since ∥ y J -y ∥ is not affected by the choice of w ∈ W J . Similarly, for the constrained solutions with index set J \ { j } for j ∈ J , we have that

since W J \{ j } ⊆ W J for all j ∈ J .

As the third (and central) step, we use Farkas' lemma to argue that there exist λ ∈ R J with λ j ≥ 0 for all j ∈ J such that Xw J = ∑ j ∈ J λ j Xw J \{ j } .

̸

Assume first that X ˆ w J = y J . Then we must have that ˆ w J ∈ ∂ W J . Indeed, if ˆ w J ∈ W J \ ∂ W J then there exists some ε > 0 such that ˆ w J ( ε ) = ˆ w J (1 -ε ) + w J ε ∈ W J , for which ∥ X ˆ w J ( ε ) -y J ∥ = ∥ X ( ˆ w J ( ε ) -w J ) ∥ = (1 -ε ) ∥ X ( ˆ w J -w J ) ∥ < ∥ X ˆ w J -y J ∥ , contradicting the choice of ̂ W J and

ˆ w J . Hence ˆ w J ∈ ∂ W J , so there is some j with ˆ w J ∈ W J \{ j } , which implies that ˆ w J \{ j } = ˆ w J and X ˆ w J = X ˆ w J \{ j } . This means that we can choose λ as the indicator for component j .

Assume now that X ˆ w J = y J , and that there exists no such λ . Then, by Farkas' lemma, there exists v ∈ R T \ { 0 } such that v ′ X ˆ w J < 0 and v ′ X ˆ w J \{ j } ≥ 0 for all j ∈ J . Define the projection matrix Π = vv ′ v ′ v ∈ R T × T , and let W ∗ = arg min w ∈W J ;Π X ( w -ˆ w J )= X ( w -ˆ w J ) v ′ Xw ⊆ W J . Then the minimum is attained at a boundary point w ∗ ∈ W ∗ ∩ ∂ W J of the feasible set. Indeed, the feasible set is non-empty since it includes ˆ w J , and it is compact and convex. The minimum of the linear function is therefore attained at a boundary point, which is in ∂ W J . As a consequence, w ∗ ∈ W J \{ j } for some j ∈ J . Since ˆ w J ∈ W J , we have that v ′ Xw ∗ ≤ v ′ X ˆ w J < v ′ X ˆ w J \{ j } . Hence there is some ε ∈ (0 , 1] such that ˆ w J \{ j } ( ε ) = ˆ w J \{ j } (1 -ε ) + w ∗ ε ∈ W J \{ j } fulfills v ′ X ˆ w J \{ j } ( ε ) = v ′ X ˆ w J . Since we therefore have Π X ˆ w J \{ j } ( ε ) = Π X ˆ w J , as well as ( I -Π) X ˆ w J \{ j } ( ε ) = ( I -Π) X ( ˆ w J \{ j } (1 -ε ) + ε ˆ w J ) since Π X ( w ∗ -ˆ w J ) = X ( w ∗ -ˆ w J ) , we have that

contradicting the choice of ˆ w J \{ j } . Hence, such λ must exist.

As the fourth step, we expand the previous result on fitted values to the weights themselves in the case of penalized synthetic control, and show that the weights sum to one in that case. To this end, note that we can write the penalized synthetic control estimator from (2) as ˆ w J η = arg min w ∈W J ∥ Xw -y ∥ 2 + η ∥ w ∥ 2 . Write now ˜ X J η = ( X ′ J X J + η I ) 1 / 2 ∈ R J × J for the symmetric positive-definite matrix square root of the symmetric positive-definite X ′ J X J + η I , where X J is a matrix of the columns of X with index in J , and ˜ y J η = ( ˜ X J η ) -1 X ′ J y ∈ R J . For w J the entries of w ∈ W J corresponding to the index set J , we find

Hence, we can write (noting that W J \{ j } ⊆ W J )

so we can interpret penalized synthetic control on units J and J \ { j } with time periods { 1 , . . . , T } and the original outcomes as non-penalized synthetic control on units J and J \{ j } with time periods J and transformed outcomes, where we note that the synthetic-control solutions are unique in this case. Hence, we can apply the previous result to conclude that there exists λ η ∈ R J with λ η,j ≥ 0 for all j ∈ J such that ˜ X J η ˜ w J η = ∑ j ∈ J λ η,j ˜ X J η ˜ w J \{ j } η . Since ˜ X J η is invertible, it now also follows that ˜ w J η = ∑ j ∈ J λ η,j ˜ w J \{ j } η . Since also ˜ w J η ∈ W and ˜ w J \{ j } η ∈ W for all j ∈ J , we have that ∑ j ∈ J λ η,j = ∑ j ∈ J λ η,j 1 ′ ˜ w J \{ j } η = 1 ′ ˜ w J η = 1 . This establishes the main claim of the proposition for penalized synthetic control.

As the fifth and final step, we derive the main result on minimum-norm synthetic control from the above results on penalized synthetic control. Consider some sequence ( η ι ) ∞ ι =1 in (0 , ∞ ) with η ι → 0 , and for every ι apply the previous step to the penalized synthetic control estimator with penalty η ι to obtain a weight vector λ η ι ∈ Λ J = { λ ∈ [0 , 1] J ; ∑ J j =1 λ j = 1 } . Since Λ J is compact, ( λ η ι ) ∞ ι =1 must have a converging subsequence with some limit λ ∈ Λ J . Using the limit along this subsequence, we have that

Proof of Proposition 5. By Jensen's inequality applied to an average over the bounds in (3),

B Details of the Empirical Illustrations

B.1 Many-Regressor Linear Least-Squares on CPS Data

We utilize the publicly available 5 CPS control and NSW experimental control datasets, drawn from the study presented in LaLonde (1986) as used by Dehejia and Wahba (1999, 2002). The resulting data has 15,992 observations for CPS and 260 for NSW, with both datasets containing an identical set of variables, detailed in Table 1.

We use re78 as the outcome variable and all other variables as covariates. In order to achieve high dimensionality, we first discretize the continuous income covariates into 50 bins via quantile binning. We then construct a series of dummies for each discrete variable, corresponding to indicators for each discretized value. We then interact all these dummy variables, as well as those covariates which were originally dummies, taking care not to interact those which are mutually exclusive (e.g. originating from the same original covariate or corresponding to race). We then drop any interactions that are zero for all observations in the data. The resulting transformed dataset contains 8,408 dummy covariates, as well as the unmodified outcome variable. In order to ensure that the covariate matrix is full row rank for an arbitrary subset of columns, we go on to add iid N (0 , 0 . 0004) noise to each of the covariate values (again leaving the outcome variable unaffected). We then select a random subset of 3,000 observations from the CPS dataset as our in-sample set, using the 260 NSW observations as our out-of-sample set.

For fitting models of varying complexity, we randomly permute the order of the columns of the covariate matrix; denote the resulting matrix as X . We then add an intercept and iterate over varying levels of complexity ℓ , ranging from 1 to 8,409, corresponding to the number of covariates that we will use for estimation. We then estimate the OLS coefficient vector ˆ β ℓ = X ℓ † y , where † denotes the Moore-Penrose pseudoinverse.

To evaluate performance, we first select a sample size m and then draw 1000 samples of size m from the NSW set. We then take define our evaluation metric to be the RMSE across these 1000 samples, with error defined as the difference between the average predicted outcome and the true average outcome for each sample:

This metric assesses the ability of the given model to accurately predict mean outcomes, a quantity of direct relevance to ATE estimaiton. Where X ∗ , y ∗ denote the out-of-sample covariate and outcome variables, respectively, and can correspond to either the CPS or NSW held-out samples. The subscript ji refers to the i th observation of the j th sample of size m . In order to smooth out the effects of the random ordering of columns, we repeat this exercise for five different random orderings and take a pointwise average to obtain smooth RMSE vs. complexity curves (Figure 1b). A zoomed-in version of this plot, focusing on the highly overparametrized regime and emphasizing the ability of some of the depicted models to outperform simple baselines can be found in Figure 7.

5000

5500

6000

6500

7000

7500

8000

8500

Number of covariates

0

1

2

3

4

5

ATE RMSE (thousands $)

m = 5

m = 50

m = 100

As a further illustration of the effects of increasing complexity, Figure 8 shows the average norm of the model coefficients across different model complexities, with the average taken over random covariate orderings as before. Starting small, model coefficients initially grow (in terms of their Euclidean norm), reaching their peak at the interpolation threshold. To the right of the interpolation threshold, the norm of the model coefficients decreases mechanically, since the estimator now minimizes the norm among all interpolating solutions with fewer and fewer sparsity constraints.

0

2000

4000

6000

8000

Number of covariates

0

250

500

750

1000

1250

1500

1750

2000

Coefficient norm

B.2 Many-Unit Synthetic Control on Smoking Data

For our synthetic-control exercise, we utilize public data from the Centers for Disease Control and Prevention 6 containing annual cost, revenue, tax, and quantity data for cigarette sales by state for the years 1970 to 2019. We follow the approach of Abadie et al. (2010) in using synthetic control to estimate per-capita cigarette pack consumption for the target state, California, as a function of the other 49 states and Washington, D.C. For our evaluation, we utilize two years of data (1987 and 1988) as a hold-out sample and fit the model on three years (1984 to 1986). All of our data precedes the year in which anti-smoking legislation took effect in California (1989).

We begin by selecting a random subset of 20 states to serve as our donor pool, for computational tractability. We then select a level of complexity ℓ and select a subset of ℓ states from the chosen 20. Using that subset, we then estimate synthetic control weights based on the in-sample period,

choosing convex weight vector ˆ w ℓ ∈ W = { w ∈ [0 , 1] N ; ∑ n i =1 w i = 1 } as described in Section 3.1:

Here, y 0 denotes the target state, California. We then compute the out-of-sample prediction error as:

We then iterate over all ( 20 ℓ ) possible combinations of donor units for the given complexity level and take the average RMSE value to be the predictive error for the given complexity level. We vary ℓ from 1 to 20 to trace out the curve of synthetic control prediction risk vs. complexity (Figure 4).

As a robustness check, we also consider synthetic control with 10 pre-treatment periods, where the included control states are chosen among all 50 available donors (49 other states and Washington, D.C.). In particular, at each complexity level ℓ we consider min { ( 50 ℓ ) , 10000 } combinations of donor units. The results are qualitatively similar, and provided in Figure 9.

0

10

20

30

40

50

Number of control states

0

5

10

15

20

RMSE (daily cigarette packs)

Out-of-time

Training

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates over-parametrized models in causal inference, specifically focusing on high-dimensional linear regression models and high-dimensional synthetic control estimators with a large number of control units. The authors examine the prediction risk behaviors associated with these two estimators, and present a unified theoretical perspective on the descent phenomena in the interpolating (=high-dimensional) regime, which they refer to as the “model-averaging property” as demonstrated in Proposition 1 and Proposition 4; also see Eq. (MA). By introducing the high-level assumption of the model-averaging property (MA), which posits that the full model can be expressed as a convex combination of simpler “leave-one-out” counterparts, and another technical assumption (P) concerning the optimality of the aggregated model against a random permutation of the weights, the authors assert that the complex model cannot perform worse than the average performance of the simpler models (Proposition 5). \n\nIn summary, this paper provides a concise exposition of the “benign overfitting” phenomena observed in two estimators widely utilized in causal inference. The authors rely on two high-level geometric assumptions, the model-averaging property and Assumption (P), which provide insights into the descent behavior of these estimators and warrant further investigations in follow-up studies.\n\nSTRENGTHS:\nThis paper offers a simple yet intriguing perspective on understanding the double descent phenomenon in the interpolating regime by leveraging the mechanical properties of predictive estimators that are largely agnostic about the underlying data generating process. The authors adeptly combine knowledge from linear algebra and convex analysis to provide concise explanations for the returns of complex models in high-dimensional causal estimators. The practical implications of their findings, particularly in the context of synthetic control with many control units, are noteworthy. The paper alleviates concerns about overfitting when using a large number of control units, eliminating the need for pre-selecting an appropriate subset, which could be challenging in practice.\n\nFurthermore, the authors' focused approach on linear regression models and synthetic control estimators enhances the clarity and concreteness of their arguments. They effectively develop their findings in these two settings first, and thereafter, hinting at the potential extension of the abstract results based on the model-averaging property (MA) and the permutation property (P) to broader settings beyond the two examined scenarios.\n\nWhile it is yet challenging to assess the significance of this work within the broader NeurIPS readership, I believe the authors make a meaningful contribution to the sub-community of econometrics/causal inference. The paper offers a fresh and clear perspective on understanding the phenomenon of benign overfitting, delivering an important message that reassures the practical utilization of high-dimensional synthetic control, specifically.\n\nWEAKNESSES:\nAlthough this paper has several strengths, there are three concerns/suggestions that could be addressed to further improve its quality.\n\nFirstly, there is a concern about the possibly limited scope of applicability for the presented results. Although the paper establishes theoretical support for the concept of \"benign overfitting\" in synthetic control estimators and acknowledges the possibility of extending the analysis to a broader class of estimators through the abstract conditions (MA) and (P), it remains unclear how far-reaching this perspective can be and to what extent it can be extended. It would be valuable if the authors could provide a discussion on the expected scope of extensions, limitations, and potential challenges that may arise.\n\nAdditionally, in order to strengthen the positioning of the paper, it would be beneficial to provide additional comparisons and contrasts to previous approaches. Given the focus on synthetic control estimators, the authors could comment on the limitations of applying existing results on double descent and benign overfitting in over-parametrized models to the specific context of synthetic control estimators. This would effectively highlight the authors' contributions in this work and emphasize the distinctive features of the techniques employed.\n\nLastly, while the paper primarily presents a novel theoretical perspective, it would be valuable to complement the theory with a more extensive set of numerical experiments. For instance, conducting an ablation study with synthetic datasets to verify the assumptions and theory, as well as performing experiments with real-world datasets at scale to confirm the descent behavior of the synthetic control method, could strengthen the paper's contributions. By incorporating such experiments, the authors can provide empirical evidence to support their theoretical findings and enhance the practical relevance of the paper.\n\nBy addressing these concerns, the authors would be able to further strengthen their contributions in this paper, clarify the scope of their results, highlight its distinctive features, and provide a more comprehensive explanation for the implications of their findings.\n\nQUESTIONS:\n1.  I would like to request the authors to address the three concerns raised in the “Weaknesses” section.\n\n2.  Miscellaneous suggestions:\n  (a) It would be clearer if the authors replace the term \"full rank\" with \"full row rank\" for clarity, e.g., in lines 127, 171, 174, 191, although it is obvious in the interpolating regime. \n  (b) In lines 153-155, the authors state, \"In this case, loss continues to decrease throughout, ultimately reaching a minimum that is below the lowest error achieved left of the interpolation threshold.\" However, this is not easily discernible in Figure 1-(b).\n\nLIMITATIONS:\nThe paper suggests several potential avenues for future research, recognizing some limitations of their findings, such as exploring more fundamental conditions beyond (P). However, it could benefit from a more explicit discussion on the limitations and potential negative societal impact of the proposed approaches. It would be valuable for the authors to provide a more detailed acknowledgment of the technical limitations of their methods and offer insights into the potential adverse consequences that may arise when applying these approaches in real-world settings, specifically in the context of synthetic control methods. Nonetheless, it is important to note that given the paper's primary focus on theoretical aspects, a comprehensive and extensive examination of these limitations may not be deemed critical.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the issue of double and single descent for linear regression as well as synthetic control. For the former, empirical motivation is given in the context of predicting wages and theoretical results are given from the perspective of model averaging. For the latter, empirical motivation is given in the context of imputing counterfactual California smoking rates and again theoretical results are given from the model averaging perspective.\n\nSTRENGTHS:\n- To my reading, the most impressive result is Proposition 4, which shows that synthetic control has the model-averaging property. I presume that this is new in the literature but it is not explicitly mentioned in the paper. It would be good to clarify the novelty of Proposition 4 and emphasize its implications.\n- The two numerical examples are illustrative and provide very good empirical motivation.\n\nWEAKNESSES:\n- To my reading of the literature, one of the most under-explored but central issues in double descent or benign overfitting is the analysis of bias. For example, Tsigler and Bartlett (2023, JMLR) entitled \"Benign overfitting in ridge regression\" provides sharp bounds for the bias term. The current paper does not provide any result regarding the bias term, which will not be zero with over-parametrization.\n- It seems that it is not totally new to combine model averaging with double descent in the literature. For example, see the following quote from Wilson and Izmailov (2020, NeurIPS, page 8) entitled \"Bayesian deep learning and a probabilistic perspective of generalization\": _\"Double descent [e.g., 3] describes generalization error that decreases, increases, and then again decreases, with increases in model flexibility. ... However, our perspective of generalization suggests that performance should monotonically improve as we increase model flexibility when we use Bayesian model averaging with a reasonable prior.\"_ It would be helpful to provide a more thorough discussion of the literature.\n\nQUESTIONS:\n- Tsigler and Bartlett (2023, JMLR) gives general sufficient conditions under which the optimal regularization parameter is negative. In view of this, I am wondering what would happen if $\\eta$ in equation (2) is negative. In other words, would it be possible to study the penalized synthetic control estimator with a negative regularization parameter?\n- Model averaging is popular in both statistics and econometrics: e.g., Claeskens and Hjort (2008), Model selection and model averaging, Cambridge University Press; Hansen (2007), Least Squares Model Averaging. Econometrica. Some discussion of related papers on model averaging would be helpful.\n\nLIMITATIONS:\n- Lines 221-223: it is stated that \"We believe that these variance and geometric properties of linear regression are well understood in the literature and likely not new, although we are not aware of an explicit statement of the model-averaging connection between more and less complex interpolating linear-regression models.\" The statement is a bit unclear. It would be better if the literature review is more thoroughly done in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines single and double descent phenomena in two causal inference estimators: high-dimensional linear regression and synthetic control estimators with many controls. To begin, the paper starts with a high-dimensional linear regression problem and illustrates the double descent phenomena using the famous LaLonde dataset. Then, they show that complex models can be seen as the model averaging of simple models when using interpolating linear least squares estimators. Their main contribution is to explore the single and double descent phenomena in synthetic control methods for the first time in the literature. While it is commonly recommended not to include too many control units, the authors found the single descent phenomena: the performance monotonically increased as the number of control units increased. They derive the model-averaging-based risk bounds to explain the single phenomena they observed in the real-world example.\n\nSTRENGTHS:\n**Originality** \nTo the best of my knowledge, this is the first paper to explore the single and double descent phenomena in causal inference settings. In particular, for synthetic control problems, there are a large number of control units in many important settings. For example, when a company tries to estimate the effect of a certain internal policy on its performance, there are potentially a large number of firms they can use as the donor pool. The conventional recommendation is to focus only on a small number of selected control units, but this paper opens up a new opportunity for researchers to incorporate a large number of control units beyond the number of pre-treatment periods allowed. \n\n**Quality** \nThis paper provides a very useful insight into the single and double descent phenomena from a model averaging perspective. Providing both intuitive and geometric interpretations of results was helpful.\n\n**Clarity** \nThis paper is clearly written, and starting with a simpler linear regression as a basis for the synthetic control method was also a very effective presentation. \n\n**Significance** \nAs mentioned above, it will be significant as this paper will open up a new opportunity for researchers to incorporate a large number of control units beyond the number of pre-treatment periods allowed.\n\nWEAKNESSES:\nI do not think there is any obvious weakness in the paper with respect to the goal of the paper. I have some suggestions about making the paper more relevant to realistic causal inference settings, and I list them as clarifying questions below.\n\nQUESTIONS:\n**(1) Comparison to a \"reasonable\" simple model in a bit more realistic setting**\n\nIn many deep-learning settings, it is probably difficult to think about a \"reasonable\" set of covariates as the variables might have very little substantive meaning in some applications like image detection. But, for many causal inference problems like those examined in this paper, there are many \"simple\" models that any social scientist would start with. While this paper provides very interesting insight about the single and double descent phenomenon, I think the paper would be even stronger if the authors can demonstrate the idea of double descent and over-parametrization can actually beat simple yet reasonable models. As far as I understand, in both empirical examples (Figure 1 and Figure 4), the baseline model is to include variables \"randomly\" (random simple models) and then average the performance across such random simple models. And the average of such random simple models can have low performance as it averages over many \"random\" non-reasonable models. I want to emphasize that the original authors already acknowledged this point in the paper, and I am here to encourage the authors to explicitly include some empirical evidence about this point. \n\n(1.1) LaLonde data \n\nIn the LaLonde data example, because many variables are constructed just from eight variables, it is expected that most of the 8000 variables have very low signals and a random subset of such variables can be far from informative variables. **Can the authors include the RMSE based on a simple linear regression that includes the original eight variables additively?** Can overparameterization outperform this baseline model? \n\n(1.2) Synthetic Control Problem \n\nIt is interesting to see that it only shows the single descent phenomenon. But I was wondering whether this is due to the very small number of pre-treatment periods (3). Essentially the regime before the interpolation threshold could be too small to show any bias-variance tradeoff in a meaningful sense. **Can the authors include at least ten pre-treatment periods, which is often recommended in practice?** Do we still see the single phenomenon with a larger number of pre-treatment periods? \n\n**(2) Overparametrization vs Regularization** \n\nAnother simple benchmark is a classical regularized model. For example, for the Figure 1 problem, **can the over-parameterized model beat a simple Lasso?** For the linear regression problem, the minimal-norm interpolation solution picks a solution that has the minimum norm rather than solving the objective function with a penalty term. **What is the theoretical connection between a classical penalized regression (adding a penalty to the objective function to make a solution unique) and the over-parametrized model (finding all the solutions that make the objective function equal to zero and finding one solution that has the minimum norm)?** \n\n**(3) Inference** \n\nThis paper considers the RMSE of causal estimation. But, in many causal inference settings, researchers are also interested in estimating confidence intervals. I am curious to know whether **over-parametrized models** will make inference intractable or can be addressed similarly in the double machine learning or the semiparametric inference literature (for a linear regression case; as far as I know, there is no unified inference framework for the synthetic control method yet). \n\n**(4) Connection to Super Learners** \n\nSuperLearners by van der Laan and his colleagues also use the convex combination of individual prediction methods to improve the performance of the ML prediction. **Is there any connection to SuperLeaners?**\n\nLIMITATIONS:\nThey clarify the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies overparameterized linear regression in the context of imputing data for estimation of causal effects. This involves both unconstrained regression for imputing wages in the CPS dataset and regression under a simplex constraint on the weights for synthetic controls in the California smoking rates dataset.\n\nFavorable performance is observed for the overparameterized estimators, where the number of donors is larger than the number of samples. The main contribution of the paper is an explanation of this favorable performance. It is shown that minimum norm solutions in overparamterized linear regression can be expressed as weighted averages of least squares solutions under subsets of the donors.\nThen it is claimed that such averaging results in better generalization in imputing the missing values.\n\nSTRENGTHS:\nI think that problems regarding the use of overparameterized models in causal inference are very important and interesting.\nThe paper makes simple yet non-trivial observations about the model-averaging property of min-norm linear regression models when used in causal inference tasks.\nIt offers some observations that might prove useful in practical considerations on whether one should include as many donors as possible versus selecting donors carefully from a large pool.\n\nFinally, I found the presentation quite simple and fluent. Overall, reading the paper was an enjoyable experience.\n\nWEAKNESSES:\nThe main weaknesses of the paper in my humble opinion are as follows:\n1) I think that the explanation for why model averaging results in better generalization is somewhat lacking. The closest result to a generalization bound is proposition 5, though this only proves the resulting classifier is better than the worst-case classifier. It is quite far from results on benign overfitting such as Liang and Rakhlin, Bartlett et al. and others, which bound the excess risk with respect to the optimal hypothesis and also derive conditions on the data distribution for these results to hold. The results in this paper are not dependent on properties of the data, and the generalization guarantees are quite weak, hence I am not entirely convinced that model averaging is the reason behind the improved generalization, and it might be a red herring. Hence for the unconstrained case, I am not sure whether the model averaging interpretation offers the same understanding of generalization of overparameterized models as previous works such on benign overfitting.\n2) Unless there is something I've missed in the paper, the model averaging properties seem to be properties of linear regression (and linear regression under simplex constraints) in general, and they are not specific to causal inference problems. Hence it might be useful to write the paper without the focus on causal inference, to make it appeal to a broader audience which might not be fluent with causal inference techniques. Instead, it would be nice to give the causal inference problems as examples/applications of the more generic result on the types of linear regression.\n3) The paper does not discuss the interpretation of regression weights in the overparamterized case. Giving causal interpretations (under certain conditions) to regression weights is one of the major differences in using such models for causal inference, instead of for standard prediction tasks. Hence I'd expect some kind of discussion on such interpretations in this paper, as it focuses on causal estimation.\n\nQUESTIONS:\n1) Are the results data-dependent in some way? Under what formal conditions can we give meaningful bounds in the residual error between the overparameterized solution and the optimal subset of donors (or w.r.t to some baseline donor selection method)?\n2) Is there any point in trying to give a causal interpretation to regression coefficients in the method under some standard assumptions? While my intuition is that we should not do that, it is interesting to discuss this explicitly.\n\nLIMITATIONS:\nThe authors have properly discussed limitations, there does not seem to be a concern for negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-08-01"}