Double and Single Descent in Causal Inference with an Application to High-Dimensional Synthetic Control
Authors: Jann Spiess
Guido Imbens
Amar Venugopal
Venue: NeurIPS 2023 poster (2023)
Keywords: Double descent
interpolating regression
synthetic control
causal inference

Abstract: Motivated by a recent literature on the double-descent phenomenon in machine learning, we consider highly over-parameterized models in causal inference, including synthetic control with many control units. In such models, there may be so many free parameters that the model fits the training data perfectly. We first investigate high-dimensional linear regression for imputing wage data and estimating average treatment effects, where we find that models with many more covariates than sample size can outperform simple ones. We then document the performance of high-dimensional synthetic control estimators with many control units. We find that adding control units can help improve imputation performance even beyond the point where the pre-treatment fit is perfect. We provide a unified theoretical perspective on the performance of these high-dimensional models. Specifically, we show that more complex models can be interpreted as model-averaging estimators over simpler ones, which we link to an improvement in average performance. This perspective yields concrete insights into the use of synthetic control when control units are many relative to the number of pre-treatment periods.

Double and Single Descent in Causal Inference with an Application to High-Dimensional Synthetic Control
Jann Spiess
Guido Imbens Stanford University Stanford, CA. USA
Amar Venugopal
jspiess, imbens, amarvend@stan ord. edu
Abstract
Motivated by recent literature on the double-descent phenomenon in machine learning, we consider highly wer-parameterrizd models in causal inference, in- cluding synthetic control with many control units. such models, there may be so many free parameters that the model fits the training data perfectly. We first inves- tigate high-dimensional linear regression for imputing wage data and estimating average treatment effects, where we find that models with many than sample size can outperform simple ones. We then document the performance of high-dimensional synthetic control estimators with many control units. We find that adding control units can help improve imputation performance even beyond the point where the pre- pre-treatment fit is perfect. We provide unified theoretical perspective on the performance of these nigh-dimensional models. Specifically, we show that more complex models can be interpreted model-averaging estimators over simpler ones, which we link to an improvement in average performance. This perspective yields concrete insights into the use of synthetic control when control units are many relative to the number of pre-treatment periods.
covariates
Introduction
Motivated by recent literature on the double-decent phenomenon in machine learning, investigate the properties of common econometric estimators when we increase their complexity. For high- dimensional linear regression and for synthetic control with many control units, document empirical applications where extremely over-parameterized models impute missing out-of-sample and out-of-time outcomes well. We then provide common explanation for the returns complexity in high-dimensional linear regression and synthetic control in terms of a simple model-averaging property, which link to improvement in imputation performance.
We often conceptualize the effect complexity on econometric models in terms of a bias variance trade-off: Adding complexity makes models more expressive and reduces bias. while increased overfitting leads to additional variance. In this view, an overly complex model fits overly well in the training sample, but fails to recover true parameters or generalize regression with too many variables or synthetic control with many control units may perform poorly because of overfitting and excess variance. Consistency results for high-dimensional econometric models therefore usually assume that the expressiveness of models is limited relative to the available sample size, and practical advice often highlights choosing simple models regularizing complex ones, in a way that balances bias and variance optimally to achieve good estimation and prediction performance.
data. For example, linear
Replication code is available at gittub..co/mmmaeee//caaallleesss
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A recent literature in statistics and machine learning has highlighted the surprisingly good prediction performance of extremely -ishniimessooal models that fit the data perfectly. That literature has documented so-called double-descent phenomenon for deep neural networks and other high- dimensional regression models: increasing complexity beyond the interpolation threshold which the training sample error is zero can lead to gradual reduction in variance and improvement in -alppe performance. In such cases, there are often two complexity regimes: below the interpolation threshold, the usual bias-variance trade-off leads to a decrease (first descent) and then an increase in out-of-sample loss, while beyond the interpolation threshold, out-of-sample loss decreases again (second descent).
In order to investigate the properties of highly over peeeraaattter models in causal inference, we first demonstrate a double-descent curve for linear regression in imputing wage outcomes and estimating average-treatment effects. In the Laldo (1986) sample of the Current Population Survey (CPS) drawn from Dehejia and Wahba (1999, 2002), we generate over 8,000 variables from binning and interacting the original eight demographic and employment-related variables. We then fit linear regression model on 3,000 training units and an increasing, randomly chosen subset of these variables, choosing the norm-minimizing solution once the model fits perfectly. Evaluated on the LaLonde (1986) control sample from the National Supported Work Demonstration (NSW) experiment, we observe the usual bias-variance trade-off for a low and moderate number of included covariates, where performance first increases slightly, before deteriorating substantially when approaching the interpolation threshold. Beyond the interpolation threshold, however, out-of- sample performance increases again. Ultimately, an extremely over-parannn wer-parameterized model on over 8,000 variables even outperforms less complex regressions with a randomly chosen set of covariates and achieves performance comparable to linear regression the original, unmanipulated set of covariates.
Having demonstrated the returns to complexity in high-dimensional linear regression, we document the performance of synthetic control estimators with many control units. In the California smoking data (Abadie et al., 2010), we impute missing smoking rates based on a small number of pre-treatment periods and an increasingly large number of control states. As in the case of linear regression, we see returns increasing model complexity, even beyond the point where some of the synthetic-contro models fit the training data perfectly. However, for synthetic control we do not observe an initial trade-off between bias and variance: in empirical example, the performance on future periods is always better for the more complex models, with no intermittent increase in errors. Unlike the double- descent relationship for linear regression, the performance of synthetic control in our application yields single-descent curve that only improves with complexity, no matter whether there are a few or many control states.
We then connect the returns to complexity high-dimensional linear regression and in synthetic control in terms of a simple model-averaging property. While both estimators are constructed differently and represent different regressions, they both share a common feature: More complex models can be represented averaging property applies to linear regression in the interpolation regime, as well as to synthetic control in general. The property holds purely mechanically and does not depend the training or target distributions. It relates to other model-agnostic properties of linear regression, for which we also show reduction in (conditional) variance for minimal-nomn least-squares estimators beyond the interpolation threshold.
convex averages over simpler models. We show that this model-
Having established model-averaging property for interpolation linear regression and for synthetic control, we provide high-level assumptions under which this property translates to better imputation performance. A direct consequence of model averaging is that the (convex) prediction loss of more complex model cannot be worse than the corresponding average prediction loss of simpler models, when the same weights are used to average. When the complex model average also outperforms comparable models of the same complexity, then we show that the model-averaging property translates into a reduction in average out-of-sample error relative a randomly selected simpler model. These results only put high-level, largely model-agnostic assumptions on the data-generating process, and are driven by mechanical properties of the underlying estimators.
Our results have practical implications for the use of synthetic control with many control units. Conventional wisdom may indicate that synthetic control with very large number of donor units relative to the number of pre- treatment periods is problematic, and that selecting among many, ex-ante
2

---

exchangeable units in this case may represent a challenge. However, our results imply that this is not the case: as long as ties are broken by suitable regularization procedure (such as the minimum- norm solution in our case), making ex-ante choice among many control units is not necessary. That said, ex-ante information available about which units are particularly suitable as controls, then using this information can still be helpful and improve imputation.
We build upon results on over-parameterized regression in machine learning and statistics, where double-descent curves for kernel and linear regression and the performance of norm-minimizing interpolating solutions have been studied extensively (including Liang and Rakhlin, 2020: Liang et al., 2020; Liang and Recht, 2023; Bartlett et al., 2020; Hastie et al., 2022) as part of a broader literature on double descent and interpolation in deep learning (Zhang et al., 2016; Belkin al., 2018; Belkin, 2021; Mei and Montanari, 2022). Kelly et al. (2022) documents the benefits of over- parameterized models in asset return prediction both theory and empirically. Kato and Imaizumi (2022) provides results on the estimation of conditional average causal effects using over wver-parametrized linear regression. Relative to this work on interpolating regression, we show connections to synthetic control based on simple mechanical properties that are largely agnostic about the true data-generating process. Thereby, we also relate to work on high-dimensional and regularized synthetic control (Doudchenko and Imbens, 2016; Abadie and L'Hour, 2021; Ben-Michael et al., 2021) and the
connections between synthetic control and linear regression (Athey et al., 2021; Agarwal et al., 2021; Shen et al., 2022; Bruns Smith et al., 2023).' Finally, connect averaging in econometrics and statistics (e.g. Hansen, 2007; Claeskens and Hjort, 2008). More specifically, Wilson and Izmailov (2020) consider Bayesian model averaging in deep learning, and discuss relationships to double descent. While our linear-regression solutions are closely related to available results norm-minimizing regression, we are not aware that the results on synthetic control were noted previously.
literature model
The remaining note is structured as follows. Section 2 provides an empirical example of double descent for linear regression and discusses some properties of the norm-minimizing rigees-regression estimator in the interpolating case. Section 3 discusses the relationship of complexity and imputation performance of synthetic control in an empirical example, and makes connections to the properties of interpolating linear regression. Section 4 discusses high-level consequences of the model- averaging property of interpolating linear regression and synthetic control. Section concludes by discussing some limitations and open questions.
2 Double Descent for Linear Regression
In this section, we consider imputation by high-dimensional linear regression as an illustration of the performance of highly over-parameterized models. We start with an empirical illustration wage imputation in CPS data with a varying number of randomly ordered covariates, which we evaluate in terms of its ability to estimate an average treatment effect. We then discuss theoretical properties of the interpolating linear-regression estimator, followed by a graphical illustration. These results are closely related to prior work on interpolating linear and kernel regression (including Bartlett et al., 2020; Liang et al., 2020; Hastie et al., 2022; Kato and Imaizumi, 2022).
2.1 Setup and Estimator
We consider linear- regersreegessoon estimator in data (y:, 2,) E R² from n observations of a scalar outcome and k scalar covariates. We write X (x)) Ra and Y = (y:) R. For subset J C (1,.... k} of the covariates, denote by Bd = arg min 3) the set of least-squares linear-regression estimates on J. Among these, we choose the norm-minimizing estimate / arg min ||||] Throughout, we assume that X, (eig)ie (etg)Ee(1,.... (EE E |nx|J| is of full row rank. Hence, there are multiple least-squares solu- tions. |8'1 > 1, if and only if [J] > n. In that case, Y = X3² and 3 is the minimal-norm interpolating solution (cf. Liang al., 2020).
For example, Bruns-Smith et al. (2023) shows that synthetic- contiol-ooototttoo balancing estimators with non- negative weights can be related to outcome regressions, which connects our OLS setup in Section 2 to the synthetic-control setup in Section 3.
3

---

2.2 Empirical Motivation
We impute wages in the non-experimental LaLonde (1986) sample of the Current Population Survey (CPS) using linear regression large number of covariates, based on data drawn from Dehejia and Wahba (1999, 2002). From the original eight covariates, we obtain k = 8408 explanatory variables by binning and interacting the available features. We train minimal-norm least-squares regression models on a training sample of n = 3000 randomly chosen observations, and evaluate their mean- -rean-squared error imputing average wages for subsets (of various sizes) of the 260 LaLonde (1986) National Supported Work Demonstration (NSW) experimental controls as provided through Dehejia and Wahba (1999, 2002). To these covariates add small amount of iid Gaussian noise to avoid rank deficiency when estimating linear regression. When fitting wage imputation models, we vary the set of covariates included in the regression. Specifically, we order all features randomly, For complexity E (I....;
,.....,, k), we then choose the first & covariates, and obtain the estimate A for J = (1...., f}. Our presented results reflect an average over five such random orderings of covariates.
In order to assess the viability of this approach for applications to average treatment effect (ATE) estimation, we assess each model by its ability to accurately predict the average outcome on new dataset. In particular, we consider various subsets of size m of the NSW experimental control set. For each m, we draw 1000 samples of size m without replacement from the NSW experimental control dataset. For each sample, we average the m observation-level outcome predictions and compare the result to the true mean outcome of the given sample. We then take the RMSE across the 1000 draws of size m to obtain our evaluation metric.? ?
In Figure la, we consider CPS data and plot the average root-mean-squared error (RMSE) for pointwise prediction of the wages, reporting the in-sample performance as well. The vertical dashed line denotes the interpolation threshold where l = n. Figure 1b shows the ATE RMSE metric, once again averaged over five random orderings of the covariates, for various subset sizes. The horizontal dashed lines show the RMSE of simple linear regression on the original, unmanipulated (low-dimed )iwddmmensiona)) features, colored according to corresponding sample size m. A zoomed-in version of Figure lb focusing on the highly overparametrized regime can be found in Figure 7.
Nanber C000TSTC
(a) Average out-of-sample (blue) and in-sample (orange) pointwise RMSE for CPS data.
Average counterfactual prediction RMSE varying subset sizes on NSW controls.
Figure 1: Average RMSE for linear regression for varying number of covariates.
The out-of-sample losses these illustrations show a descent loss right of the interpolation threshold (denoted by the vertical dashed line), at which point in-sample loss is zero For NSW experimental controls, out-of-sample error initially decreases (first descent), while for CPS non- experimental controls it remains initially flat. For both out-of-sample datasets, loss then goes on to increase and peaks sharply at = n, at which point the linear models start to fit perfectly. As complexity increases further, loss decreases again (second descent). In both cases, loss continues to decrease throughout. For NSW experimental controls, loss ultimately reaches a minimum that is below the lowest error achieved left of the interpolation threshold, while the loss achieved in the CPS case is similar between the right tails and the minimum on the left. Furthermore, as the sample size m increases in Figure 1b, the highly complex interpolating models outperform simple model based
For m = 1. taking 1000 draws would necessarily yield duplication; in this case, we instead simply consider the full set of 260 NSW experimental control observations. Note that results for m = 1 therefore correspond to standard (observation-Led level) RMSE calculation, analogous to the out- out-of of-sle curve in Figure la.

---

on the original set of provided features; the m 50 and m 100 curves have minima below their corresponding dashed lines.
In this setting, highly wver-paraeetrrize linear models constructed via discretizing and interacting available features exhibit performance improvements over linear model fitted on the original features. This result appears remarkable, since the heavily wver-parameterized models do not include the original non-binary covariates, but only indicators obtained from quantile binning (see Section B. for details). Further improvement could be achieved by always including the original covariates in the model.
We note that our illustration is extreme that it artificially creates a large number of low-signal covariates, for which the best model only slightly outperforms small, hand-curated model based on the original covariates. Nevertheless, this simple empirical exercise demonstrates the non- monotonicity in the relationship of complexity to variance and loss that has been studied by the literature on interpolating regression and double descent, and extends it to causal target parameters like the average treatment effect.
2.3 Theoretical Properties and Geometric Illustration
Motivated by the empirical illustration, we note some theoretical properties that are direct conse- quences of the structure of the norm-minimin com-minimizing linear least-squares estimator, and will later serve as comparison point for synthetic control. We note that formal results on the bias-variance properties terms of more primitive properties of the data-genee data-generating process are available, including in Bartlett et al. (2020); Liang et al. (2020); Hastie et al. (2022). Here, we focus illustrating properties that follow mechanically from the construction of the estimator. Our focus on comparing more complex models, with covariates J. to slightly simpler (more sparse) ones, with covariates J \ {j). where the j-th covariate dropped. Throughout, we assume that the covariate matrices are of full row rank for the more and less complex models
Assumption 1 (Full rank). The covariate matrix X, Rnx[J] with columns as well as the covariate matrices (())) E [R*((J]-11 for all j E J are full row rank.
We first consider the geometry of interpolating solutions, for which we note that more complex model can be expressed by average over simpler models.
Proposition 1 (Model averaging for interpolating linear least-squares regression). For every X and with [J] > n such that Assumption holds there exists
A 1J,,
Σ>== JEJ
such that
3J
Ex
JEJ
We can choose the weights in Proposition 1 as a function of the covariate matrix X, only. Specif- ically, weights can be chosen as X3 = IxX](X,X) xf) where X, is the j-th column X and (X))X),) -X, can be seen as the "leverage" of feature j J, analogously to the leverage of an observation the usual (low-dimensional) linear regression case."
[
gria
g012
g(2)
((2)
g(
81
(a) Non-intcrpolating case with
(b) Interpolating case with n 1.
Figure 2: Minimal-norm least-squares solutions for linear regression with J (1,2), where varies.
We thank Tengyuan Liang for pointing out this connection to as well as for suggesting direct proof via the Sherman Morrison-Wooodee formula.
5

---

The model averaging property of interpolating linear regression is illustrated in the right panel of Figure 2 for the simple case of two covariates (J = {1, 2}) and a single data point (n 1). Here, the models / E R that for the model perfectly lie on the line through 301 and [(2), with the norm- minimizing solution 3(1.2) lies between the two. In contrast, if ((1) and (12) are not interpolating (such as in the case of the left panel of Figure 2, where n = 2), then the more complex model ((1.2)
does not generally lie in the convex hull of the simpler ones.
As an immediate consequence, any interpolating model can in this case be expressed as a convex average of simpler interpolating models, 3 = EEccAii== Xy.² for all & E (n...., |/1} (provided all X are of full row rank). A particularly interesting special case is f = n, for which we express 3' as model average of just-interpolating models 3² with 8 = X
In addition to the model-averaging property, the geometry of interpolating solutions also implies that the variance of the norm- norm-minimizing linear least-squares estimator generically decreases in the n, for which the complex estimator 3 along with the simpler estimator 3/16 both fit the training data perfectly.
interpolation regime |J]
Proposition 2 (Variance reduction for linear least-squares regression). Suppose Assumption I and that has a second moment. If [J] > n then a.s. tr (3 & minjed tr Var War(33N55)]x)
The reduction in variance is a direct consequence of a more general property of the least-squares solution. Specifically, the next proposition shows that if we redraw new outcome data in the interpolation regime, then the distance between more complex solutions is smaller than the distance between less complex models.
Proposition 3 (Variation hierarchy for linear least-squares regression). For fixed X and J for which Assumption 1 holds consider two draws YA and YB yielding minimal-norm norm least-squares estimates B1, A BNG and Bg. 3 33,(1,, respectively.
1. If|J] S n, then 1131 xx 2 maxjeJ /1 -9
3-D |xxx.
2. If| > n, then 113/ -B81l S minjes 11 B1D
Here, we write | = 3 for some positive semi-definite symmetric matrix M E Re*k
In words, the variation of models across draws of the outcome data increases with complexity on the left of the interpolation threshold. while it decreases on the right. Here, the choice of norm is essential for these results to hold uniformly across simpler models.*
Figure 3 depicts this graphically. In Figure 3a, for the con-interpolating case we observe that the norm of the difference between the model coefficient vectors making use of both covariates (,1112), 3(1.2)) is larger than the norm of the differences between the coefficient vectors taking into consideration just single covariate at time. In Figure 3b for the interpolating case, we observe that the reverse is true; in this case, the norm of the difference between the complex models is smaller than the norms of the differences between the simpler models.
In practice, we may care about model properties beyond variance, and consider imputation loss beyond the above norms in the parameters. In Section 4, we will leverage the model-averaging properties from Proposition to establish such bounds on more general imputation errors.
We believe that these variance and geometric properties of linear regression are well understood in the literature and likely not new, although we are not aware of an explicit statement of the model-averaging connection between more and less complex interpolating linear-regression models.
3 Single Descent for Synthetic Control
We next consider imputation using synthetic control with many control units. As in the case of linear regression, we start with an empirical illustration. In the Abadie et al. (2010) California smoking
"The second result still holds for an alternatvee norm || = MB with M positive definite and sym- metrie, provided that the same norm is used when selecting the norm minimizing estimator in the interpolation regime.

---

32
32
((12))
3f2.2
912)
312
BULAy
812)
30.22
B(D g0)
30
B
81
(a) -on-interpolating with
(b) Interpolating case with
Figure 3: Minimal-norm least-squares solutions for draws YA (black) and Yn (orange) for linear regression with J [1, where varies
dataset, we impute smoking rates for a target state for a varying number of control states. We then discuss theoretical properties of the synthetic control estimator, which connect to its imputation quality in the following Section 4.
3.1 Setup and Estimator
We consider panel of N + 1 units observed over T time periods, Y = (O).,,.... x(N+1)xT where i denotes the target unit. Our goal is impute you for E (,. S) given for i E N.....,N,, E {T' + 1,..., T - S} by the synthetic-control estimator yo: = Wieee with convex weights v E W {w 1 E wy = =1}. Specifically, for subset (L....,,N) of control units, we consider the synthetic control weights
d arg min ||l||
W
arg min wEW:w,=Ovje
Σ You Σ watha)
(1)
WEW
Here, we choose the (unique) norm- nommmiiiimziin synthetic control weights whenever there is more than one empirical risk minimizer. We can also interpret this solution as the limit u of a ridge penalized synthetic control estimator oi
N
2 + n|
w' = lim 1
T arg min wwW Pecay--00 OvjeJ 1=1
VoE
Wilie
(2)
where dn puts a penalty on the Euclidean norm |/| of the weights, multiplied by factor >> 0. This form of the penalized synthetic-conte estimator is also considered by Shen et al. (2022).
We note that, unlike the linear-regression case, we can now end up with non-interpolating solutions in the of high model complexity (many control units). The reason is that the convexity restriction v E W allows for interpolation only if the target outcomes are in the convex hull of the control outcomes.
3.2 Empirical Motivation
To illustrate some propertics of the synthetic control estimator, we impute California smoking rates in the Abadic et al. (2010) dataset. In that dataset, California experiences the introduction of smoking legislation in 1989, for which Abadie et al. (2010) provides a causal effect estimate by imputing counterfactual smoking rates for those years when the legislation is in effectt We instead consider only the time before the legislation introduced, giving us access to observed control outcomes in all years, even for California. Specifically, fit synthetic control models for California on T years of data 1984-1986), and evaluate their imputation performance on the following S = 2 years (1987-1988) in terms of mean-squared error.
When fitting synthetic control models, we vary how many of the N 20 control states we include in the estimation process. Specifically, for given complexity & E {l...., . N), average out-of-time
7

---

root-mean squared error (RMSE) across all (*) combinations of control states. We report results in Figure 4.
RASE (dally digarette
Number of can states
Figure 4:
Average out-of-time (blue) and training (orange) RMSE for synthetic control for varying
number of control units.
Unlike the linear-regression case, the loss in the synthetic-control case changes monotonically: we increase the number of control units, average RMSE decreases. We therefore observe a single-descent curve in the relationship of complexity and loss, with no notable change regimes when the number of control states surpasses the number T of training periods.
As before, our illustration is extreme: by using only three training periods and random selection of control states, we can provide a stark illustration of the difference in behavior between the synthetic control and regresseg estimators.
3.3 Theoretical Properties and Graphical Illustration
While linear regression and synthetic control behave differently in terms of their double- vs single- descent behavior, we note that both exhibit continuing returns to increasing complexity, with no limit. In our empirical illustration, that return to complexity kicks in in the interpolation regime for linear regression and throughout for synthetic control. We now connect this commonality in returns to complexity to corresponding theoretical connection.
Proposition (Model averaging for synthetic control). For all J with [J] > 1 and data there exists
00. 1J"
Σ JEJ
such that
v
Σ A0) JEJ
Hence, synthetic control has the same model-averaginn property as interpolating linear regression (Proposition 1). Now, however, the model-averaging property also holds without interpolation, and instead driven by the convexity of synthetic control weights. Furthermore, the weights depend outcome data, although only for pre-treatment outcomes. We further note that the result extends to penalized synthetie control with some fixed penalty parameter 17.
We illustrate the -veer-gv property for synthetic control in Figure 5, where we show that synthetic Califoria with |J| = 2 (left) and |J| 3 (right) control states is convex combination of synthetic California with fewer control states. Here, we focus on the in the training data, and do not explicitly show the underlying synthetic-control weights. We note that Figure 5 covers both cases where the synthetic estimator for California has perfect training fit (right) and cases where it does not (left).
Unlike in the case of linear regression (Proposition 3) we do not, however, obtain an immediate bound on the variation of synthetic control models. Indeed, as the case of [J] = 2 controls in Figure clarifies, the complex model can have more variance weights than simple ones (which here do not vary at all), despite the model-averaging property. In the next section, we will therefore connect model averaging directly to improvements of imputation quality, without relying on explicit variation results.

---

IL MCA
(MAJIL)
(MAJLL)
MCA
(MM)
MCA
(N)
MCA
C.A
MA
CAA
MA
(IL.NN)
VCA
(MA,II.N))
SCA
NJ
(a) Non-interpolating case, |J]
(b) Interpolating case, |J]
3.
Figure 5: Synthetic-control examples for T = 2, where the set J of included units varies.
4 Model-Averaging Based Risk Bounds
Above, we argued that more complex models are model averages over simpler models in two cases that relevant to causal inference: interpolating linear regression and synthetic control. In this section, we discuss conditions under which model averaging leads to better imputation quality.
To unify the above cases, we now consider generic estimated functions f : + R that can be related to simpler models fü :2- + R. for an index set j E J by the model-averaging property
Σ Af JEJ
for some
Å [0, 11"
(MA)
JEJ
We see this model-averaging property purely mechanical property of estimators, which we applies interpolating linear regression (Proposition and synthetic control (Proposition 4) by our results above
Model averaging provides some insurance against excess loss. Intuitively, being able to represent more complex model f* in terms of model average over simpler models diversifies the risk of bad imputation fit. When considering convex loss functions, we can make this intuition precise via simple application of Jensen's inequality, which yields for the case squared error that
(y - (x)) S Σ f())
JEJ
for any target point (y.z) E R x X. Hence, imputation loss using the complex model at most a weighted average over the loss of simpler models. The relationship also extends directly to estimating averages of outcomes y averages of predictions (1r), as in the case of average treatment effects in Section 2.
In principle, the simple portfolio bound in (3) leaves open the possibility that the more complex model f" performs poorly the worst of the simpler models fJ. However, for this to occur, the weights would have to be positively correlated with bad performance. A condition on imputation quality we therefore consider imposing that the selection of weights not, average, working against imputation quality. The following result formalizes this idea on a (very) high level.
Proposition (Model-agun agnostic risk bound). Assume that (MA) holds and that for some distribution over training and target data we have that for all permutations т. +
B - i²) Bl( }=(x))
where
f
(P)
JEJ
Then we obtain the bound El(y fr (x)) Eje -((y-f(<)))
In words, if the model chosen by the data average over some distribution is not worse than a model where we mix up weights, then the imputation performance of the complex model not worse than the average of the imputation performances of simple models, leading to observations like those in Figures 1b and where increased model complexity leads improved imputation quality for aidommlyordeen models.

---

We make two comments on the formal conditions of the proposition. First, is enough to assume that (P) holds on average over permutations chosen uniformly random. Second, the distribution behind the expectation E can incorporate prior distributions over underlying parameters, in which case the resulting bound holds on average over the same distribution.
In the examples of linear regression and synthetic control, Figure 6 illustrates the respective permuted models. In those cases, the permutation assumption (P) assumes that, on average, the model chosen by the data is not worse than a model where we mix up the weights (in gray).
912
CLLY BCA
IL
(MAIL)
Ue,CA
(MAJL))
(1.2))
MAA
((2)
CAA
CA
MA
G(122)
A1
(a) Interpolating linear regression
(b) Synthetic control
Figure 6: Illustration of permutation bound based the examples from Figures 2 and 5.
While more primitive conditions may be helpful to judge when a condition like (P) holds, we note two attractive properties. First, the assumption complements the model-averaging property (MA) in an important way: While model averaging relates more complex to less complex models, the permutation property only compares models of comparable complexity (assuming that there are no systematic ex-ante differences between the f+). To violate this property would thus amount to assuming that selection among models with comparable complexity is disadvantageous, which may be unreasonable to expect on average. Second, we can formulate this condition on the level of estimators, without explicit reference to the underlying data-generating process.
5
Conclusion
We study the imputation performance of interpolating linear regression and synthetic control, and provide unified perspective returns complexity in both cases: More complex models be expressed as model averages over simpler ones. While we provide some high-level assumptions on when this model-averaging property improves average imputation risk, more work is needed to establish primitive sufficient conditions. This includes, in particular, studying how the bias changes as models become more complex. In addition, we limit our analysis comparing more complex to simpler models when features or control units are randomly ordered, but in practice, we may have knowledge about which simple models more plausible than others. However, our results show that highly over- peer-aramtterized models that achieve perfect in in-sample fit can yield measurable performance improvements over non-random simple models in causal settings. Future research could explore conditions under which this phenomenon holds, namely where complex models can beat non-random simple ones.
10

---

Acknowledgments and Disclosure of Funding
We thank Tengyuan Liang, Sendhil Mullainathan, Ashesh Rambachan, audiences at Emory University, the 2023 "Econometrics in the Era of Machine Learning" conference at the University of Chicago, the 2023 "HDMetrics: Big Data, nigh-Diimeniionl Methods, and Machine Learning" workshop the University of Illinois at Urbana-Champaa Champaign, the 2023 CEME Conference for Young Econometricians Georgetown University, and the 2023 California Econometries Conference at the University of Washington, as well as four anonymous reviewers for helpful comments and discussions.
Computational support was provided by the Data, Analytics, and Research Computing (DARC) group at the Stanford Graduate School of Business (RRID:SCR_022)) -22933).
References
Abadie, Alberto, Alexis Diamond, and Jens Hainmueller (2010). Synthetic control methods for comparative case studies: Estimating the effect of california's 's tobacco control program. Journal of the American statistical Association, 105(490):493-505 (Cited on pages 2. 6, 7, and 17.)
Abadie, Alberto and Jérémy L'Hour (2021). A penalized synthetic control estimator for disaggregated data. Journal of the American Statistical Association, 1166536):8177-))
(Cited on page 3.)
Agarwal, Anish, Munther Dahleh, Devavrat Shah, and Dennis Shen (2021). Causal matrix completion. arXiv preprint arXiv:2109, 151.544409. (Cited on page 3.)
Athey, Susan, Mohsen Bayati, Nikolay Doudchenko, Guido Imbens, and Khashayar Khosravi (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association, 116(536)
(Cited on page 3.)
Bartlett, Peter L,, Philip M Long, Gábor Lugosi, and Alexander Tsigler (2020). Benign overfitting in linear regression. Proceedings of the National Acudemy of Sciences of the United Stutes of America, 117(445)3006333000 (Cited on pages and 5.)
Belkin, Mikhail (2021). Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-248. (Cited on page 3.)
Belkin, Mikhail, Siyuan Ma. and Soumik Mandal (2018). To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pages 541-549. PMLR. (Cited on page 3.)
Ben-Michael, Eli, Avi Feller, and Jesse Rothstein (2021). The augmented synthetic control method.
Journal of the American Statistical Association, (16(53)):1889-1333 (Cited on page 3.)
Bruns-Smith, David, Oliver Dukes, Avi Feller, and Elizabeth Ogburn (2023). Augmented balancing weights as undersmoothed regressions. (Cited page 3.)
Claeskens, Gerda and Nils Lid Hjort (2008). Model selection and model averaging. (Cited page 3.)
Dehejia, Rajeev H and Sadek Wahba (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American statistical Association, 94(448):1053- 1062. (Cited pages 2, and 16.)
Dehejia, Rajeev H and Sadek Wahba (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and statistics, 84(1):151-161. (Cited on pages 2, 4, and 16.)
Doudchenko, Nikolay and Guido W Imbens (2016). Balancing, regression, difference in in-differences and synthetic control methods: A synthesis. Technical report, National Bureau of Economic Research. (Cited on page 3.)
Hansen, Bruce E (2007). Least squares model averaging. Econometrica, 75(4):1175-11) (Cited on page 3.)
Hastie, Trevor, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani (2022). Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2)999--986 (Cited on pages 3 and 5.)
11

---

Kato, Masahiro and Masaaki Imaizumi (2022). effect prediction with linear regression. arXiv preprint arXiv:2202.05245. (Cited on page 3.)
in conditional average treatment
Kelly, Bryan T. Semyon Malamud, and Kangying Zhou (2022). The virtue of complexity return prediction. Technical report, National Bureau of Economic Research. (Cited on page 3.)
LaLonde, Robert J (1986). Evaluating the econometric evaluations of training programs with experimental data. The American economic review, pages 604-620. (Cited on pages 2, 4, and 16.)
Liang, Tengyuan and Alexander Rakhlin (2020). Just interpolate: Kernel "Ridgeless" regression can generalize. The Annals of Statistics, 43(3):1329-1333 (Cited on page 3.)
Liang, Tengyuan, Alexander Rakhlin, and Xiyu Zhai (2020). On the Multiple Descent of Minimum- Norm Interpolants and Restricted Lower Isometry of Kernels. In Abernethy, Jacob and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 2683-2711, PMLR. (Cited on pages 3 and 5.)
Liang, Tengyuan and Benjamin Recht (2023). Interpolating Classifiers Make Few Mistakes. Journal of machine learning research: JMLR. (Cited on page 3.)
Mei, Song and Andrea Montanari (2022). The generalization error of random features regression: Pre-
cise asymptotics and the double descent curve. Communications Pure and Applied Mathematics, 75(4):667-766. (Cited on page 3.)
Shen, Dennis, Peng Ding, Jasject Sekhon, and Bin Yu (2022). Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data. (Cited on pages 7.)
Wilson, Andrew G and Pavel Izmailov (2020). Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 15:4697-4700 (Cited on page 3.)
Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals (2016). Under- standing deep learning requires rethinking generalization. (Cited page 3.)
12

---

A Proofs
Proof of Proposition 1. We provide a direct proof for the choice X, X-x,(X, via the formula. We first note that, for k 2 A E Bunk of full row rank a E R2, and á = arg e ||a|] we have that á = ('AAA)) a. Indeed, Aá = a, and for
1
any a E R with Aa = and a & 6, for II- = ('AAA) -1 A we have that
|all2 |T| 2 + (I mall || + (I(T [a) a)]l ||@||2 1a-aa-at at|² > ||ā|l?.
We next write X E B for the matrix with columns Xy = X3 for j E J and X = for j $
Applying the above result to 8 and B for all j E J. we find
3J = x (X,x)"
grG) = (XAG)
Using that X X)O) ( = X,X, - X,X which is invertible by the assumption that X)) is of full row rank, we find by the Moriian--ooriioooooooo that
(X^oIXmu)) -x,x=-XXx++++ with x)(X,X,)) ( -X, + 1. Plugging in,
(Xxx)) + (Xxx))
(4)
goMt = -(x -XU3((XX1)XX1)))
X
= 33'- XX -
1- XX,
+ "X"(XXXI)XXXXXXXX
1- X-X
1 =3² + (X"XXX,, Xxxx((XX3)) -x (x3) v 1 11XXXXX))"XXXXXxxXX Xx
Since
ΣjEr X,X1- X and E5er x0I _X we have that
Σ 3/ = |X JEJ
= 3d Σ Aj-
JEJ
JEJ
Finally, Xx(X,X,)) -x, 2 0 since X,XX positive definite, X((X,X))-1X < 1 since
X = X,X, - X उ X,X in (4), and Σee, X/(XXX))) (EEE = n, so l E [0, 1] for all j E and Σ/ \
Proof of Proposition 2. The result follows from Proposition 3 by noting that, for two indpendent draws YA and YB for fixed X, we have that
E[| A x = E - E [x]) (3/ - [x)])11)
E|||8X - E| | E 11
2 (
(and the same for J \ {()), and thus
18'x)=
_ BII
1
3/ - } E
min || 13 xp/x]
1 B1| - 3/ | x = tr Var (x)) 3 2
< min
Proof of Proposition 3. Consider first the case |J| < Under Assumption 1, we define the projec- tion matrices
II' = (X(X,) - 'X R"X", 11 = Xxun xx Xn(nn
Xh E ga*n
13

---

Since IA = II,,, we have that x3n00 MAMy = MAMYY = INGB.. Therefore,
|| - X ||x - - (x x1 - -11 - B 1x-
((II--VV)))(3))) x1 = - x3
2 | ( - -x
Consider now the case ||| > n. Using the notation from the proof of Proposition 1, under As-
sumption we have that x = x33= = x x3d(55 = X 33(3) and thus 1133 = 11331
(as well as (I - 11)3²* = 0) for the projection matrix 11 = X (X) E RR As consequence,
|| 3/11² = |11(3 2111 + 1141 -I - - -3200)11 )
< || - 33 3 )12 + II - -
3/ 1
Proof of Proposition 4. Building upon the notation from Section 3.1, for J C {1, {I...., N) write W = {w E W; wy 0 for all j & J} (where W {w E [0, 1] 1]", E Di= Wy 1] is the N 1- simplex) and let ow = Ujes W WW be the boundary of W For outcomes, will also be convenient to write X = (wa)ue (1e.))eEL...T.,.e((....... E RTxN for the pre-treatment outcomes of the control units (with columns representing units), and y (Uo..)E( R for the pre-treatment outcomes of the treated unit.
As the first step, we note that we can express the quality of synthetic control weights w E W as
[| - u|1 = |X - 3 1|a + 1 - yl|²
in terms of the fitted values D = X for the solution Th to a relaxed problem that drops the non-negativity constraint. That solution with weights in W {w R. >i wi = 13 is defined, analogously to the synthetic-control solution in (1), as
T
arg min || E W", wEW
W
arg min =WW* =0vjj
| yllcnl C..
For this solution, we note that ||X - y | (w
2 X'''XXX || - v12.. Assume now that (w - '''''XX''' - y) + O. Then there is some # 0 such that
y) +
w' (E) = कh (w- - wh) E E W with Wy = 0 for j & J fulfills || (E) - yll < | y|l-
contradicting the choice of w. Hence we must have that || w - --l²² = [|X w )1( + |
y||² [| w J' 112 + l! - yyl², which establishes (5).
As the second step, we note that we can therefore define the synthetic control solution in (1) in terms of the fitted values Y of the relaxed solution
arg min ||m||| E W
W
arg min || - I''WW' wEW!
wEW
This follows immediately from (5) since T|T - y/l is not affected by the choice of w E W Similarly, for the constrained solutions with index set J \ {/} for J, we have that
(( arg min ||le|| E W/),
Wh arg min ||x I' W/G)
wew w)W
wEW-iG>
since w/ C W for all j E
As the third (and central) step, we use Farkas' lemma to argue that there exist À E IR with Aj 2 0 for all j E such that Xw = EEer A\)
Assume first that X # T¹ Then we must have that v6 E OW Indeed, if w E W \ ow then there exists some E >> such that ( = w (1 - E) + w & E W for which || (E) - 7
| X (E) - w')l (1 =)) - < | I |]. contradicting the choice of W and
14

---

w. Hence w E aW so there is some j with v- E MV(G), which implies that x))) = à and Xw = = This means that we can choose À as the indicator for component j.
Assume now that X = = and that there exists no such A. Then, by Farkas' lemma, there exists E IRT \ {0) such that o' o' < 0 and v' (x" 2 0 for all j E J. Define the projection matrix IT my' &RTxT. and let W arg " (=x(==
o' Xww C WV Then the minimum is attained at a boundary point w" E W n aw of the feasible set. Indeed, the feasible set is non- non-empty since it includes v.' and it is compact and convex. The minimum of the linear function is therefore attained at boundary point, which is in awV As a consequence, w E ?W'Ms for some j E J. Since w' E W we have that e' v'Xw" S v'Xw" < x'W"" Hence there is some e E (0, 1] such that u (E) = Aot (1 (1-88 a1) + w= s se W/6) fulfills e' x'X ".w (E) = y' v'Xw". Since we therefore have IIX (ot()) = IIX as well as (I II W 10 (c) = (T- X) (1 - E) + Ewi)) since II - X(w - w²), we have
s'o
that
NOX (5) 12 = ( (E) w" )12 || (O) - w'))l I12 + - II <|ITIX )|2 + |I(T -I) - 1
0 (1- mIx w² 1
=
contradicting the choice of nV,), Hence, such À must exist.
As the fourth step, we expand the previous result on fitted values to the weights themselves in the case of penalized synthetic control, and show that the weights sum to one in that case. To this end, note that we can write the penalized synthetic control estimator from (2) Fंÿj = arg min || v|l Write now X = (X, 1/2 E JJ for the symmetric positive-definite matrix square root of the symmetric positive-definite XjX,, + n/I,, where X, is a matrix of the columns of X with index in J. and بي =
E R.. For wy the entries of w E W corresponding to
the index set J, we find
I 211 + 1 m|1vo|1² = |wn, y/1² + n||
X'xXj w. - 2w', + y'y + me w. = (X) nI)w, +'yy
= w-- 2w' " (X
+uyy = | / - 11 + 110/1
Hence, we can write (noting that W/() C CW)
ti arg min 18/100 - 7/
MM arg min || w. - wy
=EW
wEWr)))
so we can interpret penalized synthetic control on units J and J \ {i} with time periods { .4...,Ty T} and the original outcomes as non-penalized synthetic control on units J and J [i) with time periods J/ and transformed outcomes, where we note that the -ynthetic-control solutions are unique in this case. Hence, we can apply the previous result to conclude that there exists X, E R with Xy 2 for all j E J such that X jn = EEee An. X NG) Since X is invertible, it now also follows
Wig
that / = =EEEJ Anj Wy (,) Since also / E W and w/(i) E W for all j J, we have that Ejeu An. = Ejes An.jj 1' i for penalized synthetic control.
I' AM) 1' 1. This establishes the main claim of the proposition
As the fifth and final step, we derive the main result on minimum-noni synthetic control from the above results on penalized synthetic control. Consider some sequence (n.),e in (0, oo) with n,, -+ 0, and for every apply the previous step to the penalized synthetic control estimator with penalty n. to obtain weight vector A E A' (> E 1];; E/= Aÿ 1}. Since A is compact, (A,, )
must have a converging subsequence with some limit À E AJ. Using the limit along this subsequence, we have that
w
lim wh
lim Am.
MM) W n1
lim Am.i
lim (Bl) t00
w.y
Σ XGw JEJ
JEJ
JEJ
15

---

Proof of Proposition 5. By Jensen's 's inequality applied to average over the bounds in (3),
B - f()))
J/! Σ
El(y (+)))
Σ
Σ
-
JEJ
B Details of the Empirical Illustrations
B.1 Many-Regressor Linear Least-Squares on CPS Data
We utilize the publicly available" CPS control and NSW experimental control datasets, drawn from the study presented ondd (1986) as used by Dehejia and Wahba (1999, 2002). The resulting data has 15,992 observations for CPS and 260 for NSW, with both datasets containing identical set of variables, detailed in Table 1.
Variable age education
Data Type
Description
Discrete
Age
Discrete
Years of education
black
Dummy
Black
hispanic
Dummy
Hispanic
married
Dummy
Marital status
nodegree
Dummy
Lack of college degree
re74
Continuous
Income in 1974
re75
Continuous Income 1975
re78
Continuous
Income in 1978
Table 1: CPS and NSW dataset variables
We use re78 the outcome variable and all other variables covariates. In order to achieve high dimensionality, we first discretize the continuous income covariates into 50 bins via quantile binning. We then construct series of dummies for each discrete variable, corresponding indicators for each discretized value. We then interact all these dummy variables, as well as those covariates which were originally dummies, taking care not interact those which mutually exclusive (e.g. originating from the same original covariate or corresponding to race). We then drop any interactions that for all observations in the data. The resulting transformed dataset contains 8,408 dummy covariates, as well as the unmodified outcome variable. In order to ensure that the covariate matrix is go add iid ((0, 0.0004) noise each of the covariate values (again leaving the outcome variable unaffected). We then select random subset of 3,000 observations from the CPS dataset our in-sample set, using the 260 NSW observations as our out-of-sample set.
full row rank for arbitrary subset of columns,
For fitting models of varying complexity, randomly permute the order of the columns of the covariate matrix; denote the resulting matrix as X. We then add an intercept and iterate over varying levels of complexity f, ranging from to 8,409, corresponding to the number of covariates that
will use for estimation. We then estimate the OLS coefficient vector 3 x y. where denotes the Moore-Penrose pseudoinverse.
To evaluate performance, we first select a sample size and then draw 1000 samples of size from the NSW set. We then take define our evaluation metric to be the RMSE across these 1000 samples, with error defined the difference between the average predicted outcome and the true average outcome for each sample:
1000
RMSE m)
Be 74
1000
"users..berr wwwer.org//............... wo..oo////r aegg/"rdehejta/dttaa hww./tassdaaaaaaaaa html.
We
use the files corresponding to
cps_controls. txt and cowre74_contro..oa
16

---

This metric assesses the ability of the given model to accurately predict mean outcomes, a quantity of direct relevance to ATE estimaiton. Where X y denote the out-of-sample covariate and outcome variables, respectively, and can correspond to either the CPS or NSW held-out samples. The subscript ji refers to the ith observation of the jth sample of size m. order to smooth out the effects of the random ordering of columns, we repeat this exercise for five different random orderings and take pointwise average to obtain smooth RMSE vs. complexity curves (Figure lb). zoomed-in version of this plot, focusing the highly overparametrized regime and emphasizing the ability of some of the depicted models to outperform simple baselines can be found in Figure
Figure 7: Average RMSE for linear regression for a varying number of covariates (E > 5000).
As a further illustration of the effects of increasing complexity, Figure 8 shows the average norm of the model coefficients across different model complexities, with the average taken over random covariate orderings as before. Starting small, model coefficients initially grow (in terms of their Euclidean norm), reaching their peak at the interpolation threshold. To the right of the interpolation threshold, the norm of the model coefficients decreases mechanically, since the estimator now minimizes the norm among all interpolating solutions with fewer and fewer sparsity constraints.
170
300
covar alcs
Figure 8: Average of the coefficient norm || for varying size of the set J of covariates.
B.2 Many-Unit Synthetic Control on Smoking Data
For our synthetic-contrer exercise, we utilize public data from the Centers for Disease Control and Prevention containing annual cost, revenue, tax, and quantity data for cigarette sales by state for the years 1970 to 2019. We follow the approach of Abadie al. (2010) in using synthetic control to estimate per-capita cigarette pack consumption for the target state, California, as a function of the other 49 states and Washington, D.C. For our evaluation, we utilize two years of data (1987 and 1988) as hold-out sample and fit the model on three years (1984 to 1986). All of our data precedes the year in which anti-smoking legislation took effect in California (1989).
We begin by selecting a random subset of 20 states to serve as our donor pool, for computational tractability. We then select level of complexity & and select subset of states from the chosen 20. Using that subset, we then estimate synthetic control weights based on the in-sample period,
'chromi coata. cdc, www./oliccyTTee--.oo Tax- Siix-Burden--oe on- 1at0aaccoo-970-0099//111110
17

---

choosing convex weight vector vf E W fw E ;
Ei= Wys = = as described in Section 3.1:
1986
wh arg
arg min |||
W
arg min
Σ
(yor
wiika )
WEW
wEWaw, Ovj4J
1=1984
Here, yo denotes the target state, California. We then compute the out-of-sample prediction as:
RMSE(()
1
Σ 1=198
1988
(3/0
Σ wff
2
We then iterate over all (0 possible combinations of donor units for the given complexity level and take the average RMSE value be the predictive error for the given complexity level. We vary & from to 20 to trace out the curve synthetic control prediction risk vs. complexity (Figure 4).
As robustness check, also consider synthetic control with 10 pre-treatment periods, where the included control states are chosen among all 50 available donors (49 other states and Washington, D.C.). particular, at each complexity level l consider min (*7) 100000 combinations of donor units. The results are qualitatively similar, and provided in Figure 9.
Qucckiia
RASE diprrtte
Number of control states
Figure 9: Average out-of-time (blue) and training (orange) RMSE for synthetic control for varying number of control units as in Figure 4, but with ten pre-treatment periods and control units chosen randomly among all available donors.
18


Reviews:
[{"review_id": "QoNHkGec50", "reviewer": "Reviewer_TPer", "summary": "The paper studies overparameterized linear regression in the context of imputing data for estimation of causal effects. This involves both unconstrained regression for imputing wages in the CPS dataset and regression under a simplex constraint on the weights for synthetic controls in the California smoking rates dataset.\n\nFavorable performance is observed for the overparameterized estimators, where the number of donors is larger than the number of samples. The main contribution of the paper is an explanation of this favorable performance. It is shown that minimum norm solutions in overparamterized linear regression can be expressed as weighted averages of least squares solutions under subsets of the donors.\nThen it is claimed that such averaging results in better generalization in imputing the missing values.", "questions": "1) Are the results data-dependent in some way? Under what formal conditions can we give meaningful bounds in the residual error between the overparameterized solution and the optimal subset of donors (or w.r.t to some baseline donor selection method)?\n2) Is there any point in trying to give a causal interpretation to regression coefficients in the method under some standard assumptions? While my intuition is that we should not do that, it is interesting to discuss this explicitly. ", "limitations": "The authors have properly discussed limitations, there does not seem to be a concern for negative societal impact.", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "I think that problems regarding the use of overparameterized models in causal inference are very important and interesting.\nThe paper makes simple yet non-trivial observations about the model-averaging property of min-norm linear regression models when used in causal inference tasks.\nIt offers some observations that might prove useful in practical considerations on whether one should include as many donors as possible versus selecting donors carefully from a large pool.\n\nFinally, I found the presentation quite simple and fluent. Overall, reading the paper was an enjoyable experience.", "weaknesses": "The main weaknesses of the paper in my humble opinion are as follows:\n1) I think that the explanation for why model averaging results in better generalization is somewhat lacking. The closest result to a generalization bound is proposition 5, though this only proves the resulting classifier is better than the worst-case classifier. It is quite far from results on benign overfitting such as Liang and Rakhlin, Bartlett et al. and others, which bound the excess risk with respect to the optimal hypothesis and also derive conditions on the data distribution for these results to hold. The results in this paper are not dependent on properties of the data, and the generalization guarantees are quite weak, hence I am not entirely convinced that model averaging is the reason behind the improved generalization, and it might be a red herring. Hence for the unconstrained case, I am not sure whether the model averaging interpretation offers the same understanding of generalization of overparameterized models as previous works such on benign overfitting.\n2) Unless there is something I've missed in the paper, the model averaging properties seem to be properties of linear regression (and linear regression under simplex constraints) in general, and they are not specific to causal inference problems. Hence it might be useful to write the paper without the focus on causal inference, to make it appeal to a broader audience which might not be fluent with causal inference techniques. Instead, it would be nice to give the causal inference problems as examples/applications of the more generic result on the types of linear regression.\n3) The paper does not discuss the interpretation of regression weights in the overparamterized case. Giving causal interpretations (under certain conditions) to regression weights is one of the major differences in using such models for causal inference, instead of for standard prediction tasks. Hence I'd expect some kind of discussion on such interpretations in this paper, as it focuses on causal estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\n1. The question **under which conditions we obtain better generalization** goes to the heart of the issue. While the model-averaging property holds mechanically for these estimators, the resulting generalization error bounds (like the one in _Proposition 5_) require restrictions on the data-generating process. We agree that understanding lower-level conditions under which model averaging translates into better performance is an important next step. At the same time, we want to clarify that the high-level condition in _Proposition 5_ applies beyond the worst case; it provides a condition under which a classifier of higher complexity outperforms an average classifier of lower complexity, not just a worst-case classifier.\n\n2. We agree that the question of **how to interpret models** is an interesting one, and we also agree with your assessment that these coefficients should not be given a causal interpretation. This does not preclude a causal use of the resulting predictions, which can be helpful for imputing unrealized potential outcomes.\n\nIn addition, we agree that **our results hold beyond the causal-inference context**. At the same time, we believe that our unique contribution is to show that they can be particularly relevant there, since estimation challenges in causal inference have frequently been framed around bias–variance trade-offs. Developments in the literature on double descent and benign overfitting, as well as our own results on synthetic control, have the potential to amend this view in causal inference in particular. In order to further clarify the role in causal inference, we have added some results on using linear regression specifically for the estimation of average treatment effects (see **attached PDF** and our response to reviewer xBLi)."}, {"review_id": "gmUa5D4pE9", "reviewer": "Reviewer_xBLi", "summary": "This paper examines single and double descent phenomena in two causal inference estimators: high-dimensional linear regression and synthetic control estimators with many controls. To begin, the paper starts with a high-dimensional linear regression problem and illustrates the double descent phenomena using the famous LaLonde dataset. Then, they show that complex models can be seen as the model averaging of simple models when using interpolating linear least squares estimators. Their main contribution is to explore the single and double descent phenomena in synthetic control methods for the first time in the literature. While it is commonly recommended not to include too many control units, the authors found the single descent phenomena: the performance monotonically increased as the number of control units increased. They derive the model-averaging-based risk bounds to explain the single phenomena they observed in the real-world example.  ", "questions": "**(1) Comparison to a \"reasonable\" simple model in a bit more realistic setting**\n\nIn many deep-learning settings, it is probably difficult to think about a \"reasonable\" set of covariates as the variables might have very little substantive meaning in some applications like image detection. But, for many causal inference problems like those examined in this paper, there are many \"simple\" models that any social scientist would start with. While this paper provides very interesting insight about the single and double descent phenomenon, I think the paper would be even stronger if the authors can demonstrate the idea of double descent and over-parametrization can actually beat simple yet reasonable models. As far as I understand, in both empirical examples (Figure 1 and Figure 4), the baseline model is to include variables \"randomly\" (random simple models) and then average the performance across such random simple models. And the average of such random simple models can have low performance as it averages over many \"random\" non-reasonable models. I want to emphasize that the original authors already acknowledged this point in the paper, and I am here to encourage the authors to explicitly include some empirical evidence about this point. \n\n(1.1) LaLonde data \n\nIn the LaLonde data example, because many variables are constructed just from eight variables, it is expected that most of the 8000 variables have very low signals and a random subset of such variables can be far from informative variables. **Can the authors include the RMSE based on a simple linear regression that includes the original eight variables additively?** Can overparameterization outperform this baseline model? \n\n(1.2) Synthetic Control Problem \n\nIt is interesting to see that it only shows the single descent phenomenon. But I was wondering whether this is due to the very small number of pre-treatment periods (3). Essentially the regime before the interpolation threshold could be too small to show any bias-variance tradeoff in a meaningful sense. **Can the authors include at least ten pre-treatment periods, which is often recommended in practice?** Do we still see the single phenomenon with a larger number of pre-treatment periods? \n\n**(2) Overparametrization vs Regularization** \n\nAnother simple benchmark is a classical regularized model. For example, for the Figure 1 problem, **can the over-parameterized model beat a simple Lasso?** For the linear regression problem, the minimal-norm interpolation solution picks a solution that has the minimum norm rather than solving the objective function with a penalty term. **What is the theoretical connection between a classical penalized regression (adding a penalty to the objective function to make a solution unique) and the over-parametrized model (finding all the solutions that make the objective function equal to zero and finding one solution that has the minimum norm)?** \n\n**(3) Inference** \n\nThis paper considers the RMSE of causal estimation. But, in many causal inference settings, researchers are also interested in estimating confidence intervals. I am curious to know whether **over-parametrized models** will make inference intractable or can be addressed similarly in the double machine learning or the semiparametric inference literature (for a linear regression case; as far as I know, there is no unified inference framework for the synthetic control method yet). \n\n**(4) Connection to Super Learners** \n\nSuperLearners by van der Laan and his colleagues also use the convex combination of individual prediction methods to improve the performance of the ML prediction. **Is there any connection to SuperLeaners?**  ", "limitations": "They clarify the limitations of their approach. ", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "**Originality** \nTo the best of my knowledge, this is the first paper to explore the single and double descent phenomena in causal inference settings. In particular, for synthetic control problems, there are a large number of control units in many important settings. For example, when a company tries to estimate the effect of a certain internal policy on its performance, there are potentially a large number of firms they can use as the donor pool. The conventional recommendation is to focus only on a small number of selected control units, but this paper opens up a new opportunity for researchers to incorporate a large number of control units beyond the number of pre-treatment periods allowed. \n\n**Quality** \nThis paper provides a very useful insight into the single and double descent phenomena from a model averaging perspective. Providing both intuitive and geometric interpretations of results was helpful.\n\n**Clarity** \nThis paper is clearly written, and starting with a simpler linear regression as a basis for the synthetic control method was also a very effective presentation. \n\n**Significance** \nAs mentioned above, it will be significant as this paper will open up a new opportunity for researchers to incorporate a large number of control units beyond the number of pre-treatment periods allowed.", "weaknesses": "I do not think there is any obvious weakness in the paper with respect to the goal of the paper. I have some suggestions about making the paper more relevant to realistic causal inference settings, and I list them as clarifying questions below. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and thoughtful comments and questions.\n\n1. In response to your question about **reasonable comparison models**, we are reporting additional comparisons:\n   - **For linear regression**, we report a comparison to low-dimensional linear regression using the original covariates for the task of estimating average treatment effects, with a varying size of the target control group (see **attached PDF**).\nSpecifically, _Figure R1_ reports the average RMSE for estimating the average treatment effect (ATE) of subsets of varying size $m$ in the experimental NSW sample. (The results reported in _Figure 1 (a)_ correspond to $m=1$, for which the RMSE of imputing control outcomes is the same as the RMSE of estimating the corresponding treatment effect.) In the figure, we now add horizontal lines for the performance of a simple linear regression on the original eight covariates, which shows that the performance of the simple regression is better than the best interpolating regression when $m=1$, but that for $m=50$ and $m=100$ very high-dimensional over-parametrized solutions perform comparably to (and ultimately slightly better than) the simple model.\nWe see this as an encouraging finding for interpolating solutions in linear regression, since the complex models do not explicitly use the original regressors (only their binned and interacted versions) but are still able to recover comparable predictive performance. A potential improvement in performance could be to always include the original eight variables in the complex models, without any explicit or implicit regularization, in which case we would expect the comparison to be more advantageous for our complex models.\n   - **For synthetic control**, we also report results for more pre-treatment periods (see **attached PDF**). Specifically, _Figure R2_ reports results analogous to _Figure 4_, but for ten pre-treatment periods. The results are qualitatively similar.\n\n2. For the **comparison to regularized models**, we report below the performance of LASSO solutions on all covariates in recovering the ATE on the evaluation sample. The relative performance depends on the size of the evaluation sample and the regularization parameter. In this table, every row corresponds to a different sample size of subset on which the ATE is evaluated (see our response above), and each column corresponds to a LASSO penalty parameter or the interpolating solution with all covariates included. \n\n   |$m$   | $\\alpha=0.01$ | $\\alpha=0.005$ | $\\alpha=0.001$ | $\\alpha=0.0005$ | Interpolating |\n   |----|---------------|----------------|----------------|-----------------|--------------|\n   |1   | 6.403         | 6.564          | 8.149          | 8.765           | 8.168        |\n   |5   | 3.009         | 3.005          | 3.666          | 3.956           | 3.692        |\n   |10  | 2.293         | 2.226          | 2.588          | 2.752           | 2.512        |\n   |20  | 1.779         | 1.604          | 1.773          | 1.890           | 1.757        |\n   |50  | 1.540         | 1.238          | 1.112          | 1.098           | 1.154        |\n   |100 | 1.319         | 0.985          | 0.730          | 0.699           | 0.771        |\n\n   - The LASSO uses explicit regularization, while our main specification only regularizes among perfectly fitting solutions (for which it chooses the norm-minimal model). The latter approach has the advantage that it does not require us to choose a regularization parameter, but we do not claim that it is optimal. The explicit regularization approach of the LASSO has the advantage that it may further improve performance. Indeed, the finding that very complex, interpolating solutions can perform well and that adding complexity (in the form of additional covariates or donor units) can improve out-of-sample performance still leaves room for improvement by additional, explicit regularization.\n   - The LASSO uses the Taxicab (L1) norm for regularization, while our approach uses the Euclidean (L2) norm. In principle, we could also use the L1 norm for choosing among interpolating solutions, but in that case we may obtain non-unique solutions (at least in some edge cases) since we lose strict convexity. Furthermore, we know from theory that LASSO regression does well in cases with approximately sparse parameters, but may do poorly if most regressors are relevant for prediction.\n\n3. We agree that **inference** on treatment effects when nuisance components are estimated by very high-dimensional and interpolating models is an important open question. To the degree that inference results for e.g. double machine learning rely on predictive fit or variance properties, our analysis suggests that these results may be feasible even with interpolating solutions, since good out-of-sample performance and low prediction variance do not rely on low dimensionality. Establishing sufficient conditions for valid inference on causal parameters in the presence of interpolating estimation of nuisance components could be a promising direction for future research.\n\n4. **Super learners** start with a set of candidate learners. The super learner then combines these candidate learners, for example by picking one out of the set, or by picking some linear combination, or a convex combination. Our method is in the spirit of the third flavor of the super learner, if one views the original regressors all as candidate learners. Picking a convex combination as in the super learner then has the averaging property that we exploit. We appreciate you pointing out the connection."}, {"review_id": "WcVgHdmAWP", "reviewer": "Reviewer_wCp4", "summary": "The paper studies the issue of double and single descent for linear regression as well as synthetic control. For the former, empirical motivation is given in the context of predicting wages and theoretical results are given from the perspective of model averaging. For the latter, empirical motivation is given in the context of imputing counterfactual California smoking rates and again theoretical results are given from the model averaging perspective. ", "questions": "- Tsigler and Bartlett (2023, JMLR) gives general sufficient conditions under which the optimal regularization parameter is negative. In view of this, I am wondering what would happen if $\\eta$ in equation (2) is negative. In other words, would it be possible to study the penalized synthetic control estimator with a negative regularization parameter?\n- Model averaging is popular in both statistics and econometrics: e.g., Claeskens and Hjort (2008), Model selection and model averaging, Cambridge University Press; Hansen (2007), Least Squares Model Averaging. Econometrica. Some discussion of related papers on model averaging would be helpful. ", "limitations": "- Lines 221-223: it is stated that \"We believe that these variance and geometric properties of linear regression are well understood in the literature and likely not new, although we are not aware of an explicit statement of the model-averaging connection between more and less complex interpolating linear-regression models.\" The statement is a bit unclear. It would be better if the literature review is more thoroughly done in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- To my reading, the most impressive result is Proposition 4, which shows that synthetic control has the model-averaging property. I presume that this is new in the literature but it is not explicitly mentioned in the paper. It would be good to clarify the novelty of Proposition 4 and emphasize its implications.\n- The two numerical examples are illustrative and provide very good empirical motivation.\n  ", "weaknesses": "- To my reading of the literature, one of the most under-explored but central issues in double descent or benign overfitting is the analysis of bias. For example, Tsigler and Bartlett (2023, JMLR) entitled \"Benign overfitting in ridge regression\" provides sharp bounds for the bias term. The current paper does not provide any result regarding the bias term, which will not be zero with over-parametrization.\n- It seems that it is not totally new to combine model averaging with double descent in the literature. For example, see the following quote from Wilson and Izmailov (2020, NeurIPS, page 8) entitled \"Bayesian deep learning and a probabilistic perspective of generalization\": _\"Double descent [e.g., 3] describes generalization error that decreases, increases, and then again decreases, with increases in model flexibility. ... However, our perspective of generalization suggests that performance should monotonically improve as we increase model flexibility when we use Bayesian model averaging with a reasonable prior.\"_ It would be helpful to provide a more thorough discussion of the literature.\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and suggestions.\n\n1. For the point about **bias**, we agree that our analysis does not separately provide results for the bias of the estimator, which will generally depend on the relationship of the estimator to the data-generating process. We note, however, that our results on mean-squared error implicitly captures both the variance and bias components of the loss. For example, assumption (P) in _Proposition 5_ also constrains the bias of the estimator, although only implicitly. We agree that a study of bias is a relevant extension to our work, especially for the case of synthetic control.\n\n2. For **negative regularization**, our results do not directly extend to this case, but this may be an interesting case for further analysis. For the specific case of synthetic control, the optimization problem with a negative cost of complexity is not generally convex any more, solutions are not guaranteed to be unique, and the model-averaging property does not have to hold (although we suspect that this is due to non-uniqueness if ties are broken inconsistently).\n\n3. On the **relationship to the model-averaging literature**, we agree that it would be worthwhile to add additional references and connections. However, many approaches to model averaging rely on using outcome data to decide on weights to assign to different models, and effectively choose the weights that maximize prediction fit in the training data. This stands in contrast to some of the weights we consider. For example, in the case of interpolating linear regression, weights do not depend on the outcomes, since all component models fit the data perfectly, and we do not use their empirical performance to combine simpler models into a more complex one.\n\n4. Thank you for pointing out the **relationship to prior work on Bayesian model averaging**, which we will reference. In this context, our approach could be seen as the uninformative limit of a Bayesian regularization approach.\n"}, {"review_id": "MR10yjOSpn", "reviewer": "Reviewer_aRQr", "summary": "This paper investigates over-parametrized models in causal inference, specifically focusing on high-dimensional linear regression models and high-dimensional synthetic control estimators with a large number of control units. The authors examine the prediction risk behaviors associated with these two estimators, and present a unified theoretical perspective on the descent phenomena in the interpolating (=high-dimensional) regime, which they refer to as the “model-averaging property” as demonstrated in Proposition 1 and Proposition 4; also see Eq. (MA). By introducing the high-level assumption of the model-averaging property (MA), which posits that the full model can be expressed as a convex combination of simpler “leave-one-out” counterparts, and another technical assumption (P) concerning the optimality of the aggregated model against a random permutation of the weights, the authors assert that the complex model cannot perform worse than the average performance of the simpler models (Proposition 5). \n\nIn summary, this paper provides a concise exposition of the “benign overfitting” phenomena observed in two estimators widely utilized in causal inference. The authors rely on two high-level geometric assumptions, the model-averaging property and Assumption (P), which provide insights into the descent behavior of these estimators and warrant further investigations in follow-up studies.", "questions": "1.  I would like to request the authors to address the three concerns raised in the “Weaknesses” section.\n\n2.  Miscellaneous suggestions:\n  (a) It would be clearer if the authors replace the term \"full rank\" with \"full row rank\" for clarity, e.g., in lines 127, 171, 174, 191, although it is obvious in the interpolating regime. \n  (b) In lines 153-155, the authors state, \"In this case, loss continues to decrease throughout, ultimately reaching a minimum that is below the lowest error achieved left of the interpolation threshold.\" However, this is not easily discernible in Figure 1-(b).", "limitations": "The paper suggests several potential avenues for future research, recognizing some limitations of their findings, such as exploring more fundamental conditions beyond (P). However, it could benefit from a more explicit discussion on the limitations and potential negative societal impact of the proposed approaches. It would be valuable for the authors to provide a more detailed acknowledgment of the technical limitations of their methods and offer insights into the potential adverse consequences that may arise when applying these approaches in real-world settings, specifically in the context of synthetic control methods. Nonetheless, it is important to note that given the paper's primary focus on theoretical aspects, a comprehensive and extensive examination of these limitations may not be deemed critical.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper offers a simple yet intriguing perspective on understanding the double descent phenomenon in the interpolating regime by leveraging the mechanical properties of predictive estimators that are largely agnostic about the underlying data generating process. The authors adeptly combine knowledge from linear algebra and convex analysis to provide concise explanations for the returns of complex models in high-dimensional causal estimators. The practical implications of their findings, particularly in the context of synthetic control with many control units, are noteworthy. The paper alleviates concerns about overfitting when using a large number of control units, eliminating the need for pre-selecting an appropriate subset, which could be challenging in practice.\n\nFurthermore, the authors' focused approach on linear regression models and synthetic control estimators enhances the clarity and concreteness of their arguments. They effectively develop their findings in these two settings first, and thereafter, hinting at the potential extension of the abstract results based on the model-averaging property (MA) and the permutation property (P) to broader settings beyond the two examined scenarios.\n\nWhile it is yet challenging to assess the significance of this work within the broader NeurIPS readership, I believe the authors make a meaningful contribution to the sub-community of econometrics/causal inference. The paper offers a fresh and clear perspective on understanding the phenomenon of benign overfitting, delivering an important message that reassures the practical utilization of high-dimensional synthetic control, specifically.", "weaknesses": "Although this paper has several strengths, there are three concerns/suggestions that could be addressed to further improve its quality.\n\nFirstly, there is a concern about the possibly limited scope of applicability for the presented results. Although the paper establishes theoretical support for the concept of \"benign overfitting\" in synthetic control estimators and acknowledges the possibility of extending the analysis to a broader class of estimators through the abstract conditions (MA) and (P), it remains unclear how far-reaching this perspective can be and to what extent it can be extended. It would be valuable if the authors could provide a discussion on the expected scope of extensions, limitations, and potential challenges that may arise.\n\nAdditionally, in order to strengthen the positioning of the paper, it would be beneficial to provide additional comparisons and contrasts to previous approaches. Given the focus on synthetic control estimators, the authors could comment on the limitations of applying existing results on double descent and benign overfitting in over-parametrized models to the specific context of synthetic control estimators. This would effectively highlight the authors' contributions in this work and emphasize the distinctive features of the techniques employed.\n\nLastly, while the paper primarily presents a novel theoretical perspective, it would be valuable to complement the theory with a more extensive set of numerical experiments. For instance, conducting an ablation study with synthetic datasets to verify the assumptions and theory, as well as performing experiments with real-world datasets at scale to confirm the descent behavior of the synthetic control method, could strengthen the paper's contributions. By incorporating such experiments, the authors can provide empirical evidence to support their theoretical findings and enhance the practical relevance of the paper.\n\nBy addressing these concerns, the authors would be able to further strengthen their contributions in this paper, clarify the scope of their results, highlight its distinctive features, and provide a more comprehensive explanation for the implications of their findings.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions.\n\n1. For the **concerns** you lay out:\n    - We believe that **important next steps on this agenda** include (i) developing low-level sufficient conditions under which more complex synthetic control performs better than less complex synthetic control, (ii) providing tools for valid inference following interpolating linear regression and high-dimensional synthetic control, and (iii) understanding implications for other causal-inference tools that rely on bias–variance trade-offs. Our current analysis is limited since assumptions are on a very high level and the analysis is limited to two specific methods. We believe that deriving inference results in particular may be methodologically challenging, but hope that our analysis can motivate further research in this area.\n    - We now provide **additional comparisons** and results, following the suggestions from other reviewers (see **attached PDF** and our response to reviewer xBLi). In particular, we also consider low-dimensional linear regression, a comparison to the LASSO, and additional evidence for synthetic control with more pre-treatment periods. For our contribution, we believe that the results on synthetic control are new and not previously available in the literature.\n    - Some additional **numerical experiments** are provided in the **attached PDF** and in the response to reviewer xBLi.\n\n2. Thank you for your **suggestions**, which we will address in our manuscript.\n    - We will replace “full rank” by “full row rank”.\n    - Thank you for your suggestion about Figure 1. We agree that the error in the right tail is comparable to that on the left for the CPS (Figure 1(b)), while it shows a clear improvement for the NSW controls (Figure 1(a)), which we will clarify in the text. In addition, in the graphs in the **attached PDF** we added a zoomed-in panel to facilitate the comparison of performances."}]

Author rebuttal:
We are attaching a one-page PDF with two additional figures: _Figure R1_ presents the performance of linear regression in estimating average treatment effects in samples of varying size, and _Figure R2_ presents the performance of high-dimensional synthetic control with ten pre-treatment periods. Both figures are referenced in our responses to the reviewers, and described in more detail in our response to reviewer xBLi.

Meta-review: The paper builds on recent findings from the ML overparameterization community to give new and important insight into several causal inference methods, including synthetic controls. The paper also includes a sound empirical evaluation, though reviewers had several suggestions as to how to improve those. While there are limitations to the technical novelty (as discussed extensively by the reviewers), it is ultimately more important to open up these connections between two disparate fields, rather than give the best possible theoretical result in a conference paper. The results in the paper are sufficiently interesting on their own, and well-grounded, and I believe this work will generate a good deal of follow up in the community. One missing related paper which should be cited and discussed in the final version is [1].

[1] Kato, Masahiro, and Masaaki Imaizumi. "Benign-Overfitting in Conditional Average Treatment Effect Prediction with Linear Regression." arXiv preprint arXiv:2202.05245 (2022).
Decision: Accept (poster)