SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jann Spiess Guido Imbens Amar Venugopal
Stanford, CA, USA
jspiess,imbens,amarvenu@stanford.edu
Motivated by a recent literature on the double-descent phenomenon in machine learning, we consider highly over-parameterized models in causal inference, including synthetic control with many control units. In such models, there may be so many free parameters that the model fits the training data perfectly. We first investigate high-dimensional linear regression for imputing wage data and estimating average treatment effects, where we find that models with many more covariates than sample size can outperform simple ones. We then document the performance of high-dimensional synthetic control estimators with many control units. We find that adding control units can help improve imputation performance even beyond the point where the pre-treatment fit is perfect. We provide a unified theoretical perspective on the performance of these high-dimensional models. Specifically, we show that more complex models can be interpreted as model-averaging estimators over simpler ones, which we link to an improvement in average performance. This perspective yields concrete insights into the use of synthetic control when control units are many relative to the number of pre-treatment periods.
Motivated by a recent literature on the double-decent phenomenon in machine learning, we investigate the properties of common econometric estimators when we increase their complexity. For high-dimensional linear regression and for synthetic control with many control units, we document empirical applications where extremely over-parameterized models impute missing out-of-sample and out-of-time outcomes well. We then provide a common explanation for the returns to complexity in high-dimensional linear regression and synthetic control in terms of a simple model-averaging property, which we link to an improvement in imputation performance.
We often conceptualize the effect of complexity on econometric models in terms of a bias-variance trade-off: Adding complexity makes models more expressive and reduces bias, while increased overfitting leads to additional variance. In this view, an overly complex model fits overly well in the training sample, but fails to recover true parameters or generalize to new data. For example, linear regression with too many variables or synthetic control with many control units may perform poorly because of overfitting and excess variance. Consistency results for high-dimensional econometric models therefore usually assume that the expressiveness of models is limited relative to the available sample size, and practical advice often highlights choosing simple models or regularizing complex ones, in a way that balances bias and variance optimally to achieve good estimation and prediction performance.
Replication code is available at github.com/amarvenu/causal-descent.
A recent literature in statistics and machine learning has highlighted the surprisingly good prediction performance of extremely high-dimensional models that fit the data perfectly. That literature has documented a so-called double-descent phenomenon for deep neural networks and other high-dimensional regression models: increasing complexity beyond the interpolation threshold at which the training sample error is zero can lead to a gradual reduction in variance and improvement in out-of-sample performance. In such cases, there are often two complexity regimes: below the interpolation threshold, the usual bias-variance trade-off leads to a decrease (first descent) and then an increase in out-of-sample loss, while beyond the interpolation threshold, out-of-sample loss decreases again (second descent).
In order to investigate the properties of highly over-parameterized models in causal inference, we first demonstrate a double-descent curve for linear regression in imputing wage outcomes and estimating average-treatment effects. In the LaLonde (1986) sample of the Current Population Survey (CPS) drawn from Dehejia and Wahba (1999, 2002), we generate over 8,000 variables from binning and interacting the original eight demographic and employment-related variables. We then fit a linear regression model on 3,000 training units and an increasing, randomly chosen subset of these variables, choosing the norm-minimizing solution once the model fits perfectly. Evaluated on the LaLonde (1986) control sample from the National Supported Work Demonstration (NSW) experiment, we observe the usual bias-variance trade-off for a low and moderate number of included covariates, where performance first increases slightly, before deteriorating substantially when approaching the interpolation threshold. Beyond the interpolation threshold, however, out-of-sample performance increases again. Ultimately, an extremely over-parameterized model on over 8,000 variables even outperforms less complex regressions with a randomly chosen set of covariates and achieves performance comparable to a linear regression on the original, unmanipulated set of covariates.
Having demonstrated the returns to complexity in high-dimensional linear regression, we document the performance of synthetic control estimators with many control units. In the California smoking data (Abadie et al., 2010), we impute missing smoking rates based on a small number of pre-treatment periods and an increasingly large number of control states. As in the case of linear regression, we see returns to increasing model complexity, even beyond the point where some of the synthetic-control models fit the training data perfectly. However, for synthetic control we do not observe an initial trade-off between bias and variance: in our empirical example, the performance on future periods is always better for the more complex models, with no intermittent increase in errors. Unlike the double-descent relationship for linear regression, the performance of synthetic control in our application yields a single-descent curve that only improves with complexity, no matter whether there are a few or many control states.
We then connect the returns to complexity in high-dimensional linear regression and in synthetic control in terms of a simple model-averaging property. While both estimators are constructed differently and represent different regressions, they both share a common feature: More complex models can be represented as convex averages over simpler models. We show that this model-averaging property applies to linear regression in the interpolation regime, as well as to synthetic control in general. The property holds purely mechanically and does not depend on the training or target distributions. It relates to other model-agnostic properties of linear regression, for which we also show a reduction in (conditional) variance for minimal-norm least-squares estimators beyond the interpolation threshold.
Having established a model-averaging property for interpolation linear regression and for synthetic control, we provide high-level assumptions under which this property translates to better imputation performance. A direct consequence of model averaging is that the (convex) prediction loss of a more complex model cannot be worse than the corresponding average prediction loss of simpler models, when the same weights are used to average. When the complex model on average also outperforms comparable models of the same complexity, then we show that the model-averaging property translates into a reduction in average out-of-sample error relative to a randomly selected simpler model. These results only put high-level, largely model-agnostic assumptions on the data-generating process, and are driven by mechanical properties of the underlying estimators.
Our results have practical implications for the use of synthetic control with many control units. Conventional wisdom may indicate that synthetic control with a very large number of donor units relative to the number of pre-treatment periods is problematic, and that selecting among many, ex-ante
exchangeable units in this case may represent a challenge. However, our results imply that this is not the case: as long as ties are broken by a suitable regularization procedure (such as the minimum-norm solution in our case), making an ex-ante choice among many control units is not necessary. That said, if ex-ante information is available about which units are particularly suitable as controls, then using this information can still be helpful and improve imputation.
We build upon results on over-parameterized regression in machine learning and statistics, where double-descent curves for kernel and linear regression and the performance of norm-minimizing interpolating solutions have been studied extensively (including Liang and Rakhlin, 2020; Liang et al., 2020; Liang and Recht, 2023; Bartlett et al., 2020; Hastie et al., 2022) as part of a broader literature on double descent and interpolation in deep learning (Zhang et al., 2016; Belkin et al., 2018; Belkin, 2021; Mei and Montanari, 2022). Kelly et al. (2022) documents the benefits of overparameterized models in asset return prediction both in theory and empirically. Kato and Imaizumi (2022) provides results on the estimation of conditional average causal effects using over-parametrized linear regression. Relative to this work on interpolating regression, we show connections to synthetic control based on simple mechanical properties that are largely agnostic about the true data-generating process. Thereby, we also relate to work on high-dimensional and regularized synthetic control (Doudchenko and Imbens, 2016; Abadie and L'Hour, 2021; Ben-Michael et al., 2021) and the connections between synthetic control and linear regression (Athey et al., 2021; Agarwal et al., 2021; Shen et al., 2022; Bruns-Smith et al., 2023). Finally, we connect to a literature on model averaging in econometrics and statistics (e.g. Hansen, 2007; Claeskens and Hjort, 2008). More specifically, Wilson and Izmailov (2020) consider Bayesian model averaging in deep learning, and discuss relationships to double descent. While our linear-regression solutions are closely related to available results on norm-minimizing regression, we are not aware that the results on synthetic control were noted previously.
The remaining note is structured as follows. Section 2 provides an empirical example of double descent for linear regression and discusses some properties of the norm-minimizing linear-regression estimator in the interpolating case. Section 3 discusses the relationship of complexity and imputation performance of synthetic control in an empirical example, and makes connections to the properties of interpolating linear regression. Section 4 discusses high-level consequences of the model-averaging property of interpolating linear regression and synthetic control. Section 5 concludes by discussing some limitations and open questions.
2 Double Descent for Linear Regression
In this section, we consider imputation by high-dimensional linear regression as an illustration of the performance of highly over-parameterized models. We start with an empirical illustration of wage imputation in CPS data with a varying number of randomly ordered covariates, which we evaluate in terms of its ability to estimate an average treatment effect. We then discuss theoretical properties of the interpolating linear-regression estimator, followed by a graphical illustration. These results are closely related to prior work on interpolating linear and kernel regression (including Bartlett et al., 2020; Liang et al., 2020; Hastie et al., 2022; Kato and Imaizumi, 2022).
2.1 Setup and Estimator
We consider a linear-regression estimator in data  $(y_{i},x_{i})_{i = 1}^{n}\in \mathbb{R}\times \mathbb{R}^{k}$  from  $n$  observations of a scalar outcome and  $k$  scalar covariates. We write  $X = (x_i')_{i = 1}^n\in \mathbb{R}^{n\times k}$  and  $Y = (y_{i})_{i = 1}^{n}\in \mathbb{R}^{n}$ . For a subset  $J\subseteq \{1,\ldots ,k\}$  of the covariates, we denote by  $\mathcal{B}^J = \arg \min_{\beta \in \mathbb{R}^k;\beta_j = 0\forall j\notin J}\sum_{i = 1}^n (y_i - x_i'\beta)^2$  the set of all least-squares linear-regression estimates on  $J$ . Among these, we choose the norm-minimizing estimate  $\hat{\beta}^J = \arg \min_{\beta \in \mathcal{B}^J}\| \beta \|$ . Throughout, we assume that  $X_J = (x_{ij})_{i\in \{1,\dots,n\},j\in J}\in \mathbb{R}^{n\times |J|}$  is of full row rank. Hence, there are multiple least-squares solutions,  $|\mathcal{B}^J | > 1$ , if and only if  $|J| > n$ . In that case,  $Y = X\hat{\beta}^{J}$  and  $\hat{\beta}^J$  is the minimal-norm interpolating solution (cf. Liang et al., 2020).
For example, Bruns-Smith et al. (2023) shows that synthetic-control-type balancing estimators with nonnegative weights can be related to outcome regressions, which connects our OLS setup in Section 2 to the synthetic-control setup in Section 3.
2.2 Empirical Motivation
We impute wages in the non-experimental LaLonde (1986) sample of the Current Population Survey (CPS) using linear regression on a large number of covariates, based on data drawn from Dehejia and Wahba (1999, 2002). From the original eight covariates, we obtain  $k = 8408$  explanatory variables by binning and interacting the available features. We train minimal-norm least-squares regression models on a training sample of  $n = 3000$  randomly chosen observations, and evaluate their mean-squared error in imputing average wages for subsets (of various sizes) of the 260 LaLonde (1986) National Supported Work Demonstration (NSW) experimental controls as provided through Dehejia and Wahba (1999, 2002). To these covariates we add a small amount of iid Gaussian noise to avoid rank deficiency when estimating linear regression. When fitting wage imputation models, we vary the set of covariates included in the regression. Specifically, we order all features randomly. For a complexity  $\ell \in \{1, \dots, k\}$ , we then choose the first  $\ell$  covariates, and obtain the estimate  $\hat{\beta}^J$  for  $J = \{1, \dots, \ell\}$ . Our presented results reflect an average over five such random orderings of covariates.
In order to assess the viability of this approach for applications to average treatment effect (ATE) estimation, we assess each model by its ability to accurately predict the average outcome on a new dataset. In particular, we consider various subsets of size  $m$  of the NSW experimental control set. For each  $m$ , we draw 1000 samples of size  $m$  without replacement from the NSW experimental control dataset. For each sample, we average the  $m$  observation-level outcome predictions and compare the result to the true mean outcome of the given sample. We then take the RMSE across the 1000 draws of size  $m$  to obtain our evaluation metric. $^2$
In Figure 1a, we consider CPS data and plot the average root-mean-squared error (RMSE) for pointwise prediction of the wages, reporting the in-sample performance as well. The vertical dashed line denotes the interpolation threshold where  $\ell = n$ . Figure 1b shows the ATE RMSE metric, once again averaged over five random orderings of the covariates, for various subset sizes. The horizontal dashed lines show the RMSE of a simple linear regression on the original, unmanipulated (low-dimensional) features, colored according to corresponding sample size  $m$ . A zoomed-in version of Figure 1b focusing on the highly overparametrized regime can be found in Figure 7.
(a) Average out-of-sample (blue) and in-sample (orange) pointwise RMSE for CPS data.
(b) Average counterfactual prediction RMSE for varying subset sizes on NSW controls.
Figure 1: Average RMSE for linear regression for a varying number of covariates.
The out-of-sample losses in these illustrations show a descent in loss right of the interpolation threshold (denoted by the vertical dashed line), at which point in-sample loss is zero. For NSW experimental controls, out-of-sample error initially decreases (first descent), while for CPS non-experimental controls it remains initially flat. For both out-of-sample datasets, loss then goes on to increase and peaks sharply at  $\ell = n$ , at which point the linear models start to fit perfectly. As complexity increases further, loss decreases again (second descent). In both cases, loss continues to decrease throughout. For NSW experimental controls, loss ultimately reaches a minimum that is below the lowest error achieved left of the interpolation threshold, while the loss achieved in the CPS case is similar between the right tails and the minimum on the left. Furthermore, as the sample size  $m$  increases in Figure 1b, the highly complex interpolating models outperform a simple model based
For  $m = 1$ , taking 1000 draws would necessarily yield duplication; in this case, we instead simply consider the full set of 260 NSW experimental control observations. Note that results for  $m = 1$  therefore correspond to a standard (observation-level) RMSE calculation, analogous to the out-of-sample curve in Figure 1a.
on the original set of provided features; the  $m = 50$  and  $m = 100$  curves have minima below their corresponding dashed lines.
In this setting, highly over-parameterized linear models constructed via discretizing and interacting available features exhibit performance improvements over a linear model fitted on the original features. This result appears remarkable, since the heavily over-parameterized models do not include the original non-binary covariates, but only indicators obtained from quantile binning (see Section B.1 for details). Further improvement could be achieved by always including the original covariates in the model.
We note that our illustration is extreme in that it artificially creates a large number of low-signal covariates, for which the best model only slightly outperforms a small, hand-curated model based on the original covariates. Nevertheless, this simple empirical exercise demonstrates the non-monotonicity in the relationship of complexity to variance and loss that has been studied by the literature on interpolating regression and double descent, and extends it to causal target parameters like the average treatment effect.
2.3 Theoretical Properties and Geometric Illustration
Motivated by the empirical illustration, we note some theoretical properties that are direct consequences of the structure of the norm-minimizing linear least-squares estimator, and will later serve as a comparison point for synthetic control. We note that formal results on the bias-variance properties in terms of more primitive properties of the data-generating process are available, including in Bartlett et al. (2020); Liang et al. (2020); Hastie et al. (2022). Here, we focus on illustrating properties that follow mechanically from the construction of the estimator. Our focus is on comparing more complex models, with covariates  $J$ , to slightly simpler (more sparse) ones, with covariates  $J \setminus \{j\}$ , where the  $j$ -th covariate is dropped. Throughout, we assume that the covariate matrices are of full row rank for the more and less complex models
Assumption 1 (Full rank). The covariate matrix  $X_{J} \in \mathbb{R}^{n \times |J|}$  with columns  $J$  as well as the covariate matrices  $X_{J \setminus \{j\}} \in \mathbb{R}^{n \times (|J| - 1)}$  for all  $j \in J$  are of full row rank.
We first consider the geometry of interpolating solutions, for which we note that more complex model can be expressed by an average over simpler models.
Proposition 1 (Model averaging for interpolating linear least-squares regression). For every  $X$  and  $J$  with  $|J| > n$  such that Assumption 1 holds there exists
$$
\lambda \in [ 0, 1 ] ^ {J}, \sum_ {j \in J} \lambda_ {j} = 1
$$
$$
\hat {\beta} ^ {J} = \sum_ {j \in J} \lambda_ {j} \hat {\beta} ^ {J \backslash \{j \}}.
$$
We can choose the weights in Proposition 1 as a function of the covariate matrix  $X_{J}$  only. Specifically, weights can be chosen as  $\lambda_{j} = \frac{1 - X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j}}{|J| - n}$ , where  $X_{j}$  is the  $j$ -th column of  $X$  and  $X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j}$  can be seen as the "leverage" of feature  $j \in J$ , analogously to the leverage of an observation in the usual (low-dimensional) linear regression case. $^3$
(a) Non-interpolating case with  $n = 2$
(b) Interpolating case with  $n = 1$
Figure 2: Minimal-norm least-squares solutions for linear regression with  $J = \{1,2\}$ , where  $n$  varies.
<sup>3</sup>We thank Tengyuan Liang for pointing out this connection to us, as well as for suggesting a direct proof via the Sherman-Morrison-Woodbury formula.
The model averaging property of interpolating linear regression is illustrated in the right panel of Figure 2 for the simple case of two covariates  $(J = \{1,2\})$  and a single data point  $(n = 1)$ . Here, the models  $\beta \in \mathbb{R}^J$  that for the model perfectly lie on the line through  $\hat{\beta}^{\{1\}}$  and  $\hat{\beta}^{\{2\}}$ , with the norm-minimizing solution  $\hat{\beta}^{\{1,2\}}$  lies between the two. In contrast, if  $\hat{\beta}^{\{1\}}$  and  $\hat{\beta}^{\{2\}}$  are not interpolating (such as in the case of the left panel of Figure 2, where  $n = 2$ ), then the more complex model  $\hat{\beta}^{\{1,2\}}$  does not generally lie in the convex hull of the simpler ones.
As an immediate consequence, any interpolating model can in this case be expressed as a convex average of simpler interpolating models,  $\hat{\beta}^J = \sum_{L\subseteq J;|L| = \ell}\lambda_j\hat{\beta}^L$  for all  $\ell \in \{n,\dots ,|J|\}$  (provided all  $X_{L}$  are of full row rank). A particularly interesting special case is  $\ell = n$ , for which we express  $\hat{\beta}^J$  as a model average of just-interpolating models  $\hat{\beta}^L$  with  $\hat{\beta}_L^L = X_L^{-1}Y$ .
In addition to the model-averaging property, the geometry of interpolating solutions also implies that the variance of the norm-minimizing linear least-squares estimator generically decreases in the interpolation regime  $|J| > n$ , for which the complex estimator  $\hat{\beta}^J$  along with the simpler estimator  $\hat{\beta}^J \setminus \{j\}$  both fit the training data perfectly.
Proposition 2 (Variance reduction for linear least-squares regression). Suppose Assumption 1 and that  $Y$  has a second moment. If  $|J| > n$  then a.s.  $\operatorname{tr} \operatorname{Var}(\hat{\beta}^J | X) \leq \min_{j \in J} \operatorname{tr} \operatorname{Var}(\hat{\beta}^{J \setminus \{j\}} | X)$ .
The reduction in variance is a direct consequence of a more general property of the least-squares solution. Specifically, the next proposition shows that if we redraw new outcome data in the interpolation regime, then the distance between more complex solutions is smaller than the distance between less complex models.
Proposition 3 (Variation hierarchy for linear least-squares regression). For fixed  $X$  and  $J$  for which Assumption 1 holds consider two draws  $Y_{A}$  and  $Y_{B}$  yielding minimal-norm least-squares estimates  $\hat{\beta}_{A}^{J}, \hat{\beta}_{A}^{J \setminus \{j\}}$  and  $\hat{\beta}_{B}^{J}, \hat{\beta}_{B}^{J \setminus \{j\}}$ , respectively.
1. If  $|J|\leq n$ , then  $\| \hat{\beta}_A^J -\hat{\beta}_B^J\|_{X'X}\geq \max_{j\in J}\| \hat{\beta}_A^{J\setminus \{j\}} - \hat{\beta}_B^{J\setminus \{j\}}\|_{X'X}$ .
2. If  $|J| > n$ , then  $\| \hat{\beta}_A^J - \hat{\beta}_B^J \| \leq \min_{j \in J} \| \hat{\beta}_A^{J \setminus \{j\}} - \hat{\beta}_B^{J \setminus \{j\}}$
Here, we write  $\| \beta \| _M = \sqrt{\beta'M\beta}$  for some positive semi-definite symmetric matrix  $M\in \mathbb{R}^{k\times k}$ .
In words, the variation of models across draws of the outcome data increases with complexity on the left of the interpolation threshold, while it decreases on the right. Here, the choice of norm is essential for these results to hold uniformly across simpler models.