SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Proposition 2. The result follows from Proposition 3 by noting that, for two independent draws  $Y_{A}$  and  $Y_{B}$  for fixed  $X$ , we have that
$$
\begin{array}{l} \operatorname {E} \left[ \| \hat {\beta} _ {A} ^ {J} - \hat {\beta} _ {B} ^ {J} \| ^ {2} | X \right] = \operatorname {E} \left[ \| \left(\hat {\beta} _ {A} ^ {J} - \operatorname {E} [ \hat {\beta} ^ {J} | X ]\right) - \left(\hat {\beta} _ {B} ^ {J} - \operatorname {E} [ \hat {\beta} ^ {J} | X ]\right) \| ^ {2} | X \right] \\ = \operatorname {E} [ \| \hat {\beta} _ {A} ^ {J} - \operatorname {E} [ \hat {\beta} ^ {J} | X ] \| ^ {2} | X ] + \operatorname {E} [ \| \hat {\beta} _ {B} ^ {J} - \operatorname {E} [ \hat {\beta} ^ {J} | X ] \| ^ {2} | X ] = 2 \operatorname {t r} \operatorname {V a r} (\hat {\beta} ^ {J} | X) \\ \end{array}
$$
(and the same for  $J \setminus \{j\}$ ), and thus
$$
\begin{array}{l} \operatorname {t r} \operatorname {V a r} (\hat {\beta} ^ {J} | X) = \frac {1}{2} \operatorname {E} [ \| \hat {\beta} _ {A} ^ {J} - \hat {\beta} _ {B} ^ {J} \| ^ {2} | X ] \leq \frac {1}{2} \operatorname {E} \left[ \min  _ {j \in J} \| \hat {\beta} _ {A} ^ {J \setminus \{j \}} - \hat {\beta} _ {B} ^ {J \setminus \{j \}} \| ^ {2} \right| X \Bigg ] \\ \leq \min _ {j} \frac {1}{2} \operatorname {E} [ \| \hat {\beta} _ {A} ^ {J \setminus \{j \}} - \hat {\beta} _ {B} ^ {J \setminus \{j \}} \| ^ {2} | X ] = \operatorname {t r} \operatorname {V a r} (\hat {\beta} ^ {J \setminus \{j \}} | X). \\ \end{array}
$$
Proof of Proposition 3. Consider first the case  $|J| \leq n$ . Under Assumption 1, we define the projection matrices
$$
\Pi^ {J} = X _ {J} (X _ {J} ^ {\prime} X _ {J}) ^ {- 1} X _ {J} ^ {\prime} \in \mathbb {R} ^ {n \times n}, \quad \Pi^ {J \setminus \{j \}} = X _ {J \setminus \{j \}} (X _ {J \setminus \{j \}} ^ {\prime} X _ {J \setminus \{j \}}) ^ {- 1} X _ {J \setminus \{j \}} ^ {\prime} \in \mathbb {R} ^ {n \times n}.
$$
Since  $\Pi^{J\setminus \{j\}} = \Pi^{J\setminus \{j\}}\Pi^{J}$ , we have that  $X\hat{\beta}^{J\setminus \{j\}} = \Pi^{J\setminus \{j\}}Y = \Pi^{J\setminus \{j\}}\Pi^{J}Y = \Pi^{J\setminus \{j\}}\hat{\beta}^{J}$ . Therefore,
$$
\begin{array}{l} \| \hat {\beta} _ {A} ^ {J} - \hat {\beta} _ {B} ^ {J} \| _ {X ^ {\prime} X} ^ {2} = \| X \hat {\beta} _ {A} ^ {J} - X \hat {\beta} _ {B} ^ {J} \| ^ {2} = \| \Pi^ {J \backslash \{j \}} (X \hat {\beta} _ {A} ^ {J} - X \hat {\beta} _ {B} ^ {J}) \| ^ {2} + \| (\mathbb {I} - \Pi^ {J \backslash \{j \}}) (X \hat {\beta} _ {A} ^ {J} - X \hat {\beta} _ {B} ^ {J}) \| ^ {2} \\ \geq \| \Pi^ {J \setminus \{j \}} (X \hat {\beta} _ {A} ^ {J} - X \hat {\beta} _ {B} ^ {J}) \| ^ {2} = \| X \hat {\beta} _ {A} ^ {J \setminus \{j \}} - X \hat {\beta} _ {B} ^ {J \setminus \{j \}} \| ^ {2} = \| \hat {\beta} _ {A} ^ {J \setminus \{j \}} - \hat {\beta} _ {B} ^ {J \setminus \{j \}} \| _ {X ^ {\prime} X} ^ {2}. \\ \end{array}
$$
Consider now the case  $|J| > n$ . Using the notation from the proof of Proposition 1, under Assumption 1 we have that  $X^{J}\hat{\beta}^{J} = X\hat{\beta}^{J} = Y = X\hat{\beta}^{J\setminus \{j\}} = X^{J}\hat{\beta}^{J\setminus \{j\}}$  and thus  $\Pi \hat{\beta}^{J} = \Pi \hat{\beta}^{J\setminus \{j\}}$  (as well as  $(\mathbb{I} - \Pi)\hat{\beta}^{J} = \mathbf{0}$ ) for the projection matrix  $\Pi = X^{J\prime}(X_JX_J)^{-1}X^J\in \mathbb{R}^{k\times k}$ . As a consequence,
$$
\begin{array}{l} \| \hat {\beta} _ {A} ^ {J} - \hat {\beta} _ {B} ^ {J} \| ^ {2} = \| \Pi (\hat {\beta} _ {A} ^ {J} - \hat {\beta} _ {B} ^ {J}) \| ^ {2} + \| (\mathbb {I} - \Pi) (\hat {\beta} _ {A} ^ {J} - \hat {\beta} _ {B} ^ {J}) \| ^ {2} = \| \Pi (\hat {\beta} _ {A} ^ {J \backslash \{j \}} - \hat {\beta} _ {B} ^ {J \backslash \{j \}}) \| ^ {2} \\ \leq \| \Pi (\hat {\beta} _ {A} ^ {J \backslash \{j \}} - \hat {\beta} _ {B} ^ {J \backslash \{j \}}) \| ^ {2} + \| (\mathbb {I} - \Pi) (\hat {\beta} _ {A} ^ {J \backslash \{j \}} - \hat {\beta} _ {B} ^ {J \backslash \{j \}}) \| ^ {2} = \| \hat {\beta} _ {A} ^ {J \backslash \{j \}} - \hat {\beta} _ {B} ^ {J \backslash \{j \}} \| ^ {2}. \\ \end{array}
$$
Proof of Proposition 4. Building upon the notation from Section 3.1, for  $J \subset \{1, \ldots, N\}$  write  $\mathcal{W}^J = \{w \in \mathcal{W}; w_j = 0$  for all  $j \notin J\}$  (where  $\mathcal{W} = \{w \in [0,1]^N; \sum_{i=1}^N w_i = 1\}$  is the  $N-1$ -simplex) and let  $\partial \mathcal{W}^J = \bigcup_{j \in J} \mathcal{W}^{J \setminus \{j\}} \subseteq \mathcal{W}^J$  be the boundary of  $\mathcal{W}^J$ . For outcomes, it will also be convenient to write  $X = (y_{it})_{t \in \{1, \ldots, T\}, i \in \{1, \ldots, N\}} \in \mathbb{R}^{T \times N}$  for the pre-treatment outcomes of the control units (with columns representing units), and  $y = (y_{0t})_{t \in \{1, \ldots, T\}} \in \mathbb{R}^T$  for the pre-treatment outcomes of the treated unit.
As the first step, we note that we can express the quality of synthetic control weights  $w \in \mathcal{W}^J$  as
$$
\| X w - y \| ^ {2} = \| X w - \bar {y} ^ {J} \| ^ {2} + \| \bar {y} ^ {J} - y \| ^ {2} \tag {5}
$$
in terms of the fitted values  $\overline{y}^J = X\overline{w}^J$  for the solution  $\overline{w}^J$  to a relaxed problem that drops the non-negativity constraint. That solution with weights in  $\mathcal{W}^{*} = \{w\in \mathbb{R}^{N};\sum_{i = 1}^{n}w_{i} = 1\}$  is defined, analogously to the synthetic-control solution in (1), as
$$
\overline{w}^{J} = \operatorname *{arg  min}_{w\in \overline{\mathcal{W}}^{J}}\| w\| \in \mathcal{W}^{*},\qquad \qquad \overline{\mathcal{W}}^{J} = \operatorname *{arg  min}_{w\in \mathcal{W}^{*};w_{j} = 0\forall j\notin J}\| Xw - y\| \subseteq \mathcal{W}^{*}.
$$
For this solution, we note that  $\| Xw - y\| ^2 = \| X(w - \overline{w}^J)\| ^2 +2(w - \overline{w}^J)'X'(X\overline{w}^J -y) + \| X\overline{w}^J -y\| ^2$ . Assume now that  $(w - \overline{w}^{J})'X'(X\overline{w}^{J} - y)\neq 0$ . Then there is some  $\varepsilon \neq 0$  such that  $\overline{w}^J (\varepsilon) = \overline{w}^J -(w - \overline{w}^J)\varepsilon \in \mathcal{W}^*$  with  $w_{j} = 0$  for  $j\notin J$  fulfills  $\| X\overline{w}^{J}(\varepsilon) - y\| < \| X\overline{w}^{J} - y\|$ , contradicting the choice of  $\overline{w}^J$ . Hence we must have that  $\| Xw - y\| ^2 = \| X(w - \overline{w}^J)\| ^2 +\| X\overline{w}^J -y\| ^2 = \| Xw - \overline{y}^J\| ^2 +\| \overline{y}^J -y\| ^2$ , which establishes (5).
As the second step, we note that we can therefore define the synthetic control solution in (1) in terms of the fitted values  $\overline{y}^J$  of the relaxed solution as
$$
\hat{w}^{J} = \operatorname *{arg  min}_{w\in \widehat{\mathcal{W}}^{J}}\| w\| \in \mathcal{W}^{J},\qquad \qquad \widehat{\mathcal{W}}^{J} = \operatorname *{arg  min}_{w\in \mathcal{W}^{J}}\| Xw - \overline{y}^{J}\| \subseteq \mathcal{W}^{J}.
$$
This follows immediately from (5) since  $\| \overline{y}^J - y \|$  is not affected by the choice of  $w \in \mathcal{W}^J$ . Similarly, for the constrained solutions with index set  $J \setminus \{j\}$  for  $j \in J$ , we have that
$$
\hat{w}^{J\setminus \{j\}} = \operatorname *{arg  min}_{w\in \widehat{\mathcal{W}}^{J}\setminus \{j\}}\| w\| \in \mathcal{W}^{J\setminus \{j\}},\qquad \widehat{\mathcal{W}}^{J\setminus \{j\}} = \operatorname *{arg  min}_{w\in \mathcal{W}^{J}\setminus \{j\}}\| Xw - \overline{y}^{J}\| \subseteq \mathcal{W}^{J\setminus \{j\}}
$$
since  $\mathcal{W}^{J\backslash \{j\}}\subseteq \mathcal{W}^J$  for all  $j\in J$
As the third (and central) step, we use Farkas' lemma to argue that there exist  $\lambda \in \mathbb{R}^J$  with  $\lambda_j \geq 0$  for all  $j \in J$  such that  $X\overline{w}^J = \sum_{j \in J} \lambda_j X\overline{w}^{J \setminus \{j\}}$ .
Assume first that  $X\hat{w}^J \neq \overline{y}^J$ . Then we must have that  $\hat{w}^J \in \partial \mathcal{W}^J$ . Indeed, if  $\hat{w}^J \in \mathcal{W}^J \setminus \partial \mathcal{W}^J$  then there exists some  $\varepsilon > 0$  such that  $\hat{w}^J(\varepsilon) = \hat{w}^J(1 - \varepsilon) + \overline{w}^J\varepsilon \in \mathcal{W}^J$ , for which  $\| X\hat{w}^J(\varepsilon) - \overline{y}^J\| = \| X(\hat{w}^J(\varepsilon) - \overline{w}^J)\| = (1 - \varepsilon)\| X(\hat{w}^J - \overline{w}^J)\| < \| X\hat{w}^J - \overline{y}^J\|$ , contradicting the choice of  $\widehat{W}^J$  and
$\hat{w}^J$ . Hence  $\hat{w}^J \in \partial \mathcal{W}^J$ , so there is some  $j$  with  $\hat{w}^J \in \mathcal{W}^{J \setminus \{j\}}$ , which implies that  $\hat{w}^{J \setminus \{j\}} = \hat{w}^J$  and  $X \hat{w}^J = X \hat{w}^{J \setminus \{j\}}$ . This means that we can choose  $\lambda$  as the indicator for component  $j$ .
Assume now that  $X\hat{w}^J = \overline{y}^J$ , and that there exists no such  $\lambda$ . Then, by Farkas' lemma, there exists  $v \in \mathbb{R}^T \setminus \{\mathbf{0}\}$  such that  $v'X\hat{w}^J < 0$  and  $v'X\hat{w}^{J \setminus \{j\}} \geq 0$  for all  $j \in J$ . Define the projection matrix  $\Pi = \frac{vv'}{v'v} \in \mathbb{R}^{T \times T}$ , and let  $\mathcal{W}^* = \arg \min_{w \in \mathcal{W}^J; \Pi X(w - \hat{w}^J) = X(w - \hat{w}^J)} v'Xw \subseteq \mathcal{W}^J$ . Then the minimum is attained at a boundary point  $w^* \in \mathcal{W}^* \cap \partial \mathcal{W}^J$  of the feasible set. Indeed, the feasible set is non-empty since it includes  $\hat{w}^J$ , and it is compact and convex. The minimum of the linear function is therefore attained at a boundary point, which is in  $\partial \mathcal{W}^J$ . As a consequence,  $w^* \in \mathcal{W}^{J \setminus \{j\}}$  for some  $j \in J$ . Since  $\hat{w}^J \in \mathcal{W}^J$ , we have that  $v'Xw^* \leq v'X\hat{w}^J < v'X\hat{w}^{J \setminus \{j\}}$ . Hence there is some  $\varepsilon \in (0,1]$  such that  $\hat{w}^{J \setminus \{j\}}(\varepsilon) = \hat{w}^{J \setminus \{j\}}(1 - \varepsilon) + w^*\varepsilon \in \mathcal{W}^{J \setminus \{j\}}$  fulfills  $v'X\hat{w}^{J \setminus \{j\}}(\varepsilon) = v'X\hat{w}^J$ . Since we therefore have  $\Pi X\hat{w}^{J \setminus \{j\}}(\varepsilon) = \Pi X\hat{w}^J$ , as well as  $(\Pi - \Pi)X\hat{w}^{J \setminus \{j\}}(\varepsilon) = (\Pi - \Pi)X(\hat{w}^{J \setminus \{j\}}(1 - \varepsilon) + \varepsilon\hat{w}^{J})$  since  $\Pi X(w^* - \hat{w}^J) = X(w^* - \hat{w}^J)$ , we have that
$$
\begin{array}{l} \| X \hat {w} ^ {J \backslash \{j \}} (\varepsilon) - \bar {y} ^ {J} \| ^ {2} = \| X (\hat {w} ^ {J \backslash \{j \}} (\varepsilon) - \hat {w} ^ {J}) \| ^ {2} \\ = \| \Pi X (\hat {w} ^ {J \setminus \{j \}} (\varepsilon) - \hat {w} ^ {J}) \| ^ {2} + \| (\mathbb {I} - \Pi) X (\hat {w} ^ {J \setminus \{j \}} (\varepsilon) - \hat {w} ^ {J}) \| ^ {2} = 0 + (1 - \varepsilon) ^ {2} \| (\mathbb {I} - \Pi) X (\hat {w} ^ {J \setminus \{j \}} - \hat {w} ^ {J}) \| ^ {2} \\ <   \| \Pi X (\hat {w} ^ {J \backslash \{j \}} - \hat {w} ^ {J}) \| ^ {2} + \| (\mathbb {I} - \Pi) X (\hat {w} ^ {J \backslash \{j \}} - \hat {w} ^ {J}) \| ^ {2} = \| X (\hat {w} ^ {J \backslash \{j \}} - \hat {w} ^ {J}) \| ^ {2} = \| X \hat {w} ^ {J \backslash \{j \}} - \bar {y} ^ {J} \| ^ {2}, \\ \end{array}
$$
contradicting the choice of  $\hat{w}^{J\backslash \{j\}}$  . Hence, such  $\lambda$  must exist.
As the fourth step, we expand the previous result on fitted values to the weights themselves in the case of penalized synthetic control, and show that the weights sum to one in that case. To this end, note that we can write the penalized synthetic control estimator from (2) as  $\hat{w}_{\eta}^{J} = \arg \min_{w\in \mathcal{W}^{J}}\| Xw - y\|^{2} + \eta \| w\|^{2}$ . Write now  $\tilde{X}_{\eta}^{J} = (X_{J}^{\prime}X_{J} + \eta \mathbb{I})^{1 / 2}\in \mathbb{R}^{J\times J}$  for the symmetric positive-definite matrix square root of the symmetric positive-definite  $X_J^\prime X_J + \eta \mathbb{I}$ , where  $X_{J}$  is a matrix of the columns of  $X$  with index in  $J$ , and  $\tilde{y}_{\eta}^{J} = (\tilde{X}_{\eta}^{J})^{-1}X_{J}^{\prime}y\in \mathbb{R}^{J}$ . For  $w_{J}$  the entries of  $w\in \mathcal{W}^J$  corresponding to the index set  $J$ , we find
$$
\begin{array}{l} \| X w - y \| ^ {2} + \eta \| w \| ^ {2} = \| X _ {J} w _ {J} - y \| ^ {2} + \eta \| w _ {J} \| ^ {2} \\ = w _ {J} ^ {\prime} X _ {J} ^ {\prime} X _ {J} w _ {J} - 2 w _ {J} ^ {\prime} X _ {J} ^ {\prime} y + y ^ {\prime} y + \eta w _ {J} ^ {\prime} w _ {J} = w _ {J} ^ {\prime} \left(X _ {J} ^ {\prime} X _ {J} + \eta \mathbb {I}\right) w _ {J} - 2 w _ {J} ^ {\prime} X _ {J} ^ {\prime} y + y ^ {\prime} y \\ = w _ {J} ^ {\prime} \tilde {X} _ {\eta} ^ {J \prime} \tilde {X} _ {\eta} ^ {J} w _ {J} - 2 w _ {J} ^ {\prime} \tilde {X} _ {\eta} ^ {J \prime} \left(\left(\tilde {X} _ {\eta} ^ {J}\right) ^ {- 1} X _ {J} ^ {\prime} y\right) + y ^ {\prime} y = \| \tilde {X} _ {\eta} ^ {J} w _ {J} - \tilde {y} _ {\eta} ^ {J} \| ^ {2} - \| \tilde {y} _ {\eta} ^ {J} \| ^ {2} + \| y \| ^ {2}. \\ \end{array}
$$
Hence, we can write (noting that  $\mathcal{W}^{J\setminus \{j\}}\subseteq \mathcal{W}^J$
$$
\hat {w} _ {\eta} ^ {J} = \underset {w \in \mathcal {W} ^ {J}} {\arg \min } \| \tilde {X} _ {\eta} ^ {J} w _ {J} - \tilde {y} _ {\eta} ^ {J} \|, \qquad \qquad w _ {\eta} ^ {J \setminus \{j \}} = \underset {w \in \mathcal {W} ^ {J \setminus \{j \}}} {\arg \min } \| \tilde {X} _ {\eta} ^ {J} w _ {J} - \tilde {y} _ {\eta} ^ {J} \|,
$$
so we can interpret penalized synthetic control on units  $J$  and  $J \setminus \{j\}$  with time periods  $\{1, \ldots, T\}$  and the original outcomes as non-penalized synthetic control on units  $J$  and  $J \setminus \{j\}$  with time periods  $J$  and transformed outcomes, where we note that the synthetic-control solutions are unique in this case. Hence, we can apply the previous result to conclude that there exists  $\lambda_{\eta} \in \mathbb{R}^{J}$  with  $\lambda_{\eta,j} \geq 0$  for all  $j \in J$  such that  $\tilde{X}_{\eta}^{J} \tilde{w}_{\eta}^{J} = \sum_{j \in J} \lambda_{\eta,j} \tilde{X}_{\eta}^{J} \tilde{w}_{\eta}^{J \setminus \{j\}}$ . Since  $\tilde{X}_{\eta}^{J}$  is invertible, it now also follows that  $\tilde{w}_{\eta}^{J} = \sum_{j \in J} \lambda_{\eta,j} \tilde{w}_{\eta}^{J \setminus \{j\}}$ . Since also  $\tilde{w}_{\eta}^{J} \in \mathcal{W}$  and  $\tilde{w}_{\eta}^{J \setminus \{j\}} \in \mathcal{W}$  for all  $j \in J$ , we have that  $\sum_{j \in J} \lambda_{\eta,j} = \sum_{j \in J} \lambda_{\eta,j} 1' \tilde{w}_{\eta}^{J \setminus \{j\}} = 1' \tilde{w}_{\eta}^{J} = 1$ . This establishes the main claim of the proposition for penalized synthetic control.
As the fifth and final step, we derive the main result on minimum-norm synthetic control from the above results on penalized synthetic control. Consider some sequence  $(\eta_{\iota})_{\iota = 1}^{\infty}$  in  $(0,\infty)$  with  $\eta_{\iota}\rightarrow 0$  and for every  $\iota$  apply the previous step to the penalized synthetic control estimator with penalty  $\eta_{\iota}$  to obtain a weight vector  $\lambda_{\eta_{\iota}}\in \Lambda^{J} = \{\lambda \in [0,1]^{J};\sum_{j = 1}^{J}\lambda_{j} = 1\}$ . Since  $\Lambda^J$  is compact,  $(\lambda_{\eta_{\iota}})_{\iota = 1}^{\infty}$  must have a converging subsequence with some limit  $\lambda \in \Lambda^J$ . Using the limit along this subsequence, we have that
$$
\hat {w} ^ {J} = \lim  _ {\iota \rightarrow \infty} \hat {w} _ {\eta_ {\iota}} ^ {J} = \lim  _ {\iota \rightarrow \infty} \sum_ {j \in J} \lambda_ {\eta_ {\iota}, j} \tilde {w} _ {\eta_ {\iota}} ^ {J \backslash \{j \}} = \sum_ {j \in J} \left(\lim  _ {\iota \rightarrow \infty} \lambda_ {\eta_ {\iota}, j}\right)\left(\lim  _ {\iota \rightarrow \infty} \tilde {w} _ {\eta_ {\iota}} ^ {J \backslash \{j \}}\right) = \sum_ {j \in J} \lambda_ {j} \tilde {w} ^ {J \backslash \{j \}}.
$$
Proof of Proposition 5. By Jensen's inequality applied to an average over the bounds in (3),
$$
\operatorname {E}[(y - \hat{f}^{*}(x))^{2}]\leq \frac{1}{|J|!}\sum_{\pi}\operatorname {E}[(y - \hat{f}_{\pi}^{*}(x))^{2}]\leq \sum_{j\in J}\operatorname {E}\bigg[\underbrace{\frac{1}{|J|!}\sum_{\pi}\hat{\lambda}_{\pi(j)}}_{= \frac{1}{|J|}}(y - \hat{f}^{j}(x))^{2}\bigg].
$$
B Details of the Empirical Illustrations
B.1 Many-Regressor Linear Least-Squares on CPS Data
We utilize the publicly available $^{5}$  CPS control and NSW experimental control datasets, drawn from the study presented in LaLonde (1986) as used by Dehejia and Wahba (1999, 2002). The resulting data has 15,992 observations for CPS and 260 for NSW, with both datasets containing an identical set of variables, detailed in Table 1.
Variable	Data Type	Description
age	Discrete	Age
education	Discrete	Years of education
black	Dummy	Black
hispanic	Dummy	Hispanic
married	Dummy	Marital status
nodegree	Dummy	Lack of college degree
re74	Continuous	Income in 1974
re75	Continuous	Income in 1975
re78	Continuous	Income in 1978
Table 1: CPS and NSW dataset variables
We use re78 as the outcome variable and all other variables as covariates. In order to achieve high dimensionality, we first discretize the continuous income covariates into 50 bins via quantile binning. We then construct a series of dummies for each discrete variable, corresponding to indicators for each discretized value. We then interact all these dummy variables, as well as those covariates which were originally dummies, taking care not to interact those which are mutually exclusive (e.g. originating from the same original covariate or corresponding to race). We then drop any interactions that are zero for all observations in the data. The resulting transformed dataset contains 8,408 dummy covariates, as well as the unmodified outcome variable. In order to ensure that the covariate matrix is full row rank for an arbitrary subset of columns, we go on to add iid  $\mathcal{N}(0,0.0004)$  noise to each of the covariate values (again leaving the outcome variable unaffected). We then select a random subset of 3,000 observations from the CPS dataset as our in-sample set, using the 260 NSW observations as our out-of-sample set.
For fitting models of varying complexity, we randomly permute the order of the columns of the covariate matrix; denote the resulting matrix as  $X$ . We then add an intercept and iterate over varying levels of complexity  $\ell$ , ranging from 1 to 8,409, corresponding to the number of covariates that we will use for estimation. We then estimate the OLS coefficient vector  $\hat{\beta}^{\ell} = X^{\ell \dagger}y$ , where  $\dagger$  denotes the Moore-Penrose pseudoinverse.
To evaluate performance, we first select a sample size  $m$  and then draw 1000 samples of size  $m$  from the NSW set. We then take define our evaluation metric to be the RMSE across these 1000 samples, with error defined as the difference between the average predicted outcome and the true average outcome for each sample:
$$
\operatorname {R M S E} (\ell , m) = \sqrt {\frac {1}{1 0 0 0} \sum_ {j = 1} ^ {1 0 0 0} \left[ \left(\frac {1}{m} \sum_ {i = 1} ^ {m} y _ {j i} ^ {*}\right) - \left(\frac {1}{m} \sum_ {i = 1} ^ {m} x _ {j i} ^ {* \top} \hat {\beta} \ell\right) \right] ^ {2}}
$$
$^{5}$ users.nber.org/\~rdehejia/data/.nswdata2.html. We use the files corresponding to cps Controls.txt and nswre74_control.txt.
This metric assesses the ability of the given model to accurately predict mean outcomes, a quantity of direct relevance to ATE estimation. Where  $X^{*}, y^{*}$  denote the out-of-sample covariate and outcome variables, respectively, and can correspond to either the CPS or NSW held-out samples. The subscript  $ji$  refers to the  $i$ th observation of the  $j$ th sample of size  $m$ . In order to smooth out the effects of the random ordering of columns, we repeat this exercise for five different random orderings and take a pointwise average to obtain smooth RMSE vs. complexity curves (Figure 1b). A zoomed-in version of this plot, focusing on the highly overparametrized regime and emphasizing the ability of some of the depicted models to outperform simple baselines can be found in Figure 7.
Figure 7: Average RMSE for linear regression for a varying number of covariates  $(\ell >5000)$
As a further illustration of the effects of increasing complexity, Figure 8 shows the average norm of the model coefficients across different model complexities, with the average taken over random covariate orderings as before. Starting small, model coefficients initially grow (in terms of their Euclidean norm), reaching their peak at the interpolation threshold. To the right of the interpolation threshold, the norm of the model coefficients decreases mechanically, since the estimator now minimizes the norm among all interpolating solutions with fewer and fewer sparsity constraints.
Figure 8: Average of the coefficient norm  $\|\hat{\beta}^J\|$  for varying size of the set  $J$  of covariates.
B.2 Many-Unit Synthetic Control on Smoking Data
For our synthetic-control exercise, we utilize public data from the Centers for Disease Control and Prevention<sup>6</sup> containing annual cost, revenue, tax, and quantity data for cigarette sales by state for the years 1970 to 2019. We follow the approach of Abadie et al. (2010) in using synthetic control to estimate per-capita cigarette pack consumption for the target state, California, as a function of the other 49 states and Washington, D.C. For our evaluation, we utilize two years of data (1987 and 1988) as a hold-out sample and fit the model on three years (1984 to 1986). All of our data precedes the year in which anti-smoking legislation took effect in California (1989).
We begin by selecting a random subset of 20 states to serve as our donor pool, for computational tractability. We then select a level of complexity  $\ell$  and select a subset of  $\ell$  states from the chosen 20. Using that subset, we then estimate synthetic control weights based on the in-sample period,
${}^{6}$  chronicdata.cdc.gov/Policy/The-Tax-Burden-on-Tobacco-1970-2019/7nwe-3aj9.
choosing convex weight vector  $\hat{w}^{\ell}\in \mathcal{W} = \{w\in [0,1]^{N};\sum_{i = 1}^{n}w_{i} = 1\}$  as described in Section 3.1:
$$
\hat {w} ^ {\ell} = \underset {w \in \widehat {\mathcal {W}} ^ {\ell}} {\arg \min } \| w \| \quad \widehat {\mathcal {W}} ^ {\ell} = \underset {w \in \mathcal {W}; w _ {j} = 0 \forall j \notin J} {\arg \min } \sum_ {t = 1 9 8 4} ^ {1 9 8 6} (y _ {0 t} - \sum_ {i = 1} ^ {\ell} w _ {i} y _ {i t}) ^ {2}.
$$
Here,  $y_0$  denotes the target state, California. We then compute the out-of-sample prediction error as:
$$
\operatorname {R M S E} (\ell) = \sqrt {\frac {1}{2} \sum_ {t = 1 9 8 7} ^ {1 9 8 8} \left(y _ {0 t} - \sum_ {i = 1} ^ {\ell} \hat {w} _ {i} ^ {\ell} y _ {i t}\right) ^ {2}}
$$
We then iterate over all  $\binom{20}{\ell}$  possible combinations of donor units for the given complexity level and take the average RMSE value to be the predictive error for the given complexity level. We vary  $\ell$  from 1 to 20 to trace out the curve of synthetic control prediction risk vs. complexity (Figure 4).
As a robustness check, we also consider synthetic control with 10 pre-treatment periods, where the included control states are chosen among all 50 available donors (49 other states and Washington, D.C.). In particular, at each complexity level  $\ell$  we consider  $\min \left\{\binom{50}{\ell}, 10000\right\}$  combinations of donor units. The results are qualitatively similar, and provided in Figure 9.
Figure 9: Average out-of-time (blue) and training (orange) RMSE for synthetic control for a varying number of control units as in Figure 4, but with ten pre-treatment periods and control units chosen randomly among all available donors.