SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3 depicts this graphically. In Figure 3a, for the non-interpolating case we observe that the norm of the difference between the model coefficient vectors making use of both covariates  $(\hat{\beta}^{\{1,2\}}, \tilde{\beta}^{\{1,2\}})$  is larger than the norm of the differences between the coefficient vectors taking into consideration just a single covariate at a time. In Figure 3b for the interpolating case, we observe that the reverse is true; in this case, the norm of the difference between the complex models is smaller than the norms of the differences between the simpler models.
In practice, we may care about model properties beyond variance, and consider imputation loss beyond the above norms in the parameters. In Section 4, we will leverage the model-averaging properties from Proposition 1 to establish such bounds on more general imputation errors.
We believe that these variance and geometric properties of linear regression are well understood in the literature and likely not new, although we are not aware of an explicit statement of the model-averaging connection between more and less complex interpolating linear-regression models.
3 Single Descent for Synthetic Control
We next consider imputation using synthetic control with many control units. As in the case of linear regression, we start with an empirical illustration. In the Abadie et al. (2010) California smoking
The second result still holds for an alternative norm  $\| \beta \| _M = \sqrt{\beta'M\beta}$  with  $M$  positive definite and symmetric, provided that the same norm is used when selecting the norm-minimizing estimator in the interpolation regime.
(a) Non-interpolating case with  $n = 2$ .
(b) Interpolating case with  $n = 1$
Figure 3: Minimal-norm least-squares solutions for draws  $Y_{A}$  (black) and  $Y_{B}$  (orange) for linear regression with  $J = \{1,2\}$ , where  $n$  varies
dataset, we impute smoking rates for a target state for a varying number of control states. We then discuss theoretical properties of the synthetic control estimator, which we connect to its imputation quality in the following Section 4.
3.1 Setup and Estimator
We consider a panel of  $N + 1$  units observed over  $T$  time periods,  $Y = (y_{it})_{i\in \{0,\dots ,N\} ,t\in \{1,\dots ,T\}}\in$ $\mathbb{R}^{(N + 1)\times T}$ , where  $i = 0$  denotes the target unit. Our goal is to impute  $y_{0t}$  for  $t\in \{T + 1,\ldots ,T + S\}$  given  $y_{it}$  for  $i\in \{1,\ldots ,N\}, t\in \{T + 1,\ldots ,T + S\}$  by the synthetic-control estimator  $\hat{y}_{0t} = \sum_{i = 1}^{N}\hat{w}_{i}y_{it}$  with convex weights  $\hat{w}\in \mathcal{W} = \{w\in [0,1]^N;\sum_{i = 1}^n w_i = 1\}$ . Specifically, for a subset  $J\subseteq \{1,\dots ,N\}$  of control units, we consider the synthetic control weights
$$
\hat {w} ^ {J} = \underset {w \in \widehat {\mathcal {W}} ^ {J}} {\arg \min } \| w \| \quad \widehat {\mathcal {W}} ^ {J} = \underset {w \in \mathcal {W}; w _ {j} = 0 \forall j \notin J} {\arg \min } \sum_ {t = 1} ^ {T} \left(y _ {0 t} - \sum_ {i = 1} ^ {N} w _ {i} y _ {i t}\right) ^ {2}. \tag {1}
$$
Here, we choose the (unique) norm-minimizing synthetic control weights whenever there is more than one empirical risk minimizer. We can also interpret this solution as the limit  $\hat{w}^J$  of a ridge penalized synthetic control estimator  $\hat{w}_\eta^J$ ,
$$
\hat {w} ^ {J} = \lim  _ {\eta \rightarrow 0} \hat {w} _ {\eta} ^ {J} \quad \hat {w} _ {\eta} ^ {J} = \underset {w \in \mathcal {W}; w _ {j} = 0 \forall j \notin J} {\arg \min } \sum_ {t = 1} ^ {T} \left(y _ {0 t} - \sum_ {i = 1} ^ {N} w _ {i} y _ {i t}\right) ^ {2} + \eta \| w \| ^ {2}, \tag {2}
$$
where  $\hat{w}_{\eta}^{J}$  puts a penalty on the Euclidean norm  $\| w\|^2$  of the weights, multiplied by a factor  $\eta > 0$ . This form of the penalized synthetic-control estimator is also considered by Shen et al. (2022).
We note that, unlike in the linear-regression case, we can now end up with non-interpolating solutions even in the case of high model complexity (many control units). The reason is that the convexity restriction  $\hat{w} \in \mathcal{W}$  allows for interpolation only if the target outcomes are in the convex hull of the control outcomes.
3.2 Empirical Motivation
To illustrate some properties of the synthetic control estimator, we impute California smoking rates in the Abadie et al. (2010) dataset. In that dataset, California experiences the introduction of smoking legislation in 1989, for which Abadie et al. (2010) provides a causal effect estimate by imputing counterfactual smoking rates for those years when the legislation is in effect. We instead consider only the time before the legislation is introduced, giving us access to observed control outcomes in all years, even for California. Specifically, we fit synthetic control models for California on  $T = 3$  years of data (1984-1986), and evaluate their imputation performance on the following  $S = 2$  years (1987-1988) in terms of mean-squared error.
When fitting synthetic control models, we vary how many of the  $N = 20$  control states we include in the estimation process. Specifically, for a given complexity  $\ell \in \{1,\dots ,N\}$ , we average out-of-time
root-mean squared error (RMSE) across all  $\binom{N}{\ell}$  combinations of control states. We report results in Figure 4.
Figure 4: Average out-of-time (blue) and training (orange) RMSE for synthetic control for a varying number of control units.
Unlike the linear-regression case, the loss in the synthetic-control case changes monotonically: as we increase the number of control units, average RMSE decreases. We therefore observe a single-descent curve in the relationship of complexity and loss, with no notable change in regimes when the number of control states surpasses the number  $T = 3$  of training periods.
As before, our illustration is extreme: by using only three training periods and a random selection of control states, we can provide a stark illustration of the difference in behavior between the synthetic control and linear-regression estimators.
3.3 Theoretical Properties and Graphical Illustration
While linear regression and synthetic control behave differently in terms of their double- vs single-descent behavior, we note that both exhibit continuing returns to increasing complexity, with no limit. In our empirical illustration, that return to complexity kicks in the interpolation regime for linear regression and throughout for synthetic control. We now connect this commonality in returns to complexity to a corresponding theoretical connection.
Proposition 4 (Model averaging for synthetic control). For all  $J$  with  $|J| > 1$  and data  $Y$  there exists
$$
\hat {\lambda} \in [ 0, 1 ] ^ {J}, \sum_ {j \in J} \hat {\lambda} _ {j} = 1
$$
$$
\hat {w} ^ {J} = \sum_ {j \in J} \hat {\lambda} _ {j} \hat {w} ^ {J \backslash \{j \}}.
$$
Hence, synthetic control has the same model-averaging property as interpolating linear regression (Proposition 1). Now, however, the model-averaging property also holds without interpolation, and is instead driven by the convexity of synthetic control weights. Furthermore, the weights can depend on outcome data, although only for pre-treatment outcomes. We further note that the result extends to penalized synthetic control with some fixed penalty parameter  $\eta$ .
We illustrate the model-averaging property for synthetic control in Figure 5, where we show that synthetic California with  $|J| = 2$  (left) and  $|J| = 3$  (right) control states is a convex combination of synthetic California with fewer control states. Here, we focus on the fit in the training data, and do not explicitly show the underlying synthetic-control weights. We note that Figure 5 covers both cases where the synthetic estimator for California has perfect training fit (right) and cases where it does not (left).
Unlike in the case of linear regression (Proposition 3) we do not, however, obtain an immediate bound on the variation of synthetic control models. Indeed, as the case of  $|J| = 2$  controls in Figure 5 clarifies, the complex model can have more variance in weights than the simple ones (which here do not vary at all), despite the model-averaging property. In the next section, we will therefore connect model averaging directly to improvements of imputation quality, without relying on explicit variation results.
(a) Non-interpolating case,  $|J| = 2$
(b) Interpolating case,  $|J| = 3$
Figure 5: Synthetic-control examples for  $T = 2$ , where the set  $J$  of included units varies.
4 Model-Averaging Based Risk Bounds
Above, we argued that more complex models are model averages over simpler models in two cases that are relevant to causal inference: interpolating linear regression and synthetic control. In this section, we discuss conditions under which model averaging leads to better imputation quality.
To unify the above cases, we now consider generic estimated functions  $\hat{f}^*:\mathcal{X}\to \mathbb{R}$  that can be related to simpler models  $\hat{f}^j:\mathcal{X}\rightarrow \mathbb{R}$  for an index set  $j\in J$  by the model-averaging property
$$
\hat {f} ^ {*} = \sum_ {j \in J} \hat {\lambda} _ {j} \hat {f} ^ {j} \quad \text {f o r s o m e}
$$
$$
\hat {\lambda} \in [ 0, 1 ] ^ {J}, \sum_ {j \in J} \hat {\lambda} _ {j} = 1. \quad (\mathrm {M A})
$$
We see this model-averaging property as a purely mechanical property of estimators, which we apply to interpolating linear regression (Proposition 1) and to synthetic control (Proposition 4) by our results above
Model averaging provides some insurance against excess loss. Intuitively, being able to represent a more complex model  $\hat{f}^*$  in terms of a model average over simpler models diversifies the risk of a bad imputation fit. When considering convex loss functions, we can make this intuition precise via a simple application of Jensen's inequality, which yields for the case of squared error that
$$
(y - \hat {f} ^ {*} (x)) ^ {2} \leq \sum_ {j \in J} \hat {\lambda} _ {j} (y - \hat {f} ^ {j} (x)) ^ {2}. \tag {3}
$$
for any target point  $(y, x) \in \mathbb{R} \times \mathcal{X}$ . Hence, imputation loss using the complex model is at most a weighted average over the loss of simpler models. The relationship also extends directly to estimating averages of outcomes  $y$  by averages of predictions  $\hat{f}(x)$ , as in the case of average treatment effects in Section 2.
In principle, the simple portfolio bound in (3) leaves open the possibility that the more complex model  $\hat{f}^*$  performs as poorly as the worst of the simpler models  $\hat{f}^j$ . However, for this to occur, the weights would have to be positively correlated with bad performance. A condition on imputation quality we can therefore consider imposing is that the selection of weights is not, on average, working against imputation quality. The following result formalizes this idea on a (very) high level.
Proposition 5 (Model-agnostic risk bound). Assume that (MA) holds and that for some distribution over training and target data we have that for all permutations  $\pi : J \to J$
$$
\mathrm {E} \left[ (y - \hat {f} ^ {*} (x)) ^ {2} \right] \leq \mathrm {E} \left[ (y - \hat {f} _ {\pi} ^ {*} (x)) ^ {2} \right] \quad \text {w h e r e} \quad \hat {f} _ {\pi} ^ {*} = \sum_ {j \in J} \hat {\lambda} _ {\pi (j)} \hat {f} ^ {j}. \tag {P}
$$
Then we obtain the bound  $\operatorname{E}[(y - \hat{f}^* (x))^2 ]\leq \frac{1}{|J|}\sum_{j\in J}\operatorname {E}[(y - \hat{f}^j (x))^2 ]$
In words, if the model chosen by the data on average over some distribution is not worse than a model where we mix up weights, then the imputation performance of the complex model is not worse than the average of the imputation performances of simple models, leading to observations like those in Figures 1b and 4 where increased model complexity leads to improved imputation quality for randomly-ordered models.
We make two comments on the formal conditions of the proposition. First, it is enough to assume that (P) holds on average over permutations chosen uniformly at random. Second, the distribution behind the expectation  $\mathrm{E}$  can incorporate prior distributions over underlying parameters, in which case the resulting bound holds on average over the same distribution.
In the examples of linear regression and synthetic control, Figure 6 illustrates the respective permuted models. In those cases, the permutation assumption (P) assumes that, on average, the model chosen by the data is not worse than a model where we mix up the weights (in gray).
(a) Interpolating linear regression
(b) Synthetic control
Figure 6: Illustration of permutation bound based on the examples from Figures 2 and 5.
While more primitive conditions may be helpful to judge when a condition like (P) holds, we note two attractive properties. First, the assumption complements the model-averaging property (MA) in an important way: While model averaging relates more complex to less complex models, the permutation property only compares models of comparable complexity (assuming that there are no systematic ex-ante differences between the  $\hat{f}^j$ ). To violate this property would thus amount to assuming that selection among models with comparable complexity is disadvantageous, which may be unreasonable to expect on average. Second, we can formulate this condition on the level of estimators, without explicit reference to the underlying data-generating process.
We study the imputation performance of interpolating linear regression and synthetic control, and provide a unified perspective on returns to complexity in both cases: More complex models can be expressed as model averages over simpler ones. While we provide some high-level assumptions on when this model-averaging property improves average imputation risk, more work is needed to establish primitive sufficient conditions. This includes, in particular, studying how the bias changes as models become more complex. In addition, we limit our analysis to comparing more complex to simpler models when features or control units are randomly ordered, but in practice, we may have knowledge about which simple models are more plausible than others. However, our results show that highly over-parameterized models that achieve perfect in-sample fit can yield measurable performance improvements over non-random simple models in causal settings. Future research could explore conditions under which this phenomenon holds, namely where complex models can beat non-random simple ones.
We thank Tengyuan Liang, Sendhil Mullainathan, Ashesh Rambachan, audiences at Emory University, the 2023 "Econometrics in the Era of Machine Learning" conference at the University of Chicago, the 2023 "HDMetrics: Big Data, High-Dimensional Methods, and Machine Learning" workshop at the University of Illinois at Urbana-Champaign, the 2023 CEME Conference for Young Econometricians at Georgetown University, and the 2023 California Econometrics Conference at the University of Washington, as well as four anonymous reviewers for helpful comments and discussions.
Computational support was provided by the Data, Analytics, and Research Computing (DARC) group at the Stanford Graduate School of Business (RRID:SCR_022938).
Abadie, Alberto, Alexis Diamond, and Jens Hainmueller (2010). Synthetic control methods for comparative case studies: Estimating the effect of california's tobacco control program. Journal of the American statistical Association, 105(490):493-505. (Cited on pages 2, 6, 7, and 17.)
Abadie, Alberto and Jérémy L'Hour (2021). A penalized synthetic control estimator for disaggregated data. Journal of the American Statistical Association, 116(536):1817-1834. (Cited on page 3.)
Agarwal, Anish, Munther Dahleh, Devavrat Shah, and Dennis Shen (2021). Causal matrix completion. arXiv preprint arXiv:2109.15154. (Cited on page 3.)
Athey, Susan, Mohsen Bayati, Nikolay Doudchenko, Guido Imbens, and Khashayar Khosravi (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association, 116(536):1716-1730. (Cited on page 3.)
Bartlett, Peter L, Philip M Long, Gábor Lugosi, and Alexander Tsigler (2020). Benign overfitting in linear regression. Proceedings of the National Academy of Sciences of the United States of America, 117(48):30063-30070. (Cited on pages 3 and 5.)
Belkin, Mikhail (2021). Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica, 30:203-248. (Cited on page 3.)
Belkin, Mikhail, Siyuan Ma, and Soumik Mandal (2018). To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning, pages 541-549. PMLR. (Cited on page 3.)
Ben-Michael, Eli, Avi Feller, and Jesse Rothstein (2021). The augmented synthetic control method. Journal of the American Statistical Association, 116(536):1789-1803. (Cited on page 3.)
Bruns-Smith, David, Oliver Dukes, Avi Feller, and Elizabeth L Ogburn (2023). Augmented balancing weights as undersmoothed regressions. (Cited on page 3.)
Claeskens, Gerda and Nils Lid Hjort (2008). Model selection and model averaging. (Cited on page 3.)
Dehejia, Rajeev H and Sadek Wahba (1999). Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American statistical Association, 94(448):1053-1062. (Cited on pages 2, 4, and 16.)
Dehejia, Rajeev H and Sadek Wahba (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and statistics, 84(1):151-161. (Cited on pages 2, 4, and 16.)
Doudchenko, Nikolay and Guido W Imbens (2016). Balancing, regression, difference-in-differences and synthetic control methods: A synthesis. Technical report, National Bureau of Economic Research. (Cited on page 3.)
Hansen, Bruce E (2007). Least squares model averaging. *Econometrica*, 75(4):1175-1189. (Cited on page 3.)
Hastie, Trevor, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani (2022). Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949-986. (Cited on pages 3 and 5.)
Kato, Masahiro and Masaaki Imaizumi (2022). Benign-overfitting in conditional average treatment effect prediction with linear regression. arXiv preprint arXiv:2202.05245. (Cited on page 3.)
Kelly, Bryan T, Semyon Malamud, and Kangying Zhou (2022). The virtue of complexity in return prediction. Technical report, National Bureau of Economic Research. (Cited on page 3.)
LaLonde, Robert J (1986). Evaluating the econometric evaluations of training programs with experimental data. The American economic review, pages 604-620. (Cited on pages 2, 4, and 16.)
Liang, Tengyuan and Alexander Rakhlin (2020). Just interpolate: Kernel "Ridgeless" regression can generalize. The Annals of Statistics, 48(3):1329-1347. (Cited on page 3.)
Liang, Tengyuan, Alexander Rakhlin, and Xiyu Zhai (2020). On the Multiple Descent of Minimum-Norm Interpolants and Restricted Lower Isometry of Kernels. In Abernethy, Jacob and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pages 2683-2711. PMLR. (Cited on pages 3 and 5.)
Liang, Tengyuan and Benjamin Recht (2023). Interpolating Classifiers Make Few Mistakes. Journal of machine learning research: JMLR. (Cited on page 3.)
Mei, Song and Andrea Montanari (2022). The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766. (Cited on page 3.)
Shen, Dennis, Peng Ding, Jasjeet Sekhon, and Bin Yu (2022). Same Root Different Leaves: Time Series and Cross-Sectional Methods in Panel Data. (Cited on pages 3 and 7.)
Wilson, Andrew G and Pavel Izmailov (2020). Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708. (Cited on page 3.)
Zhang, Chiyuan, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals (2016). Understanding deep learning requires rethinking generalization. (Cited on page 3.)
Proof of Proposition 1. We provide a direct proof for the choice  $\lambda_{j} = \frac{1 - X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j}}{|J| - n}$  via the Sherman-Morrison-Woodbury formula. We first note that, for  $k\geq n$ $A\in \mathbb{R}^{n\times k}$  of full row rank  $n$ $a\in \mathbb{R}^n$  , and  $\hat{\alpha} = \arg \min_{\alpha \in \mathbb{R}^k;A\alpha = a}\| \alpha \|$  we have that  $\hat{\alpha} = A^{\prime}(AA^{\prime})^{-1}a$  . Indeed,  $A\hat{\alpha} = a$  , and for any  $\alpha \in \mathbb{R}^k$  with  $A\alpha = a$  and  $\alpha \neq \hat{\alpha}$  , for  $\Pi = A^{\prime}(AA^{\prime})^{-1}A$  we have that
$$
\| \alpha \| ^ {2} = \| \Pi \alpha \| ^ {2} + \| (\mathbb {I} - \Pi) \alpha \| ^ {2} = \| \Pi \hat {\alpha} \| ^ {2} + \| (\mathbb {I} - \Pi) (\alpha - \hat {\alpha}) \| ^ {2} = \| \hat {\alpha} \| ^ {2} + \| \alpha - \hat {\alpha} \| ^ {2} > \| \hat {\alpha} \| ^ {2}.
$$
We next write  $X^{J} \in \mathbb{R}^{n \times k}$  for the matrix with columns  $X_{j}^{J} = X_{j}$  for  $j \in J$  and  $X_{j}^{J} = \mathbf{0}$  for  $j \notin J$ . Applying the above result to  $\hat{\beta}^{J}$  and  $\hat{\beta}^{J \setminus \{j\}}$  for all  $j \in J$ , we find
$$
\hat {\beta} ^ {J} = X ^ {J \prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} Y, \quad \hat {\beta} ^ {J \setminus \{j \}} = X ^ {J \setminus \{j \} \prime} (X _ {J \setminus \{j \}} X _ {J \setminus \{j \}} ^ {\prime}) ^ {- 1} Y.
$$
Using that  $X_{J \setminus \{j\}} X_{J \setminus \{j\}}' = X_J X_J' - X_j X_j'$ , which is invertible by the assumption that  $X_{J \setminus \{j\}}$  is of full row rank, we find by the Sherman-Morrison-Woodbury that
$$
\left(X _ {J \backslash \{j \}} X _ {J \backslash \{j \}}\right) ^ {- 1} = \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} + \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} X _ {j} \left(1 - X _ {j} ^ {\prime} \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} X _ {j}\right) ^ {- 1} X _ {j} ^ {\prime} \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} \tag {4}
$$
with  $X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j}\neq 1$ . Plugging in,
$$
\begin{array}{l} \hat {\beta} ^ {J \backslash \{j \}} = \left(X ^ {J} - X ^ {\{j \}}\right) ^ {\prime} \left(X _ {J \backslash \{j \}} X _ {J \backslash \{j \}}\right) ^ {- 1} Y \\ = \hat {\beta} ^ {J} - X ^ {\{j \} \prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} Y - X ^ {\{j \} \prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} Y \frac {X _ {j} ^ {\prime} (X _ {J} X _ {j} ^ {\prime}) ^ {- 1} X _ {j}}{1 - X _ {j} ^ {\prime} (X _ {J} X _ {j} ^ {\prime}) ^ {- 1} X _ {j}} \\ + X ^ {J \prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} X _ {j} X _ {j} ^ {\prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} Y \frac {1}{1 - X _ {j} ^ {\prime} (X _ {J} X _ {j} ^ {\prime}) ^ {- 1} X _ {j}} \\ = \hat {\beta} ^ {J} + \left(X ^ {J \prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} X _ {j} X _ {j} ^ {\prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1} - X ^ {\{j \} \prime} (X _ {J} X _ {J} ^ {\prime}) ^ {- 1}\right) Y \frac {1}{1 - X _ {j} ^ {\prime} (X _ {J} X _ {j} ^ {\prime}) ^ {- 1} X _ {j}}. \\ \end{array}
$$
Since  $\sum_{j\in J}X_jX_j' = X_JX_J'$  and  $\sum_{j\in J}X^{\{j\}} = X^J$ , we have that
$$
\sum_ {j \in J} \hat {\beta} ^ {J \backslash \{j \}} \lambda_ {j} = \hat {\beta} ^ {J} \sum_ {j \in J} \lambda_ {j} + (| J | - n) \left(X ^ {J \prime} \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} X _ {J} X _ {J} ^ {\prime} \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} Y - X ^ {J \prime} \left(X _ {J} X _ {J} ^ {\prime}\right) ^ {- 1} Y\right) = \hat {\beta} ^ {J} \sum_ {j \in J} \lambda_ {j}.
$$
Finally,  $X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j} \geq 0$  since  $X_{J}X_{J}^{\prime}$  positive definite,  $X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j} \leq 1$  since  $X_{J \setminus \{j\}}X_{J \setminus \{j\}}^{\prime} = X_{J}X_{J}^{\prime} - X_{j}X_{j}^{\prime} \preceq X_{J}X_{J}^{\prime}$  in (4), and  $\sum_{j \in J} X_{j}^{\prime}(X_{J}X_{J}^{\prime})^{-1}X_{j} = \operatorname{tr}\left(\sum_{j \in J} X_{j}X_{j}^{\prime}X_{J}X_{J}^{\prime}\right) = n$ , so  $\lambda_{j} \in [0,1]$  for all  $j \in J$  and  $\sum_{j=1}^{J} \lambda_{j} = 1$ .