SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To obtain efficient implementations of the E-Step of the EM algorithm as well as of the LOOCV shortcut formula, one can exploit the fact that the ridge solution is preserved under orthogonal transformations. Specifically, let  $r = \min(n,p)$  and  $m = \max(n,p)$  and let  $\mathbf{U} \mathbf{\Sigma} \mathbf{V}^{\mathrm{T}} = \mathbf{X}$  be a compact singular value decomposition (SVD) of  $\mathbf{X}$ . That is,  $\mathbf{U} \in \mathbb{R}^{n \times r}$  and  $\mathbf{V} \in \mathbb{R}^{p \times r}$  are semi-orthonormal column matrices, i.e.,  $\mathbf{U}^{\mathrm{T}} \mathbf{U} = \mathbf{I}_n$  and  $\mathbf{V}^{\mathrm{T}} \mathbf{V} = \mathbf{I}_p$ , and  $\boldsymbol{\Sigma} = \mathrm{diag}(s_1, \ldots, s_r) \in \mathbb{R}^{r \times r}$  is a diagonal matrix that contains the non-zero singular values  $\mathbf{s} = (s_1, \ldots, s_r)$  of  $\mathbf{X}$ . With this decomposition, and an additional  $O(nr)$  pre-processing step to compute  $\mathbf{c} = \boldsymbol{\Sigma} \mathbf{U}^{\mathrm{T}} \mathbf{y}$ , we can compute the ridge solution  $\alpha_{\tau} \in \mathbb{R}^{r}$  for a given  $\tau$  with respect to the rotated inputs  $\mathbf{XV}$  in time  $O(r)$  via
$$
\hat {\boldsymbol {\alpha}} _ {\tau} = \left(\boldsymbol {\Sigma} ^ {\mathrm {T}} \mathbf {U} ^ {\mathrm {T}} \mathbf {U} \boldsymbol {\Sigma} + \tau^ {- 2} \mathbf {I}\right) ^ {- 1} \boldsymbol {\Sigma} \mathbf {U} ^ {\mathrm {T}} \mathbf {y} = \left(\boldsymbol {\Sigma} ^ {2} + \tau^ {- 2} \mathbf {I}\right) ^ {- 1} \mathbf {c} = \left(1 / \left(s _ {j} ^ {2} + \tau^ {- 2}\right)\right) _ {j = 1} ^ {r} \odot \mathbf {c} \tag {12}
$$
where  $\mathbf{a} \odot \mathbf{b}$  denotes the element-wise Hadamard product of vectors  $\mathbf{a}$  and  $\mathbf{b}$ . The compact SVD itself can be obtained in time  $O(mr^2)$  via an eigendecomposition of either  $\mathbf{X}^{\mathrm{T}}\mathbf{X} = \mathbf{V}\boldsymbol{\Sigma}^2\mathbf{V}^T$  in case  $n \geq p$  or  $\mathbf{X}\mathbf{X}^{\mathrm{T}} = \mathbf{U}\boldsymbol{\Sigma}^2\mathbf{U}^{\mathrm{T}}$  in case  $n < p$  followed by the computation of the missing  $\mathbf{U} = \mathbf{X}\mathbf{V}\boldsymbol{\Sigma}^{-1}$  or  $\mathbf{V} = \mathbf{X}^{\mathrm{T}}\mathbf{U}\boldsymbol{\Sigma}^{-1}$ .
In summary, after an  $O(mr^2)$  pre-processing step, we can obtain rotated ridge solutions for an individual candidate  $\tau$  in time  $O(r)$ . Moreover, for the optimal  $\tau^{*}$ , we can find the ridge solution  $\hat{\beta}_{\tau^{*}} = \mathbf{V}\hat{\alpha}_{\tau^{*}}$  with respect to the original input matrix via an  $O(pr)$  post-processing step. Below we show how the key statistics that have to be computed per candidate  $\tau$  (and  $\sigma$ ) can be computed efficiently based on  $\hat{\alpha}_{\tau}$ , the pre-computed  $\mathbf{c}$ , and SVD. For the EM algorithm, these are the posterior squared norm and sum of squared errors, and for the LOOCV algorithm, this is the PRESS statistic. While the main focus of this work is the EM algorithm, the fast computation of the PRESS shortcut formula appears to be not widely known (e.g., the current implementation in both scikit-learn and glmnet do not use it) and may therefore be of independent interest.
ESN For the posterior expected squared norm  $\mathrm{ESN} = \sigma^2\mathrm{tr}(\mathbf{A}_\tau^{-1}) + \| \hat{\boldsymbol{\beta}}_\tau \|^2$ , we first observe that  $\| \hat{\boldsymbol{\beta}}_{\tau}\|^{2} = \| \mathbf{V}\hat{\boldsymbol{\alpha}}_{\tau}\|^{2} = \| \hat{\boldsymbol{\alpha}}_{\tau}\|^{2}$ , and then note that the trace can be computed as
$$
\begin{array}{l} \mathrm {t r} (\mathbf {A} _ {\tau} ^ {- 1}) = \mathrm {t r} (\mathbf {V} _ {p} (\mathbf {\Sigma} _ {p} ^ {2} + \tau^ {- 2} \mathbf {I} _ {p}) ^ {- 1} \mathbf {V} _ {p} ^ {\mathrm {T}}) \\ = \tau^ {2} \max  (p - n, 0) + \sum_ {j = 1} ^ {r} 1 / \left(s _ {j} ^ {2} + \tau^ {- 2}\right), \tag {13} \\ \end{array}
$$
where in the first equation we denote by  $\mathbf{V}_p$ ,  $\boldsymbol{\Sigma}_p^2$  the full matrices of eigenvectors and eigenvalues of  $\mathbf{X}^{\mathrm{T}}\mathbf{X}$  (including potential zeros), and in the second equation we used the cyclical property of the trace. Thus, all quantities required for ESN can be computed in time  $O(r)$  given the SVD and  $\hat{\alpha}_{\tau}$ .
ESS For the posterior expected sum of squared errors  $\mathrm{ESS} = \| \mathbf{y} - \mathbf{X}\hat{\beta}_{\tau}\|^{2} + \sigma^{2}\mathrm{tr}(\mathbf{X}^{\mathrm{T}}\mathbf{X}\mathbf{A}_{\tau}^{-1})$  we can compute the residual sum of squares term via
$$
\begin{array}{l} \left\| \mathbf {y} - \mathbf {X} \hat {\boldsymbol {\beta}} _ {\tau} \right\| ^ {2} = \left\| \mathbf {y} \right\| ^ {2} - 2 \mathbf {y} ^ {T} \mathbf {U} \boldsymbol {\Sigma} \hat {\boldsymbol {\alpha}} _ {\tau} + \left\| \mathbf {U} \boldsymbol {\Sigma} \hat {\boldsymbol {\alpha}} _ {\tau} \right\| ^ {2} \\ = \| \mathbf {y} \| ^ {2} - 2 \hat {\alpha} _ {\tau} ^ {\mathrm {T}} \mathbf {c} + \| \mathbf {s} \odot \hat {\alpha} _ {\tau} \| ^ {2}, \tag {14} \\ \end{array}
$$
where we use  $\hat{\beta}_{\tau} = \mathbf{V}\hat{\alpha}_{\tau}$  and  $\mathbf{X} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^{T}$  in the first equation and the orthonormality of  $\mathbf{U}$  and the definition of  $\mathbf{c} = \boldsymbol{\Sigma}\mathbf{U}^{\mathrm{T}}\mathbf{y}$  in the second. Finally, for the trace term, we find that
$$
\begin{array}{l} \operatorname {t r} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} \mathbf {A} _ {\tau} ^ {- 1}\right) = \operatorname {t r} \left(\mathbf {V} \boldsymbol {\Sigma} ^ {2} \mathbf {V} ^ {\mathrm {T}} \left(\mathbf {V} \left(\boldsymbol {\Sigma} ^ {2} + \tau^ {- 2} \mathbf {I} _ {p}\right) \mathbf {V} ^ {\mathrm {T}}\right) ^ {- 1}\right) \\ = \sum_ {j = 1} ^ {r} s _ {j} ^ {2} / \left(s _ {j} ^ {2} + \tau^ {- 2}\right). \tag {15} \\ \end{array}
$$
PRESS The shortcut formula of the PRESS statistic (4) for a candidate  $\lambda$  requires the computation of the diagonal elements of the hat matrix  $\mathbf{H}(\lambda) = \mathbf{X}(\mathbf{X}^{\mathrm{T}}\mathbf{X} + \lambda \mathbf{I}_p)^{-1}\mathbf{X}^{\mathrm{T}}$  as well as the residual vector  $\mathbf{e} = \mathbf{y} - \mathbf{H}\mathbf{y}$ . With the SVD, the first simplifies to
$$
\begin{array}{l} \mathbf {H} (\lambda) = \mathbf {U} \boldsymbol {\Sigma} \left(\boldsymbol {\Sigma} ^ {2} + \lambda \mathbf {I} _ {r}\right) ^ {- 1} \boldsymbol {\Sigma} \mathbf {U} ^ {\mathrm {T}} \\ = \mathbf {U} \operatorname {d i a g} \left(\frac {s _ {1} ^ {2}}{s _ {1} ^ {2} + \lambda}, \dots , \frac {s _ {r} ^ {2}}{s _ {r} ^ {2} + \lambda}\right) \mathbf {U} ^ {\mathrm {T}} \\ \end{array}
$$
where we use the fact that diagonal matrices commute. This allows to compute the desired diagonal elements  $h_{ii}$  in time  $O(r)$  via
$$
h _ {i i} (\lambda) = \sum_ {j = 1} ^ {r} u _ {i j} ^ {2} s _ {j} ^ {2} / \left(s _ {j} ^ {2} + \lambda\right) \tag {16}
$$
where  $u_{ij}$  denotes the elements of  $\mathbf{U}$ . Computing the residual vector is easily done via the rotated ridge solution  $\mathbf{e} = \mathbf{y} - \mathbf{U}\boldsymbol{\Sigma}\hat{\boldsymbol{\alpha}}_{\lambda}$ . However, this still requires  $O(nr)$  operations, simply because there are  $n$  residuals to compute.
Thus, in summary, by combining the pre-processing with the fast computation of the PRESS statistic, we obtain an overall  $O(mr^2 + lqnr)$  implementation of ridge regression via LOOCV where  $l$  denotes the number of candidate  $\lambda$  and  $q$  the number of regression target variables. In contrast, for the EM algorithm, by combining the fast computation of ESS and ESN, we end up with an overall complexity of  $O(mr^2 + kqr)$  where  $k$  denotes the number of EM iterations. If we further assume that  $k = o(n)$ , which is supported by experimental results, see Sec. 5, and that both  $q, p = O(\sqrt{n})$  there is an asymptotic advantage of a factor of  $l$  of the EM approach. This regime is common in settings where more data allows for more fine-grained input as well as output measurements, e.g., in satellite time series classification via multiple target regression [11, 37]. All time complexities are summarized in Tab. 1 and detailed pseudocode for both the fast EM algorithm and the fast LOOCV algorithm is provided in the Appendix (see Table 3 and 4).
5 Empirical Evaluation
In this section, we compare the predictive performance and computational cost of LOOCV against the proposed EM method. We present numerical results on both synthetic and real-world datasets. To implement the LOOCV estimator, we use a predefined grid,  $L = (\lambda_1,\dots ,\lambda_l)$ . We use the two most common methods for this task: (i) fixed grid - arbitrarily selecting a very small value as  $\lambda_{\mathrm{min}}$ , a large value as  $\lambda_{\mathrm{max}}$ , and construct a sequence of  $l$  values from  $\lambda_{\mathrm{max}}$  to  $\lambda_{\mathrm{min}}$  on log scale; (ii) data-driven grid - find the smallest value of  $\lambda_{\mathrm{max}}$  that sets all the regression coefficient vector to zero  $^2$  (i.e.  $\hat{\beta} = 0$ ), multiply this value by a ratio such that  $\lambda_{\mathrm{min}} = \kappa \lambda_{\mathrm{max}}$  and create a sequence from
For ridge regression,  $\lambda_{\mathrm{max}} = \infty$ . Following the glmnet package, the sequence of  $\lambda$  is derived for  $\alpha = 0.001$ . The penalty function used by glmnet is  $\lambda[(1 - \alpha)\|\beta\|_2^2 + \alpha\|\beta\|_1]$ , where  $\alpha = 0$  corresponds to ridge regression.
Figure 2: Comparison of EM to LOOCV variants for increasing  $n$  and  $p$  for settings with  $\mathbf{x} \sim N_p(\mathbf{0},\Sigma)$  and  $\mathbf{y}|\mathbf{x} \sim N(\mathbf{x}^T\boldsymbol {\beta},0.25)$  with random  $\beta \sim N_{p}(\mathbf{0},\mathbf{I})$  and  $\Sigma \sim W_{p}(I,p)$ .
$\lambda_{\mathrm{max}}$  to  $\lambda_{\mathrm{min}}$  on log scale. The latter method is implemented in the glmnet package in combination with an adaptive  $\kappa$  coefficient
$$
\kappa = \left\{ \begin{array}{l l} 0. 0 0 0 1 & , \text {i f} n \geq p \\ 0. 0 1 & , \text {o t h e r w i s e} \end{array} \right.,
$$
which we replicate here as input to our fast LOOCV algorithm (Appendix, Table 4) to efficiently recover the glmnet LOOCV ridge estimate.
We consider a fixed grid of  $\lambda = (10^{-10},\dots,10^{10})$  and the grid based on the glmnet heuristic; in both cases, we use a sequence of length 100. The latter is a data-driven grid, so we will have a different penalty grid for each simulated or real data set. Our EM algorithm does not require a predefined penalty grid, but it needs a convergence threshold which we set to be  $\epsilon = 10^{-8}$ . All experiments in this section are performed in Python and the R statistical platform. Datasets and code for the experimental results is publicly available. As is standard in penalized regression, and without any loss of generality, we standardized the data before model fitting. This means that the predictors are standardized to have zero mean, standard deviation of one, and the target has a mean of zero, i.e., the intercept estimate is simply  $\hat{\beta}_0 = (1 / n)\sum y_i$ .
5.1 Simulated Data
In this section, we use a simulation study to investigate the behavior of EM and LOOCV as a function of the sample size,  $n$ , and two other parameters of interest: the number of covariates  $p$ , and the noise level of the target variable. In particular, we are interested in the parameter estimation performance, the corresponding  $\lambda$ -values, and the computational cost. To gain further insights into the latter, the number of iterations performed by the EM algorithm is of particular interest, as we do not have quantitative bounds for its behavior. We consider two settings that vary in the level of sparsity and correlation structure of the covariates. The first setting (Fig. 1) assumes i.i.d Bernoulli distributed covariates with small success probabilities that result in sparse covariate matrices, while the second setting (Fig. 2) assumes normally distributed covariates with random non-zero covariances. In both cases, the target variable is conditionally normal with mean  $\mathbf{x}^{\mathrm{T}}\beta$  for a random  $\beta$  drawn from a standard multivariate normal distribution.
Looking at the results, a common feature of both settings is that the computational complexity of the EM algorithm is a non-monotone function in  $n$ . In contrast to LOOCV, the behavior of EM shows distinctive phases where the complexity temporarily decreases with  $n$  before it settles into the, usually expected, monotonically increasing phase. As can be seen, this is due to the behavior of the number of iterations  $k$ , which peaks for small values of  $n$  before it declines rapidly to a small
3 glmnet LOOCV is computed directly by model refitting via coordinate-wise descent which can be slow.
constant (around 10) when the cost of the pre-processing begins to dominate. The occurrence of these phases is more pronounced for both growing  $p$  and growing  $\sigma$ . This behavior is likely due to the convergence to normality of the posterior distribution as the sample size  $n \to \infty$ , with convergence being slower for large  $p$ .
An interesting observation is that CV with the employed glmnet grid heuristic fails, in the sense that the resulting ridge estimator does not appear to be consistent for large  $p$  in Setting 2. This is due to the minimum value of  $\lambda$  produced by the glmnet heuristic being too large, and the resulting ridge estimates being overshrunk. This clearly underlines the difficulty of choosing a robust dynamic grid – a problem that our EM algorithm avoids completely.
5.2 Real Data
We evaluated our EM method on 24 real-world datasets. This includes 21 datasets from the UCI machine learning repository [5] (unless referenced otherwise) for normal linear regression tasks and 3 time-series datasets from the UCR repository [10] for multitarget regression tasks. The latter is a multilabel classification problem in which the feature matrix was generated by the state-of-the-art HYDRA [12] time series classification procedure (which by default uses LOOCV ridge regression for classification), and we train  $q$  ridge regression models in a one-versus-all fashion, where  $q$  is the number of target classes. The datasets were chosen such that they covered a wide range of sample sizes,  $n$ , and number of predictors,  $p$ . We compared our EM algorithm against the fast LOOCV in terms of predictive performance, measured in  $R^2$  (and classification accuracy) on the test data, and computational efficiency.
Our linear regression experiments involve 3 settings: (i) standard linear regression; (ii) second-order multivariate polynomial regression with added interactions and second-degree polynomial transformations of variables, and (iii) third-order multivariate polynomial regression with added three-way interactions and cubic polynomial transformations. For each experiment, we repeated the process 100 times and used a random  $70/30$  train-test split. Due to memory limitations, we limit our design matrix size to a maximum of 35 million entries. If the number of transformed predictors exceeded this limit, we uniformly sub-sampled the interaction variables to ensure that  $p^* \leq 35000000 / (0.7n)$ , and then fit the model using the sampled variables. Note that we always keep the original variables (main effects) and sub-sampled the interactions. In the case of multitarget regression, we performed a random  $70/30$  train-test split and repeated the experiment 30 times. To ensure efficient reproducibility of our experiments, we set a maximum runtime of 3 hours for each dataset. Any settings that exceeded this time limit were consequently excluded from the result table.
Table 2 details the results of our experiments; specifically, the ratio of time taken to run fast LOOCV divided by the time taken to run our EM procedure  $(T)$ , and the  $R^2$  values obtained by both methods on the withheld test set. The number of features,  $p$ , and observations,  $n$  recorded are values after data preprocessing (missing observations removed, one-hot encoding transformation, etc.). The results demonstrate that our EM algorithm can be up to 49 times faster than the fast LOOCV, with the speed-ups becoming more apparent as the sample size  $n$  and the number of target variables  $q$  increases. In addition, we see that this advantage in speed does not come at a cost in predictive performance, as our EM approach is comparable to, if not better than, LOOCV in almost all cases (also see Appendix, Figure 1, in which most of  $R^2$  values are distributed along the diagonal line).
An interesting observation is that LOOCV using the fixed grid can occasionally perform extremely poorly (as indicated by large negative  $R^2$  values) while LOOCV using the glmnet grid does not seem to exhibit this behavior. This appears likely to be due to the grid chosen using the glmnet heuristic. Its performance is artificially improved because it is unable to evaluate sufficiently small values of  $\lambda$  and is not actually selecting the very small  $\lambda$  value that minimizes the LOOCV score. The incorrectly large  $\lambda$  values are providing protection in these examples from undershrinkage.
The introduced EM algorithm is a robust and computationally fast alternative to LOOCV for ridge regression. The unimodality of the posterior guarantees a robust behavior for finite  $n$  under mild conditions relative to LOOCV grid search, and the SVD preprocessing enables an overall faster computation with an ultra-fast  $O(k\min (n,p))$  main loop. Combining this with a procedure such as orthogonal least-squares to provide a highly efficient forward selection procedure is a promising avenue for future research. As the Q-function is an expected negative log-posterior, it offers a score
Table 2: Real datasets experiment results. The first column is the dataset (abbreviated, refer to Appendix E for the full name); the number of target variables  $q$ , the training sample size  $n$ , the raw number of features  $p$ ;  $T$  is the ratio of time  $t_{\mathrm{CV}} / t_{\mathrm{EM}}$ ;  $p^*$  is the number of features including interactions; EM, Fix, and GLM are the  $R^2$  values on the test data for the three procedures.
DATASET (q)	n	p	LINEAR	2ND ORDER	3RD ORDER
T	EM	Fix	GLM	p*	T	EM	Fix	GLM	p*	T	EM	Fix	GLM
TWITTER	408275	77	20	0.94	0.94	0.94	86	16	0.94	0.94	0.94	-	-	-	-	-
BLOG	39355	275	13	0.46	0.46	0.46	804	9.1	0.51	0.51	0.51	-	-	-	-	-
CT SLICES	37450	379	12	0.86	0.86	0.86	930	7.7	0.92	0.91	0.92	-	-	-	-	-
TOMSHW	19725	96	17	0.63	0.63	0.63	1775	6.5	0.71	0.71	0.71	-	-	-	-	-
NPD - COM	8353	13	13	0.84	0.84	0.84	104	15	1.00	1.00	1.00	559	8.6	1.00	1.00	1.00
NPD - TUR	8353	13	14	0.91	0.91	0.91	104	15	1.00	1.00	1.00	559	8.6	1.00	1.00	1.00
PT - MOTOR	4112	19	13	0.15	0.15	0.15	208	12	0.25	0.19	0.21	1539	3.9	-1.09	0.01	0.04
PT - TOTAL	4112	19	13	0.17	0.17	0.17	208	12	0.24	0.23	0.21	1539	3.7	-1.38	-0.04	0.00
ABALONE	2923	9	13	0.53	0.53	0.53	51	16	0.38	0.35	0.50	209	12	0.28	0.12	0.12
CRIME	1395	99	14	0.66	0.66	0.66	5049	1.3	0.67	-0.74	0.66	17652	1.1	0.66	-0.22	0.60
AIRFOIL	1052	5	17	0.51	0.51	0.51	20	14	0.62	0.62	0.62	55	11	0.73	0.73	0.73
STUDENT	730	39	12	0.18	0.18	0.18	801	3.8	0.19	-0.89	0.16	10693	1.1	0.19	-6.22	-0.18
CONCRETE	721	8	16	0.61	0.61	0.61	44	11	0.78	0.78	0.78	164	5.5	0.85	0.85	0.85
F.FIRES	361	12	2.4	-0.01	-0.01	-0.03	295	0.5	-0.01	-0.01	-0.14	1984	0.3	-0.01	-50.6	-0.45
B.HOUSEING	354	13	11	0.71	0.71	0.71	104	8.1	0.84	0.83	0.80	559	2.2	0.83	-3E2	0.83
FACEBOOK	346	15	15	0.91	0.91	0.91	167	6.6	-5.09	-26.4	-3.99	1087	1.9	-2.53	-2E4	-5.76
DIABETES1	309	10	13	0.49	0.49	0.49	65	6.5	0.49	0.48	0.48	285	2.1	0.47	0.47	0.47
R.ESTATE	289	6	13	0.56	0.56	0.56	27	10	0.65	0.65	0.65	83	5.5	0.65	0.65	0.65
A.MPG	278	8	13	0.81	0.81	0.81	35	8.3	0.86	0.86	0.86	119	4.1	0.86	0.86	0.86
Yacht	215	7	16	0.97	0.97	0.97	27	11	0.97	0.97	0.97	83	6.1	0.98	0.98	0.98
A.MOBILE	111	25	6.1	0.90	0.89	0.89	1076	1.7	0.90	-4E3	0.89	12924	0.5	0.88	-1E4	0.83
EYE1	84	200	2.7	0.50	0.26	0.45	20300	1.3	0.19	0.19	0.19	-	-	-	-	-
RIBO1	49	4088	2.3	0.64	0.64	0.64	-	-	-	-	-	-	-	-	-	-
CROP (24)2	11760	3072	49	0.75	0.75	0.76	-	-	-	-	-	-	-	-	-	-
ELECD (7)2	11645	4096	9.2	0.88	0.88	0.89	-	-	-	-	-	-	-	-	-	-
STARL (3)2	6465	7168	2.1	0.98	0.60	0.98	-	-	-	-	-	-	-	-	-	-
This dataset is not from the UCI repository. Data references can be found in the appendix.
2 Time-series dataset from the UCR repository. EM, Fix, and GLM are the classification accuracy on the test data
on which the usefulness of predictors themselves may be assessed, i.e., a model selection criteria, resulting in a potentially very accurate and fast procedure for sparse model learning. An important open problem is the theoretical analysis of the expected number of EM iterations  $k$  that is required for convergence. The empirical evidence suggests that  $k$  converges to a constant, and is thus negligible in the asymptotic time complexity. This is in alignment with the convergence of the posterior to a multivariate normal distribution. However, such intuitive and empirical arguments cannot replace a rigorous worst-case analysis.
We thank the anonymous reviewers for their valuable feedback that led to a substantially improved theoretical analysis. This work was supported by the Australian Research Council (DP210100045).
[1] Hirotugu Akaike. A new look at the statistical model identification. In Selected Papers of Hirotugu Akaike, pages 215-222. Springer, 1974.
[2] Mahdi Alkhamisi, Ghadban Khalaf, and Ghazi Shukur. Some modifications for choosing ridge parameters. Communications in Statistics-Theory and Methods, 35(11):2005-2020, 2006.
[3] David M Allen. The relationship between variable selection and data augmentation and a method for prediction. technometrics, 16(1):125-127, 1974.
[4] Sylvain Arlot and Alain Celisse. A survey of cross-validation procedures for model selection. 2010.
[5] Arthur Asuncion and David Newman. UCI machine learning repository, 2007.
[43] William E Strawderman. Proper Bayes minimax estimators of the multivariate normal mean. The Annals of Mathematical Statistics, 42(1):385-388, 1971.
[44] Shu Yu Tew, Daniel F Schmidt, and Enes Makalic. Sparse horseshoe estimation via expectation-maximisation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 123-139. Springer, 2022.
[45] Aad W Van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
[46] M Varah, James. On the numerical solution of ill-conditioned linear systems with applications to ill-posed problems. SIAM Journal on Numerical Analysis, 10(2):257–267, 1973.
[47] Boxiang Wang and Hui Zou. Honest leave-one-out cross-validation for estimating post-tuning generalization error. Stat, 10(1):e413, 2021.
[48] ASAR Yasin, Adnan Karaibrahimoğlu, and GENÇ Aşır. Modified ridge regression parameters: A comparative Monte Carlo study. *Hacettepe Journal of Mathematics and Statistics*, 43(5): 827-841, 2013.
[49] Yongli Zhang and Yuhong Yang. Cross-validation for selecting a model selection procedure. Journal of Econometrics, 187(1):95-112, 2015.
A EM Procedure for Multiple Means
We show that the Bayesian EM procedure provides sensible estimates of the regularization parameter even in the setting of the normal multiple means problem with known variance  $\sigma^2$ . In this setting, the LOOCV is unable to provide any guidance on how to choose  $\lambda$  due to all the information for each regression parameter being concentrated in a single observation. We use the  $\tau$  parameterisation of the hyperparameter, rather than  $\tau^2$ , as the resulting estimator has an easy to analyse form.
In the normal multiple means model, we are given  $(y_{i}|\beta_{i})\sim N(\beta_{i},1)$ , i.e.,  $\mathbf{y}$  is a  $p$ -dimensional normally distributed vector with mean  $\beta$  and identity covariance matrix. The conditional posterior distribution of  $\beta$  is:
$$
\beta | \mathbf {y}, \tau \sim \mathrm {N} ((1 - \kappa) \mathrm {y}, \sigma^ {2} (1 - \kappa)) \tag {17}
$$
where  $\kappa = 1 / (1 + \tau^2)$ . Under this setting, Strawderman [43] proved that if  $p\geq 3$ , then any estimator of the form