SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(1 - r \left(\frac {1}{2} | | y | | ^ {2}\right) \frac {p - 2}{| | y | | ^ {2}}\right) y \tag {18}
$$
where  $0 \leq r\left(\frac{1}{2}||y||^2\right) \leq 2$  and  $r(\cdot)$  is non-decreasing, is minimax, i.e., it dominates least-squares. We will now show that our EM procedure not only yields reasonable estimates in this setting, in contrast to LOOCV, but that these estimates are minimax, and hence dominate least-squares.
For the normal means model, we can obtain a closed form solution for the optimum  $\tau$ , by solving for the stationary point for which  $\tau_{t + 1} = \tau_t$ , with  $\tau \sim C^{+}(0,1)$ :
$$
\begin{array}{l} \underset {\tau} {\arg \min } \left\{\mathrm {E} _ {\boldsymbol {\beta}} [ - \log p (y | \beta , \tau) - \log p (\beta | \tau) - \log \pi (\tau) ] \right\} = \tau \\ \arg \min _ {\tau} \left\{\frac {p}{2} \log \tau^ {2} + \frac {w}{2 \tau^ {2}} + \log (1 + \tau^ {2}) \right\} = \tau \\ \sqrt {\frac {w - p + \sqrt {p ^ {2} + 8 w + 2 p w + w ^ {2}}}{2 (2 + p)}} = \tau , \\ \end{array}
$$
and with  $w = \sum_{j=1}^{p} \operatorname{E}\left[\beta_j^2\right] = (1 - \kappa)^2 s + (1 - \kappa)p$ ,  $s = ||y||^2$  and  $\tau = \sqrt{\frac{1 - \kappa}{\kappa}}$ . This yields
$$
\sqrt {\frac {(\sqrt {p ((\kappa - 2) ^ {2} p - 8 \kappa + 8) - 2 (\kappa - 1) ^ {2} s ((\kappa - 2) p - 4) + (\kappa - 1) ^ {4} s ^ {2}} - \kappa p + (\kappa - 1) ^ {2} s)}{2 (2 + p)}} = \sqrt {\frac {1 - \kappa}{\kappa}}
$$
with solution  $\kappa = (p + 2) / s$ . Plugging this  $\kappa$  solution into (17), we note that the resulting estimator of  $\beta$  (17) is of the form (18) with
$$
\begin{array}{l} r \left(\frac {1}{2} | | y | | ^ {2}\right) = \left(\frac {p + 2}{| | y | | ^ {2}}\right) / \left(\frac {p - 2}{| | y | | ^ {2}}\right) \\ = \frac {p + 2}{p - 2} \\ \end{array}
$$
As we have  $r\left(\frac{1}{2}\| y\| ^2\right)\leq 2$  when  $p\geq 6$ , the EM ridge estimator is minimax in this setting for  $p\geq 6$ .
We prove that for sufficiently large  $n$ , a continuous injective reparameterization of the negative log joint posterior of (5) & (7) is convex when restricted to  $\tau^2 \geq \epsilon$ . This is sufficient, since unimodality is preserved by strictly monotone transformations and continuous injective reparameterizations.
Specifically, for the presented hierarchical model, the negative log joint posterior up to an additive constant is
$$
\frac {n + p + 2}{2} \log \sigma^ {2} + \frac {1}{2 \sigma^ {2}} \| \mathbf {y} - \mathbf {X} \boldsymbol {\beta} \| ^ {2} + \frac {p + 2 - 2 a}{2} \log \tau^ {2} + \frac {\| \boldsymbol {\beta} \| ^ {2}}{2 \sigma^ {2} \tau^ {2}} + (a + b) \log (1 + \tau^ {2})
$$
and reparameterising with  $\phi = \beta /\sigma$ ,  $\rho = 1 / \sigma$  and  $\chi = 1 / \tau$  and reorganising terms yields
$$
- (n + p + 2) \log \rho - (p + 2 - 2 a) \log \chi + (a + b) \log (1 + \chi^ {- 2}) + \frac {1}{2} \| \rho \mathbf {y} - \mathbf {X} \phi \| ^ {2} + \frac {\| \chi \phi \| ^ {2}}{2}.
$$
The first three terms can easily be checked to be convex via a second derivative test, for which the convexity of the second term is contingent on the condition that  $a < 1 + p / 2$ , a condition that holds true in our specific scenario with  $a = b = 1 / 2$ . For the last two terms, the combined Hessian is of block form  $[\mathbf{A}, \mathbf{B}; \mathbf{B}^{\mathrm{T}}, \mathbf{C}]$  with  $\mathbf{A} = \mathbf{X}^{\mathrm{T}}\mathbf{X} + \chi^{2}\mathbf{I}_{p}$ ,  $\mathbf{B} = 2\chi \phi$ , and  $\mathbf{C} = \| \phi \|^{2}$ . Symmetric matrices of this form are positive definite if  $\mathbf{A}$  and its Schur complement
$$
\mathbf {C} - \mathbf {B} ^ {\mathrm {T}} \mathbf {A} ^ {- 1} \mathbf {B} = \| \phi \| ^ {2} - 4 \phi^ {\mathrm {T}} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} / \chi^ {2} + \mathbf {I} _ {p}\right) ^ {- 1} \phi
$$
are positive definite. Clearly,  $\mathbf{A}$  is positive definite. Moreover, for  $n > 4 / (\epsilon^2\gamma_n)$ , we have
$$
\begin{array}{l} \phi^ {\mathrm {T}} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} / \chi^ {2} + I\right) ^ {- 1} \phi = \phi^ {\mathrm {T}} \left(\mathbf {V} \boldsymbol {\Sigma} ^ {\mathrm {T}} \boldsymbol {\Sigma} \mathbf {V} ^ {\mathrm {T}} / \chi^ {2} + I\right) ^ {- 1} \phi \\ = \phi^ {\mathrm {T}} \mathbf {V} \left(\boldsymbol {\Sigma} ^ {\mathrm {T}} \boldsymbol {\Sigma} / \chi^ {2} + I\right) ^ {- 1} \mathbf {V} ^ {\mathrm {T}} \phi \\ \leq \phi^ {\mathrm {T}} \mathbf {V} \mathbf {V} ^ {\mathrm {T}} \phi \chi^ {2} / (n \gamma_ {n}) \\ = \left(\left(\mathbf {I} - \mathbf {V V} ^ {\mathrm {T}}\right) \phi + \mathbf {V V} ^ {\mathrm {T}} \phi\right) ^ {\mathrm {T}} \mathbf {V V} ^ {\mathrm {T}} \phi \chi^ {2} / \left(n \gamma_ {n}\right) \\ = \left\| \mathbf {V} \mathbf {V} ^ {\mathrm {T}} \phi \right\| ^ {2} \chi^ {2} / (n \gamma_ {n}) \\ \leq \| \phi \| ^ {2} \chi^ {2} / (n \gamma_ {n}) \\ <   \| \phi \| ^ {2} / (\epsilon^ {2} n \gamma_ {n}) \\ <   \| \phi \| ^ {2} / 4 \\ \end{array}
$$
where we used the fact that  $\mathbf{V}\mathbf{V}^{\mathrm{T}}$  is the orthogonal projection onto the column space of  $\mathbf{V}$ . The overall inequality implies the required positivity of the Schur complement.
C Derivation of Equation 9
C.1 Derivation of ESN
Here we show that
$$
\begin{array}{l} \sum_ {j = 1} ^ {p} \mathrm {E} \left[ \beta_ {j} ^ {2} \mid \hat {\tau} ^ {(t)}, \hat {\sigma} ^ {2 ^ {(t)}} \right] = \operatorname {t r} \left(\operatorname {C o v} [ \boldsymbol {\beta} ]\right) + \sum_ {j = 1} ^ {p} \mathrm {E} [ \beta_ {j} ] ^ {2} \\ = \sigma^ {2} \mathrm {t r} (\mathbf {A} _ {\tau} ^ {- 1}) + \| \hat {\boldsymbol {\beta}} _ {\tau} \| ^ {2} \\ \end{array}
$$
This is a rather straightforward proof. We use the fact that given a random variable  $x$ ; the expected squared value of  $x$  is  $\operatorname{E}\left[x^2\right] = \operatorname{Var}[x] + \operatorname{E}[x]^2$ .
C.2 Derivation of ESS
$$
\mathrm {E} _ {\boldsymbol {\beta}} \left[ \left| \left| \mathbf {y} - \mathbf {X} \boldsymbol {\beta} \right| \right| ^ {2} \mid \hat {\tau} ^ {(t)}, \hat {\sigma} ^ {2 ^ {(t)}} \right] = \left| \left| \mathbf {y} - \mathbf {X} \operatorname {E} [ \boldsymbol {\beta} ] \right| \right| ^ {2} + \operatorname {t r} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} \operatorname {C o v} [ \boldsymbol {\beta} ]\right) \tag {19}
$$
We first provide an important fact on the quadratic forms of random variables in Lemma C.1 below:
Lemma C.1. Let  $\mathbf{b}$  be a  $p$ -dimensional random vector and  $\mathbf{A}$  be a  $p$ -dimensional symmetric matrix. If  $\operatorname{E}[\mathbf{b}] = \boldsymbol{\mu}$  and  $\operatorname{Var}(\mathbf{b}) = \boldsymbol{\Sigma}$ , then  $\operatorname{E}[\mathbf{b}^{\mathrm{T}}\mathbf{A}\mathbf{b}] = \operatorname{tr}(\mathbf{A}\boldsymbol{\Sigma}) + \boldsymbol{\mu}^{\mathrm{T}}\mathbf{A}\boldsymbol{\mu}$ .
Now, we expand the left-hand side of Equation 19:
$$
\begin{array}{l} \mathrm {E} _ {\boldsymbol {\beta}} \left[ \left| \left| \mathbf {y} - \mathbf {X} \boldsymbol {\beta} \right| \right| ^ {2} \right] = \mathrm {E} _ {\boldsymbol {\beta}} \left[ (\mathbf {y} - \mathbf {X} \boldsymbol {\beta}) ^ {T} (\mathbf {y} - \mathbf {X} \boldsymbol {\beta}) \right] \\ = \mathrm {E} _ {\beta} \left[ \mathbf {y} ^ {\mathrm {T}} \mathbf {y} - 2 \mathbf {y} ^ {\mathrm {T}} \mathbf {X} \boldsymbol {\beta} + \boldsymbol {\beta} ^ {\mathrm {T}} \mathbf {X} ^ {\mathrm {T}} \mathbf {X} \boldsymbol {\beta} \right] \\ = \mathbf {y} ^ {\mathrm {T}} \mathbf {y} - 2 \mathbf {y} ^ {\mathrm {T}} \mathbf {X} \operatorname {E} [ \boldsymbol {\beta} ] + \operatorname {E} \left[ \boldsymbol {\beta} ^ {\mathrm {T}} \mathbf {X} ^ {\mathrm {T}} \mathbf {X} \boldsymbol {\beta} \right] \tag {20} \\ \end{array}
$$
The use of lemma C.1 allows Equation 20 to be rewritten as
$$
\begin{array}{l} \mathrm {E} _ {\boldsymbol {\beta}} \left[ \left| \left| \mathbf {y} - \mathbf {X} \boldsymbol {\beta} \right| \right| ^ {2} \right] = \mathbf {y} ^ {\mathrm {T}} \mathbf {y} - 2 \mathbf {y} ^ {\mathrm {T}} \mathbf {X} \operatorname {E} [ \boldsymbol {\beta} ] + \operatorname {E} [ \boldsymbol {\beta} ] ^ {\mathrm {T}} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X}\right) \operatorname {E} [ \boldsymbol {\beta} ] + \operatorname {t r} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} \operatorname {C o v} [ \boldsymbol {\beta} ]\right) \\ = \left\| \mathbf {y} - \mathbf {X} E [ \beta ] \right\| ^ {2} + \operatorname {t r} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} \operatorname {C o v} [ \beta ]\right) \\ = \left\| \mathbf {y} - \mathbf {X} \hat {\boldsymbol {\beta}} _ {\tau} \right\| ^ {2} + \sigma^ {2} \operatorname {t r} \left(\mathbf {X} ^ {\mathrm {T}} \mathbf {X} \mathbf {A} _ {\tau} ^ {- 1}\right) \\ \end{array}
$$
D Solving for the parameter updates (Derivation of Equation 11)
Rather than solving a two-dimensional numerical optimization problem (10), we show that given a fixed  $\tau^2$ , we can find a closed formed solution for  $\sigma^2$ , and vice versa. To start off, we need to find the solution for  $\sigma^2$  as a function of  $\tau^2$ . First, find the negative logarithm of the joint probability distribution of hierarchy (5):
$$
\underset {\sigma^ {2}} {\arg \min } \left\{\mathrm {E} _ {\boldsymbol {\beta}} \left[ - \log p (y | X, \beta , \sigma^ {2}) - \log p (\beta | \tau^ {2}, \sigma^ {2}) - \log p (\sigma^ {2}) - \log \pi (\tau^ {2}) \right] \right\}. \tag {21}
$$
Dropping terms that do not depend on  $\sigma^2$  yields:
$$
\begin{array}{l} \operatorname *{arg  min}_{\sigma^{2}}\left\{\operatorname{E}_{\beta}\bigl[ - \log p(y|X,\beta ,\sigma^{2}) - \log p(\beta |\tau^{2},\sigma^{2}) - \log p(\sigma^{2})\bigr] \right\} \\ = \arg \min  _ {\sigma^ {2}} \left\{\left(\frac {n + p}{2}\right) \log \sigma^ {2} + \frac {\operatorname {E S S}}{2 \sigma^ {2}} + \frac {\operatorname {E S N}}{2 \sigma^ {2} \tau^ {2}} + \log \sigma^ {2} \right\} \\ = \arg \min  _ {\sigma^ {2}} \left\{\left(\frac {n + p + 2}{2}\right) \log \sigma^ {2} + \frac {\text {E S S}}{2 \sigma^ {2}} + \frac {\text {E S N}}{2 \sigma^ {2} \tau^ {2}} \right\}. \tag {22} \\ \end{array}
$$
Solving the above minimization problem involves differentiating the negative logarithm with respect to  $\sigma^2$  and solving for  $\sigma^2$  that set the derivative to zero. This gives us:
$$
\begin{array}{l} \frac {\partial}{\partial \sigma^ {2}} \left\{\left(\frac {n + p + 2}{2}\right) \log \sigma^ {2} + \frac {\operatorname {E S S}}{2 \sigma^ {2}} + \frac {\operatorname {E S N}}{2 \sigma^ {2} \tau^ {2}} \right\} = 0 \\ \frac {2 + n + p}{2 \sigma^ {2}} - \frac {\mathrm {E S S}}{2 (\sigma^ {2}) ^ {2}} - \frac {\mathrm {E S N}}{2 (\sigma^ {2}) ^ {2} \tau^ {2}} = 0 \\ \hat {\sigma} ^ {2} = \frac {\tau^ {2} \mathrm {E S S} + \mathrm {E S N}}{(n + p + 2) \tau^ {2}} \tag {23} \\ \end{array}
$$
Next, to obtain the M-step updates for the shrinkage parameter  $\tau^2$ , we repeat the same procedure - find the negative logarithm of the joint probability distribution and remove terms that do not depend on either  $\sigma^2$  or  $\tau^2$ :
$$
\begin{array}{l} \underset {\tau^ {2}} {\arg \min } \left\{\mathrm {E} _ {\boldsymbol {\beta}} \left[ - \log p (y | X, \boldsymbol {\beta}, \sigma^ {2}) - \log p (\beta | \tau^ {2}, \sigma^ {2}) - \log p (\sigma^ {2}) - \log \pi (\tau^ {2}) \right] \right\} \\ = \arg \min  _ {\tau^ {2}} \left\{\left(\frac {n + p + 2}{2}\right) \log \sigma^ {2} + \frac {\text {E S S}}{2 \sigma^ {2}} + \frac {\text {E S N}}{2 \sigma^ {2} \tau^ {2}} + \frac {p}{2} \log \tau^ {2} + \log (1 + \tau^ {2}) + \frac {\log \tau^ {2}}{2} \right\} \tag {24} \\ \end{array}
$$
Substituting the solution for  $\sigma^2$  (23) into equation (24), yields a Q-function that depends only on  $\tau^2$ . We eliminate the dependency on  $\sigma^2$  by finding the optimal  $\sigma^2$  as a function of  $\tau^2$  and substitute it into the Q-function of (24):
$$
\left. \arg \min  _ {\tau^ {2}} \left\{\frac {1}{2} \left[ (1 + p) \log \tau^ {2} + 2 \log \left(1 + \tau^ {2}\right) + (n + p + 2) \left(1 + \log \left(\frac {\mathrm {E S N} + \tau^ {2} \mathrm {E S S}}{(n + p + 2) \tau^ {2}}\right)\right) \right] \right\} \right. \tag {25}
$$
Differentiating (25) with respect to  $\tau^2$  and solving for the  $\tau^2$  that set the derivative to zero yields:
$$
\begin{array}{l} \left. \frac {\partial}{\partial \tau^ {2}} \left\{\frac {1}{2} \left[ (1 + p) \log \tau^ {2} + 2 \log (1 + \tau^ {2}) + (n + p + 2) \left(1 + \log \left(\frac {\mathrm {E S N} + \tau^ {2} \mathrm {E S S}}{(n + p + 2) \tau^ {2}}\right)\right) \right] \right\} = 0 \right. \\ \frac {\left(3 \operatorname {E S S} + \operatorname {E S S} p\right) \left(\tau^ {2}\right) ^ {2} + \left(\operatorname {E S N} - \operatorname {E S N} n + \operatorname {E S S} + \operatorname {E S S} p\right) \tau^ {2} - \operatorname {E S N} - \operatorname {E S N} n}{2 \tau^ {2} \left(1 + \tau^ {2}\right) \left(\operatorname {E S N} + \tau^ {2} \operatorname {E S S}\right)} = 0. \tag {26} \\ \end{array}
$$
The  $\tau^2$  update is the positive solution to the quadratic equation (in terms of  $\tau^2$ ) (26):
$$
\hat {\tau} ^ {2} = \frac {(n - 1) \mathrm {E S N} - (1 + p) \mathrm {E S S} + \sqrt {(4 n + 4) \mathrm {E S N} (3 + p) \mathrm {E S S} + ((1 - n) \mathrm {E S N} + (p + 1) \mathrm {E S S}) ^ {2}}}{(6 + 2 p) \mathrm {E S S}}
$$
Table 3: Pseudo-code of EM algorithm with complexity of individual steps.
EM Algorithm with SVD	Operations
Input: Standardised predictors X ∈ Rn×p, centered targets y ∈ Rn and convergence threshold ε > 0
Output: β ∈ Rp
r = min(n,p)	O(1)
IF p ≥ n
[U,Σ, V] = svd(X)	O(mr2)
s2 = (Σ1,12, ..., Σr,r2)	O(r)
c = (UTy) ⊙ s	O(nr)
ELSE
[V, Σ2] = eigen(XT X)	O(mr2)
s2 = (Σ1,12, ..., Σr,r2)	O(r)
c = VTXTy	O(np)
Y = yTy	O(n)
τ2 ← 1	O(1)
σ2 ← (1/n) ∑i=1n (yi - y)2, y = (1/n) ∑i=1n yi	O(n)
RSS ← ∞	O(1)
DO
RSSold ← RSS	O(k)
αj ← cj/s2j + 1/τ2	O(kr)
(E-step)
ESN ← ∑j=1r αj2 + σ2(∑j=1r 1/s2j + τ-2 + τ2 max(p-n,0))	O(kr)
RSS ← Y - 2 ∑j=1r αjcj + ∑j=1r αjs2	O(kr)
ESS ← RSS + σ2(∑j=1r sj/s2j + τ-2)	O(kr)
(M-step)
g ← (4n+4)ESN (3+p)ESS + ((1-n)ESN + (p+1)ESS)2	O(k)
τ2 ← (n-1)ESN - (1+p)ESS + √g/(6+2p)ESS	O(k)
σ2 ← τ2ESS + ESN/(n+p+2)τ2	O(k)
δ ← |RSSold - RSS|/(1 + |RSS|)	O(k)
until δ < ε
αj ← cj/s2j + 1/τ2	O(kr)
β = Vα	O(pr)
return β
Table 4: Pseudocode of the fast LOOCV algorithm with complexity of individual steps.  $\mathbf{R}$  has column vectors  $\mathbf{r}_j$  for  $1\leq j\leq r$ .
Fast LOOCV ridge with SVD	Operation
Input: Standardised predictors X ∈ Rn×p, centered targets y ∈ Rn and a grid of penalty parameters L = (λ1, λ2, ..., λl)
Output: β ∈ Rp
r = min(n, p)	O(1)
IF p ≥ n
[U, Σ, V] = svd(X)	O(mr2)
s = (Σ1,1, ..., Σr, r)	O(r)
R = (s1u1, ..., sru r)	O(nr)
c = (U^T y) ⊙ s	O(nr)
ELSE
[V, Σ^2] = eigen(X^T X)	O(mr^2)
s^2 = (Σ1,1^2, ..., Σr, r^2)	O(r)
R = XV	O(nrp)
c = RTy	O(nr)
U = (r1/s1, ..., rr/s_r)	O(nr)
for λ ∈ L {
hi = ∑j=1r(s^2_j/s^2_j+λ) u_{ij}^2, (i = 1, ..., n)	O(lnr)
αj = cj/s^2_j + λ	O(lr)
e = y - Rα	O(lnr)
CVE(λ) = 1/n ∑i=1n(e_i/1-h_i)^2	O(ln)
}
Find λ* = arg min{CVE(λ)}	O(l)
αj = cj/s^2_j + λ*	O(r)
β = Vα	O(pr)
return β
E Supplementary Results Material
E.1 Real Datasets Details
Table 5: Real datasets details
DATSETS	ABBREVIATION	n	p	TARGET VARIABLE	SOURCE
BUZZ IN SOCIAL MEDIA (TWITTER)	TWITTER	583250	77	mean number of active discussion	UCI
BLOG FEEDBACK	BLOG	60021	281	number of comments in the next 24 hours	UCI
RELATIVE LOCATION OF CT SLICES ON AXIAL AXIS	CT SLICES	53500	386	reference: Relative image location on axial axis	UCI
BUZZ IN SOCIAL MEDIA (TOM'S HARDWARE)	TOMSHW	28179	97	Mean Number of display	UCI
CONDITION-BASED MAINTENANCE OF NAVAL PROPULSION PLANTS	NPD - COM	11934	16	GT Compressor decay state coefficient	UCI
CONDITION-BASED MAINTENANCE OF NAVAL PROPULSION PLANTS	NPD - TUR	11934	16	GT Turbine decay state coefficient	UCI
PARKINSON'S TELEMONITORING	PT - MOTOR	5875	26	motor UPDRS score	UCI
PARKINSON'S TELEMONITORING	PT - TOTAL	5875	26	total UPDRS score	UCI
ABALONE	ABALONE	4177	8	Rings (age in years)	UCI
COMMUNITIES AND CRIME	CRIME	1994	128	ViolentCrimesPerPop	UCI
AIRFOIL SELF-NOISE	AIRFOIL	1503	6	Scaled sound pressure level (decibels)	UCI
STUDENT PERFORMANCE	STUDENT	649	33	final grade (with G1 & G2 removed)	UCI
CONCRETE COMPRESSIVE STRENGTH	CONCRETE	1030	9	Concrete compressive strength (MPa)	UCI
FOREST FIRES	F.FIRES	517	13	forest burned area (in ha)	UCI
BOSTON HOUSING	B.HOUSING	506	13	Median value of owner-occupied homes in $1000's	[21]
FACEBOOK METRICS	FACEBOOK	500	19	Total Interactions (with comment, like, and share columns removed)	UCI
DIABETES	DIABETES	442	10	quantitative measure of disease progression one year after baseline	[13]
REAL ESTATE VALUATION	R.ESTATE	414	7	house price of unit area	UCI
AUTO MPG	A.MPG	398	8	city-cycle fuel consumption in miles per gallon	UCI
YACHT HYDRODYNAMICS	YACHT	308	7	residuary resistance per unit weight of displacement	UCI
AUTOMOBILE	A.MOBILE	205	26	price	UCI
RAT EYE TISSUES	EYE	120	200	the expression level of TRIM32 gene	[39]
RIBOFLAVIN	RIBO	71	4088	Log-transformed riboflavin production rate	[9]
CROP	CROP	24000	3072	24 crop classes	UCR
ELECTRIC DEVICES	ELECD	16637	4096	7 electric devices	UCR
STARLIGHT CURVES	STARL	9236	7168	3 starlight curves	UCR
Figure 3: Comparison of predictive performance  $(R^2)$  of EM algorithm ( $x$ -axes) against CV with fixed grid ( $y$ -axes, top) and glmnet heuristic ( $y$ -axis, bottom). Columns correspond to the results of linear features (left), second-order features (middle), and third-order features (right). Negative values are capped at 0. Points skewing toward the bottom right indicate when our EM approach is giving better/same prediction performance as LOOCV (colored in green).