SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Exploiting symmetry. Let us now exploit symmetry to avoid using non-diagonal matrices in our ellipsoids. We use the notion of axis-aligned sets in the next few results. A set  $\mathcal{X} \subseteq \mathbb{R}^d$  is axis-aligned if for any point  $\mathbf{p} \in \mathcal{X}$ , the reflections of  $\mathbf{p}$  along the axes are also contained in  $\mathcal{X}$ . Formally, for any  $\mathbf{s} \in \{\pm 1\}^d$ , we have that if  $\mathbf{p} \in \mathcal{X}$ , then  $\mathrm{Diag}(\mathbf{s})\mathbf{p} \in \mathcal{X}$ . Furthermore, with a slight abuse of notation define  $\mathrm{Diag}(\mathbf{A}) \coloneqq \mathrm{Diag}(\mathrm{diag}(\mathbf{A}))$ . That is,  $\mathrm{Diag}(\mathbf{A})$  is the diagonal matrix whose diagonal entries match those of  $\mathbf{A}$ . The idea is that the set  $\{\mathbf{p} \in \mathbb{R}_{\geq 0}^d : \mathrm{Diag}(\mathbf{p})$  is valid\} of diagonals of valid preconditioners is contained in the non-negative orthant. Yet, we can extend it by reflecting it over each of the axes. Although this may seem counter-intuitive, this translates the structure of our problem
into symmetry among all orthant, and this can be exploited elegantly. Formally, the set of diagonals of valid preconditioners reflected over each axis is given by set
$$
\mathcal {P} := \left\{\mathbf {p} \in \mathbb {R} ^ {d}: \operatorname {D i a g} (| \mathbf {p} |) \text {i s v a l i d} \right\},
$$
where  $|\mathbf{p}|$  is the entry-wise absolute value of  $\mathbf{p}\in \mathbb{R}^d$ . The following lemma shows that when looking for low volume ellipsoids that contain an axis-aligned set, we can restrict out attention to axis-aligned ellipsoids, defined by a diagonal matrix. The following lemma can be seen as the polar statement of Goemans et al. (2009, Proposition 3.1), with the benefit of not requiring any matrix inversions.
Lemma D.5. Let  $\mathcal{X} \subset \mathbb{R}^d$  be an axis-aligned convex set and let  $\mathbf{A} \in \mathbb{R}^{d \times d}$  be positive definite matrix such that  $\mathcal{X} \subseteq \mathcal{E}(\mathbf{A})$ . Then  $\mathcal{X} \subseteq \mathcal{E}(\mathrm{Diag}(\mathbf{A}))$  and  $\mathrm{Vol}(\mathcal{E}(\mathrm{Diag}(\mathbf{A}))) \leq \mathrm{Vol}(\mathcal{E}(\mathbf{A}))$ .
Proof. Let us start by showing that  $\mathcal{X} \subseteq \mathcal{E}(\mathrm{Diag}(\mathbf{A}))$ . We use the notation  $\mathrm{Diag}(\mathbf{v}) \cdot \mathcal{X}$  to denote the set  $\mathrm{Diag}(\mathbf{v}) \cdot \mathcal{X} \coloneqq \{\mathrm{Diag}(\mathbf{v}) \cdot \mathbf{x} : \mathbf{x} \in \mathcal{X}\}$ . Since  $\mathcal{X}$  is axis-aligned, we have
$$
\mathcal {X} = \operatorname {D i a g} (\mathbf {s}) \cdot \mathcal {X} \subseteq \operatorname {D i a g} (\mathbf {s}) \cdot \mathcal {E} (\mathbf {A}) = \mathcal {E} (\operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})), \quad \forall \mathbf {s} \in \{\pm 1 \} ^ {d}.
$$
Therefore,  $\mathcal{X}$  is contained in each of the  $2^{d}$  ellipsoids of the form  $\mathcal{E}(\mathrm{Diag}(\mathbf{s})\mathbf{A}\mathrm{Diag}(\mathbf{s}))$ . Thus,
$$
\mathcal {X} \subseteq \bigcap_ {\mathbf {s} \in \{\pm 1 \} ^ {d}} \mathcal {E} (\operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})) \subseteq \mathcal {E} \left(\frac {1}{2 ^ {d}} \sum_ {\mathbf {s} \in \{\pm 1 \} ^ {d}} \operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})\right),
$$
where the last inclusion follows since, for any set of positive definite matrices  $\mathcal{M}$ , one may verify that  $\cap_{\mathbf{M} \in \mathcal{M}} \mathcal{E}(\mathbf{M}) \subseteq \mathcal{E}((1/|\mathcal{M}|) \sum_{\mathbf{M} \in \mathcal{M}} \mathbf{M})$ . Finally, note that
$$
\sum_ {\mathbf {s} \in \{\pm 1 \} ^ {d}} \operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s}) = \operatorname {D i a g} (\mathbf {A}).
$$
Indeed, let  $i,j\in \{1,\dots ,d\}$ . If  $i = j$ , then  $(\mathrm{Diag}(\mathbf{s})\mathbf{A}\mathrm{Diag}(\mathbf{s}))_{i,j} = \mathbf{A}_{i,j}$  for any  $\mathbf{s}\in \{\pm 1\} ^d$ . If  $i\neq j$ , then
$$
\begin{array}{l} \sum_ {\mathbf {s} \in \{\pm 1 \} ^ {d}} (\operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})) _ {i, j} \\ = \sum_ {\mathbf {s} \in \{\pm 1 \} ^ {d}: \mathbf {s} [ i ] \neq \mathbf {s} [ j ]} (\operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})) _ {i, j} + \sum_ {\mathbf {s} \in \{\pm 1 \} ^ {d}: \mathbf {s} [ i ] = \mathbf {s} [ j ]} (\operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})) _ {i, j} \\ = 2 ^ {d - 1} \cdot (- \mathbf {A} _ {i, j}) + 2 ^ {d - 1} \cdot \mathbf {A} _ {i, j} = 0. \\ \end{array}
$$
Let us now show that  $\operatorname{Vol}(\mathcal{E}(\operatorname{Diag}(\mathbf{A}))) \leq \operatorname{Vol}(\mathcal{E}(\mathbf{A}))$ . Note that  $\log (\operatorname{Vol}(\mathcal{E}(\mathbf{A}))) = \log (\operatorname{Vol}(\mathcal{E}(\mathbf{I}))) - \frac{1}{2}\log \det(\mathbf{A})$ . Since  $\log \det(\cdot)$  is concave over positive definite matrices, we have
$$
\begin{array}{l} \log \det (\operatorname {Diag}(\mathbf{A})) = \log \det \Bigl(\frac{1}{2^{d}}\sum_{\mathbf{s}\in \{\pm 1\}^{d}}\operatorname {Diag}(\mathbf{s})\mathbf{A}\operatorname {Diag}(\mathbf{s})\Bigr) \\ \geq \frac {1}{2 ^ {d}} \sum_ {\mathbf {s} \in \{\pm 1 \} ^ {d}} \log \det  \left(\operatorname {D i a g} (\mathbf {s}) \mathbf {A} \operatorname {D i a g} (\mathbf {s})\right) = \frac {1}{2 ^ {d}} \cdot 2 ^ {d} \log \det  (\mathbf {A}) = \log \det  (\mathbf {A}). \\ \end{array}
$$
$$
\begin{array}{l} \log (\operatorname {V o l} (\mathcal {E} (\operatorname {D i a g} (\mathbf {A})))) = \log (\operatorname {V o l} (\mathcal {E} (\mathbf {I}))) - \frac {1}{2} \log \det (\operatorname {D i a g} (\mathbf {A})) \\ \leq \log (\operatorname {V o l} (\mathcal {E} (\mathbf {I}))) - \frac {1}{2} \log \det  (\mathbf {A}) \\ = \log (\operatorname {V o l} (\mathcal {E} (\mathbf {A}))), \\ \end{array}
$$
which implies that  $\mathrm{Vol}(\mathcal{E}(\mathrm{Diag}(\mathbf{A})))\leq \mathrm{Vol}(\mathcal{E}(\mathbf{A}))$
We are now in position to prove Lemma 5.2, which follows directly from the previous two results.
Proof of Lemma 5.2. By the assumptions in Proposition 4.2 we have that  $\mathbf{P} \coloneqq \mathrm{Diag}(\mathbf{p})$  fails the Armijo condition 4 condition and, thus,  $\mathbf{p} \in \mathcal{H}_{>}(\mathbf{u})$ . This together with the assumption that  $\| \mathbf{p} \|_{\mathbf{A}} \leq 1 / \sqrt{2d}$  imply via Lemma D.3 that  $\| \mathbf{u} \|_{\mathbf{A}^{-1}} \geq \sqrt{2d}$ . This allows us to use Theorem D.2 to find a new ellipsoid containing  $\mathcal{E}(\mathbf{a}) \cap \mathcal{H}_{\leq}(\mathbf{u})$  with the required volume decrease by Lemma D.4.
Yet, this ellipsoid may not be axis-aligned. We shall exploit the symmetry described in Lemma D.5 to show that the axis-aligned ellipsoid  $\mathcal{E}(\mathbf{a}^{+}(\mathbf{a},\mathbf{u}))$  enjoys the same guarantees.
Formally, we need  $\mathcal{E}(\mathbf{a}^{+}(\mathbf{a},\mathbf{u}))$  to contain  $\mathcal{E}(\mathbf{a})\cap \mathcal{H}_{\leq}(\mathbf{u})$ . Since  $\mathbf{u}\geq 0$ , we have
$$
\mathcal {H} _ {\leq} (\mathbf {u}) \subseteq \mathcal {S} (\mathbf {u}) := \left\{\mathbf {p} \in \mathbb {R}: \operatorname {D i a g} (\mathbf {s}) \cdot \mathbf {p} \in \mathcal {H} _ {\leq} (\mathbf {u}) \text {f o r a l l} \mathbf {s} \in \{\pm 1 \} ^ {d} \right\}.
$$
Thus, it suffices for  $\mathcal{E}(\mathbf{a}^{+}(\mathbf{a},\mathbf{u}))$  to contain  $\mathcal{E}(\mathbf{a})\cap \mathcal{S}(\mathbf{u})$ . From Theorem D.2 we know that  $\mathcal{E}(\mathbf{a})\cap \mathcal{S}(\mathbf{u})$  is contained in the ellipsoid given by the matrix  $\lambda \operatorname {Diag}(\mathbf{a}) + (1 - \lambda)\mathbf{uu}^{\mathrm{T}}$  for any  $\lambda$ , in particular for  $\lambda$  as in (20) since  $\| \mathbf{u}\|_{\mathbf{A}^{-1}} > \sqrt{d}$ . Since  $\mathcal{S}(\mathbf{u})$  is axis-aligned, we can exploit symmetry using Lemma D.5, which tells that  $\mathcal{E}(\mathbf{a})\cap \mathcal{S}(\mathbf{u})$  is contained in the ellipsoid given by the matrix
$$
\operatorname {D i a g} \left(\lambda \operatorname {D i a g} (\mathbf {a}) + (1 - \lambda) \mathbf {u u} ^ {\intercal}\right) = \operatorname {D i a g} (\mathbf {a} ^ {+} (\mathbf {a}, \mathbf {u})),
$$
as desired. Finally, the bound on the volume follows by Theorem D.2 and the bound on  $\nu_{d}(\mathbf{u})$  given by Lemma D.4 since  $\| \mathbf{u}\|_{\mathbf{A}^{-1}}\geq \sqrt{2d}$ .
Finally, we are in position to prove Theorem 5.3, which follows almost directly from Lemma 5.2.
Proof of Theorem 5.3. Note that (a) holds by induction and since, by Proposition 4.2, we have  $\mathrm{diag}(\mathbf{P}^*)\in \mathcal{H}_{\leq}(\mathbf{u}_t)$  for any  $\mathbf{u}_t$  used in a call to CUT. For (b), fix  $t\in \{1,\dots ,T\}$  and recall that in this case we have  $S_{t} = \{\operatorname {Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{E}(\mathbf{a}_{t})\}$ . As described in (8), one may verify that  $\mathrm{Diag}(\mathbf{q}_t^*)$  for  $\mathbf{q}_t^*$  given by
$$
\mathbf {q} _ {t} ^ {*} := \frac {1}{\| \nabla f (\mathbf {x} _ {t}) ^ {2} \| _ {\mathbf {A} _ {t} ^ {- 1}}} \cdot \mathbf {A} _ {t} ^ {- 1} \nabla f (\mathbf {x} _ {t}) ^ {2}
$$
maximizes  $\| \nabla f(\mathbf{x}_t)\|_{\mathbf{P}}$  for  $\mathbf{P}\in \mathcal{S}_t$ . Since
$$
\mathbf {P} _ {t} = \operatorname {C A N D I D A T E} \left(\mathcal {S} _ {t}, ^ {1} / \sqrt {2 d}, \mathbf {x} _ {t}\right) = \frac {1}{\sqrt {2 d}} \operatorname {D i a g} \left(\mathbf {q} _ {t} ^ {*}\right),
$$
we conclude that  $\mathbf{P}_t$  is  $1 / \sqrt{2d}$ -competitive. For (c), first note that we may assume  $(1 / L)\mathbb{1} \in \mathcal{E}(\alpha_0I)$ . To see that, assume  $(1 / L)\mathbb{1} \notin \mathcal{E}(\alpha_0I)$ , implying  $\alpha_0d > L^2$ . In this case, any candidate preconditioner computed by CANDIDATE is always valid and, thus, we never call CUT. To see this, let  $\mathbf{A}_0 \coloneqq \alpha_0\mathbf{I}$  be the matrix defining the initial ellipsoid. Then, by the definition of CANDIDATE for ellipsoids we have that  $\mathbf{P}_0 = \mathrm{Diag}(\mathbf{p}_0)$  is such that
$$
\left\| \mathbf {p} _ {0} \right\| _ {\mathbf {A} _ {0}} ^ {2} = \alpha_ {0} \left\| \mathbf {p} _ {0} \right\| ^ {2} = \frac {1}{2 d} <   \frac {1}{2} \frac {\alpha_ {0}}{L ^ {2}}.
$$
Therefore,  $\mathbf{p}_0[i] \leq 1 / L$  for all  $i \in \{1, \dots, d\}$ , which implies that  $\mathbf{P}_0$  is valid since  $\mathbf{P}_0 \preceq \frac{1}{L} \mathbf{I}$ .
Let us look now at the case  $(1 / L)\mathbb{1}\in \mathcal{E}(\alpha_0I)$ . Therefore,  $\mathcal{B}(^{1 / L}\mathbb{1})\subseteq \mathcal{E}(\mathbf{a}_t)$  for all iterations  $t$ . Since the minimum volume ellipsoid containing the box  $\mathcal{B}((1 / L)\mathbb{1})$  is the unit sphere of radius  $1 / L$ , that is,  $\mathcal{E}((L^2 /d)\mathbb{1})$ . Therefore,  $\mathrm{Vol}(\mathcal{E}(\mathbf{a}_t))\geq \mathrm{Vol}(\mathcal{E}((L^2 /d)\mathbb{1}))$ . Moreover, every time we call cut the volume of the ellipsoid goes down by  $1 / c\coloneqq \sqrt[4]{e} /\sqrt{2}$ . Therefore, the total number of calls to CUT is no more than
$$
\log_ {c} \bigg (\frac {\mathrm {V o l} (\mathcal {E} (\alpha_ {0} \mathbb {1}))}{\mathrm {V o l} (\mathcal {E} ((L ^ {2} / d) \mathbb {1}))} \bigg) = \log_ {c} \bigg (\frac {L ^ {d}}{d \alpha_ {0} ^ {d / 2}} \bigg) \leq \frac {d}{\log (c)} \log \bigg (\frac {L}{d \alpha_ {0}} \bigg) \leq 1 2 d \log \bigg (\frac {L}{\alpha_ {0}} \bigg)
$$
since  $\log (c)\geq 1 / 12$
Refining the choice of  $\lambda$ . Although we have shown in Lemma 5.2 a choice  $\lambda$  that guarantees volume decrease, it may be sub-optimal. The choice of  $\lambda$  in Equation (20) is inherited from the non-symmetric case in Theorem D.2. Although Lemma 5.2 and Theorem D.2 match when  $\mathbf{u}$  has only one non-zero entry, we should expect better choices of  $\lambda$ , leading to more volume shrinkage, to be possible in Lemma 5.2. Although we have not found a choice of  $\lambda$  that is dependent on  $\mathbf{u}$  that generically improves upon (20), in practice we can solve for a better  $\lambda$  numerically, by directly minimizing the volume of the resulting ellipsoid,
$$
\min  _ {0 <   \lambda <   1} \operatorname {V o l} (\mathcal {E} (\lambda \mathbf {a} + (1 - \lambda) \operatorname {D i a g} (\mathbf {u u} ^ {\top}))) = \min  _ {0 <   \lambda <   1} - \sum_ {i} \log (\lambda \mathbf {a} [ i ] + (1 - \lambda) \mathbf {u} [ i ] ^ {2}).
$$
As the problem is one-dimensional, numerical solvers can often find near-optimal solutions. By warm-starting a numerical solver with the  $\lambda$  defined in (20), we can guarantee that the resulting ellipsoid leads to a smaller volume and we do not lose our worst-case theoretical guarantees.
Connection to the polar problem and related work. Our results have an interesting connection to some of the results from Goemans et al. (2009), via the use of polarity theory. Here we give a quick overview of their work and the connection to our cutting plane methods. Goemans et al. (2009) shows techniques to approximate some polyhedron  $\mathcal{P} \subseteq \mathbb{R}^d$  (a polymatroid being one of the main examples) from inside by some ellipsoid  $\mathcal{E}(\mathbf{A})$ . Their algorithm maintains an ellipsoid  $\mathcal{E}(\mathbf{A}) \subseteq \mathcal{P}$  and tries to iteratively enlarge it. They assume access to an oracle such that, at each iteration, either finds a point  $\mathbf{u} \in \mathbf{P}$  that is sufficiently far from  $\mathcal{E}(\mathbf{A})$ , meaning  $\| \mathbf{u} \|_{\mathbf{A}} > \sqrt{d} + \epsilon$  for some  $\epsilon > 0$ , or guarantees that  $\mathcal{E}(\mathbf{A})$  "approximates well"  $\mathcal{P}$  from inside in the sense that  $\| \mathbf{u} \|_{\mathbf{A}} \leq (\sqrt{n} + \epsilon) / \alpha$  for all  $\mathbf{u} \in \mathcal{P}$ , where  $\alpha > 0$  is some approximation factor. In their algorithm, when the oracle finds a point  $\mathbf{u} \in \mathcal{P}$  such that  $\| \mathbf{u} \|_{\mathbf{A}} > \sqrt{d} + \epsilon$  the algorithm needs to find an ellipsoid  $\mathcal{E}(\mathbf{A}^{+})$  such that
$$
\mathcal {E} \left(\mathbf {A} ^ {+}\right) \subseteq \operatorname {c o n v} \left(\mathcal {E} (\mathbf {A}) \cup \{\mathbf {u}, - \mathbf {u} \}\right), \tag {22}
$$
where  $\operatorname{conv}(\mathcal{D})$  is the convex hull of  $\mathcal{D}$ . Interestingly, the polar problem is exactly what we need for out cutting plane method. More precisely, the polar set  $\mathcal{X}^*$  of a set  $\mathcal{X}$  is given by  $\mathcal{X}^* := \{z \in \mathbb{R}^d : \langle z, x \rangle \leq 1\}$ . Then, by taking polars and using that  $\mathcal{E}(\mathbf{A})^* = \mathcal{E}(\mathbf{A}^{-1})$ , we have that  $\mathcal{P}^* \subseteq \mathcal{E}(\mathbf{A}^{-1})$ . Moreover, taking polar on both sides of (22) yields that an equivalent problem is finding  $(\mathbf{A}^+)^{-1}$  such that
$$
\mathcal {E} \left(\left(\mathbf {A} ^ {+}\right) ^ {- 1}\right) \supseteq \mathcal {E} \left(\mathbf {A} ^ {- 1}\right) \cap \left\{- \mathbf {u}, \mathbf {u} \right\} ^ {*} = \mathcal {E} \left(\mathbf {A} ^ {- 1}\right) \cap \left\{\mathbf {z}: | \langle \mathbf {u}, \mathbf {z} \rangle | \leq 1 \right\}.
$$
That is, the problem is the one of finding a smaller ellipsoid  $\mathcal{E}((\mathbf{A}^{+})^{-1})$  that contains  $\mathcal{E}(\mathbf{A}^{-1}) \cap \{\mathbf{z}: |\langle \mathbf{u}, \mathbf{z} \rangle| \leq 1\}$ , which is broadly the goal of the subroutine CUT.
Table 1: Datasets used in our experiments, including number of samples  $n$  and dimension  $d$  ,and order of magnitude of the condition number of the regularized system  $\left( {\kappa \left( {{\mathbf{X}}^{\top }\mathbf{X} + 1/n\mathbf{I}}\right) }\right)$  and condition number of the system when using the optimal diagonal preconditioner,  ${\kappa }_{ * }$  .
Dataset	Repository/Source	n	d	κ	κ*
cpusmall	LIBSVM, Delve (comp-activ)	8192	12	1013	102
california-housing	Scikit/StatLib, Kelley Pace and Barry (1997)	20640	8	1010	104
power-plant	UCI, Tüfekci (2014)	9568	4	109	104
concrete	UCI, Yeh (1998)	1030	8	109	103
mg	LIBSVM, Flake and Lawrence (2002)	1385	6	103	103
breast-cancer	UCI	569	32	1013	102
australian	LIBSVM, Statlog	690	14	109	102
heart	LIBSVM, Statlog	270	13	107	102
diabetes	UCI	768	8	106	102
ionosphere	UCI	351	34	103	102
news20	LIBSVM, Keerthi and DeCoste (2005)	19996	1355191	1013	NA
rcv1	LIBSVM, Lewis et al. (2004)	20242	47236	1013	NA
Objective functions
We use  $L_{2}$ -regularized linear regression  $\mathcal{L}_{\mathrm{LINEAR}}$  and  $L_{2}$ -regularized logistic regression  $\mathcal{L}_{\mathrm{LOGISTIC}}(\mathbf{w})$  with a regularization coefficient of 1. Given a data matrix  $\mathbf{X} \in \mathbb{R}^{n \times d}$ , target  $y \in \mathbb{R}^n$  for regression tasks and  $y \in \{0,1\}^n$  for classification tasks, and parameters  $\mathbf{w} \in \mathbb{R}^d$ ,
$$
\mathcal {L} _ {\text {L I N E A R}} (\mathbf {w}) = \frac {1}{n} \left(\frac {1}{2} \| \mathbf {X} \mathbf {w} - \mathbf {y} \| ^ {2} + \frac {1}{2} \| \mathbf {w} \| ^ {2}\right).
$$
$$
\mathcal {L} _ {\mathrm {L O G I S T I C}} (\mathbf {w}) = \frac {1}{n} \sum_ {i = 1} ^ {n} - \mathbf {y} [ i ] \log (\sigma (\langle \mathbf {x} _ {i}, \mathbf {w} \rangle)) - (1 - \mathbf {y} [ i ]) \log (1 - \sigma (\langle \mathbf {x} _ {i}, \mathbf {w} \rangle)) + \frac {1}{n} \frac {1}{2} \| \mathbf {w} \| ^ {2}.
$$
where  $\mathbf{x}_i$  is the  $i$ th row of  $\mathbf{X}$  and  $\sigma$  is the sigmoid function,  $\sigma(z) = 1/1 + \exp(-z)$ . For all datasets, we add a bias term by preponding a feature column of ones to  $\mathbf{X}$ .
We use the datasets listed in Table 1, made available by LIBSVM (Chang and Lin, 2011), Scikit-Learn (Pedregosa et al., 2011) and the UCI repository (Dua and Graff, 2017).
Data rescaling
We do not rescale, standardize or otherwise change any of the datasets beyond adding a bias term, as our goal is to check whether preconditioned methods can handle badly scaled data.
We consider two types of initializations. The first approximates a "best-case" scenario where we start from an estimate with a reasonable loss value despite the bad scaling of the data. We set  $\mathbf{w}[i] = 0$  except for the bias term  $\mathbf{w}[0]$  which is set at the MLE of the non-regularized problem,
$$
\mathrm {w h e r e} \bar {y} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {y} [ i ]
$$
$$
\begin{array}{l} \mathbf {w} [ 0 ] = \bar {y} \\ f o r \quad \text {l i n e a r} \\ \end{array}
$$
$$
\mathbf {w} [ 0 ] = \log \left(\frac {\bar {y}}{1 - \bar {y}}\right)
$$
$$
\text {w h e r e} \bar {y} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {y} [ i ]
$$
$$
f o r \quad \text {l o g i s t i c}
$$
The results in the main text use this initialization. The second initialization takes  $\mathbf{w} \sim \mathcal{N}(0, \mathbf{I})$ , giving a starting point with potentially large loss. We give results using both initializations in the appendix.
Optimizers used
- For the small linear regression problems, we use preconditioned gradient descent with the optimal preconditioner, pre-computed using the semidefinite formulation of Qu et al. (2022), solved using CVXPY (Diamond and Boyd, 2016) based on the Matlab implementation of Qu et al.
https://github.com/Gwzwpxz/opt_dpcond
- Gradient descent with a backtracking line-search with backtracking parameter  $\gamma = 1/2$ .
- RPROP (Riedmiller and Braun, 1993) following the implementation and default hyperparameters in PyTorch (Paszke et al., 2019) (starting step-size of  $10^{-1}$ , increase step-size factor  $\eta^{+} = 1.2$ , decreasing step-size factor  $\eta^{-} = 0.5$ , minimum step-size of  $10^{-6}$  and maximum step-size of 50).
https://github.com/pytorch/pytorch/blob/v2.0.1/torch/optim/rprop.py
- Hypergradient descent to set the step-size, using (S)GD-HD (the multiplicative variant, Baydin et al., 2018). The hypergradient step-size is set to the default  $\beta = 0.02$  (Baydin et al., 2018, footnote 3). The initial step-size is set to  $\alpha_0 = 10^{-10}$ , as otherwise most runs diverged immediately.
- The diagonal Barzilai-Borwein method of Park et al. (2020), using their non-monotonic line-search. We use the default parameters suggested; a starting step-size of  $10^{-6}$ , regularization factor on the previous diagonal approximation  $\mu = 10^{-6}$ , a backtracking factor of  $1/2$  for the backtracking line-search and a window of 15 steps for the non-monotone line-search. This line-search does not use a forward step as the update can increase the preconditioner.
- Preconditioned gradient descent using the diagonal Hessian, with a backtracking line-search.
- AdaGrad (Duchi et al., 2011) but augmented with a backtracking line-search as suggested by Vaswani et al. (2020) to make it competitive in the deterministic setting, following the PyTorch (Paszke et al., 2019) implementation.
https://github.com/pytorch/pytorch/blob/v2.0.1/torch/optim/adagrad.py
Line-search and forward steps
For all methods, the backtracking line-search is augmented by a forward step. When a step-size is accepted, it is increased by a factor of 1.1 for the next step. For multidimensional backtracking, we increase the set uniformly, taking  $\mathbf{b}' = 1.1 \cdot \mathbf{b}$  for the box and  $\mathbf{a}' = \mathbf{a} / \sqrt{1.1}$  for the ellipsoid. The ellipsoid uses a slightly smaller increase factor. $^7$
Hyperparameters for the line-search and multidimensional backtracking
For the backtracking line-searches used in gradient descent, preconditioned gradient descent and used to augment the other algorithms, we start the search at an initial step-size of  $10^{10}$  and backtrack by a factor of  $1/2$  when failing the Armijo condition, implemented generically as
$$
f (\mathbf {x} - \mathbf {d}) \leq f (\mathbf {x}) - \frac {1}{2} \langle \nabla f (\mathbf {x}), \mathbf {d} \rangle
$$
For multidimensional backtracking, we initialize the sets such that the first preconditioner is on the order of  $10^{10}\mathbf{I}$ . Using the notation of Appendix A, we use the scaling factor  $c_{0} = d\cdot 10^{10}$  for the box variant and  $c_{0} = \sqrt{d}\cdot 10^{10}$  for the ellipsoid variant. The first preconditioner tried by the box variant with backtracking factor  $\gamma = 1 / 2d$  is then  $1 / 2\cdot 10^{10}\mathbf{I}$ , and the first preconditioner tried by the ellipsoid variant (assuming the gradient is uniform,  $\nabla f(\mathbf{x}_0)\propto \mathbf{1})$  is  $1 / \sqrt{2}\cdot 10^{10}\mathbf{I}$ .
To increase by a factor of 1.1 in the one-dimensional case, the update to the ellipsoid should be  $\mathbf{a}' = \mathbf{a} / 1.1^2$ .
Table 2: Running times for parts of a backtracking update using the Ellipsoid variant (Figure 11) on RCV1. Average runtime over 100 calls,  $\pm$  standard deviations over 10 repeats.
Operation	Average runtime	± std
Compute gradient, preconditioner and next iterate	24.4 ms	±0.2 ms
Compute hypergradient	12.6 ms	±0.1 ms
Compute CUT (using Lemma 5.2)	0.9 ms	±0.1 ms
Compute CUT (solving the convex combination with scipy.optimize)	7.6 ms	±0.1 ms
E.1 Performance comparison
The algorithms we compare do not have a unified notion of an "iteration". For example, a backtracking line-search and multidimensional backtracking do more work than plain gradient descent in between updates to the parameters. To make the performance comparison fair, the results in Figures 1 and 5 account for the overhead of backtracking by comparing the performance against the number of oracle calls, counting the number function and gradient evaluations.