SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To see how to ensure a competitive ratio, consider the case where  $S$  contains diagonal preconditioners whose diagonals come from the box  $\mathcal{B}(\mathbf{b}) \coloneqq \{\mathbf{p} \in \mathbb{R}_{\geq 0}^{d} : \mathbf{p} \leq \mathbf{b}\}$ . To select a candidate preconditioner that is  $\gamma$ -competitive in  $S$ , we can backtrack from the largest vector in  $\mathcal{B}(\mathbf{b})$  by some constant  $\gamma < 1$ , and take  $\mathbf{P} = \gamma \operatorname{Diag}(\mathbf{b})$ . While a large  $\gamma$  leads to more progress when the step is accepted, we will see that we need a small  $\gamma$  to ensure the volume shrinks when the step is rejected.
We can obtain the convergence rate of Theorem 1.1 depending on  $\gamma$  and the optimal preconditioned condition number  $\kappa_{*}$  if we ensure  $\mathbf{P}_{*} \in S_{t}$  and that  $\mathbf{P}_{t}$  is  $\gamma$ -competitive for all  $t$ .
Proposition 3.2. Let  $\mathbf{P}_*$ ,  $\kappa_*$  be an optimal preconditioner and condition number for  $f$  (1). If the set  $S_t$  from the algorithm in Figure 2 contains  $\mathbf{P}_*$ , and  $\mathbf{P}_t \in S_t$  is  $\gamma$ -competitive (Definition 3.1), then
$$
f \left(\mathbf {x} _ {t + 1}\right) - f \left(\mathbf {x} _ {*}\right) \leq \left(1 - \frac {\gamma}{\kappa_ {*}}\right) \left(f \left(\mathbf {x} _ {t}\right) - f \left(\mathbf {x} _ {*}\right)\right)
$$
whenever the candidate step leads to sufficient progress and is accepted.
Proof. The proof relies on three inequalities. (a) The iterate  $\mathbf{x}_{t + 1}$  yields sufficient progress (Eq. 4), (b) any accepted preconditioner  $\mathbf{P}_t$  is  $\gamma$ -competitive in  $S_{t}$  and thus with  $\mathbf{P}_{*}$ , and (c)  $f$  is  $^{1} / \kappa_{*}$ -strongly convex in  $\| \cdot \|_{\mathbf{P}_*^{-1}}$ , which implies  $\kappa_{*}\frac{1}{2}\| \nabla f(\mathbf{x}_{t})\|_{\mathbf{P}_{*}}^{2}\geq f(\mathbf{x}_{t}) - f(\mathbf{x}_{*})$ . Combining those yields
$$
f \left(\mathbf {x} _ {t + 1}\right) \stackrel {(a)} {\leq} f \left(\mathbf {x} _ {t}\right) - \frac {1}{2} \| \nabla f \left(\mathbf {x} _ {t}\right) \| _ {\mathbf {P} _ {t}} ^ {2} \stackrel {(b)} {\leq} f \left(\mathbf {x} _ {t}\right) - \gamma \frac {1}{2} \| \nabla f \left(\mathbf {x} _ {t}\right) \| _ {\mathbf {P} _ {*}} ^ {2} \stackrel {(c)} {\leq} f \left(\mathbf {x} _ {t}\right) - \frac {\gamma}{\kappa_ {*}} (f \left(\mathbf {x} _ {t}\right) - f \left(\mathbf {x} _ {*}\right)).
$$
Subtracting  $f(\mathbf{x}_*)$  on both sides yields the contraction guarantee.
Figure 3: Lack of information from the ordering and separating hyperplanes.
4 Separating hyperplanes in higher dimensions
In one dimension, if the step-size  $\alpha$  does not satisfy the sufficient progress condition (3), we know  $\alpha > 1 / L$  and can rule out any  $\alpha' \geq \alpha$ . We are looking for a generalization to higher dimensions: if the queried preconditioner fails the sufficient progress condition, we should be able to discard all larger preconditioners. The notion of valid preconditioners formalizes this idea.
Definition 4.1 (Valid preconditioner). A preconditioner  $\mathbf{P}$  is valid if  $\mathbf{P}^{1/2} \nabla^2 f(\mathbf{x}) \mathbf{P}^{1/2} \preceq \mathbf{I}$  for all  $\mathbf{x}$ , which guarantees that  $\mathbf{P}$  satisfies the sufficient progress (4) condition, and invalid otherwise.
Validity is a global property: a preconditioner  $\mathbf{P}$  might lead to sufficient progress locally but still be invalid. Using the partial order, if  $\mathbf{P}$  is invalid then any preconditioner  $\mathbf{P}' \succeq \mathbf{P}$  is also invalid. However, this property alone only discards an exceedingly small portion of the feasible region in high dimensions. Consider the example illustrated in Figure 3a: if the diagonals are in a box  $\mathcal{B}(\mathbf{b})$ , the fraction of volume discarded in this way if  $(1/2)\operatorname{Diag}(\mathbf{b})$  is invalid is only  $1/2^d$ .
To efficiently search for valid preconditioners, we show that if  $f$  is convex, then the gradient of the sufficient progress condition gives a separating hyperplane for valid preconditioners. That is, it gives a vector  $\mathbf{u} \in \mathbb{R}^d$  such that if  $\mathbf{p} \in \mathbb{R}_{\geq 0}^{d}$  satisfies  $\langle \mathbf{u}, \mathbf{p} \rangle > 1$ , then  $\mathrm{Diag}(\mathbf{p})$  is invalid, as illustrated in Figure 3b. We use the following notation to denote normalized half-spaces:
$$
\mathcal {H} _ {>} (\mathbf {u}) := \{\mathbf {p} \in \mathbb {R} _ {\geq 0} ^ {d}: \langle \mathbf {u}, \mathbf {p} \rangle > 1 \} \qquad \text {a n d} \qquad \mathcal {H} _ {\leq} (\mathbf {u}) := \{\mathbf {p} \in \mathbb {R} _ {\geq 0} ^ {d}: \langle \mathbf {u}, \mathbf {p} \rangle \leq 1 \}.
$$
Proposition 4.2 (Separating hyperplane in preconditioner space). Suppose  $\mathbf{Q} = \mathrm{Diag}(\mathbf{q})\succ 0$  does not lead to sufficient progress (4) at  $\mathbf{x}$ , and let  $h(\mathbf{q})$  be the gap in the sufficient progress condition,
$$
h (\mathbf {q}) := f (\mathbf {x} - \mathbf {Q} \nabla f (\mathbf {x})) - f (\mathbf {x}) + \frac {1}{2} \| \nabla f (\mathbf {x}) \| _ {\mathbf {Q}} ^ {2} > 0.
$$
Then  $\operatorname{Diag}(\mathbf{p})$  for any  $\mathbf{p}$  in the following half-space satisfies  $h(\mathbf{p}) > 0$  and is also invalid,
$$
\left\{\mathbf {p} \in \mathbb {R} ^ {d}: \langle \nabla h (\mathbf {q}), \mathbf {p} \rangle > \langle \nabla h (\mathbf {q}), \mathbf {q} \rangle - h (\mathbf {q}) \right\}, \tag {5}
$$
This half-space is equal to  $\mathcal{H}_{>}(\mathbf{v})$  with  $\mathbf{v}$  given by  $\mathbf{v} = \nabla h(\mathbf{q}) / (\langle \nabla h(\mathbf{q}),\mathbf{q}\rangle -h(\mathbf{q}))$ , or
$$
\mathbf {v} := \frac {\left(\frac {1}{2} \mathbf {g} - \mathbf {g} ^ {+}\right) \odot \mathbf {g}}{f (\mathbf {x}) - \langle \mathbf {g} ^ {+} , \mathbf {Q} \mathbf {g} \rangle - f \left(\mathbf {x} ^ {+}\right)}, \quad \text {w i t h} \quad \left\{ \begin{array}{l} \mathbf {x} ^ {+} := \mathbf {x} - \mathbf {Q} \nabla f (\mathbf {x}), \\ (\mathbf {g}, \mathbf {g} ^ {+}) := (\nabla f (\mathbf {x}), \nabla f \left(\mathbf {x} ^ {+}\right)). \end{array} \right. \tag {6}
$$
Proof idea. If  $f$  is convex, then  $h$  also is. Convexity guarantees that  $h(\mathbf{p}) \geq h(\mathbf{q}) + \langle \nabla h(\mathbf{q}), \mathbf{p} - \mathbf{q} \rangle$  for any  $\mathbf{p}$ . A sufficient condition for  $h(\mathbf{p}) > 0$ , which means  $\mathbf{p}$  is invalid, is whether  $h(\mathbf{q}) + \langle \nabla h(\mathbf{q}), \mathbf{p} - \mathbf{q} \rangle > 0$  holds. Reorganizing yields Equation (5), and Equation (6) expresses the half-space in normalized form,  $\mathcal{H}_{>}(\mathbf{v})$ , expanding  $h$  in terms of  $f$ , its gradients, and  $\mathbf{Q}$ .
The half-space in Proposition 4.2 is however insufficient to find good enough cutting-planes, as it uses convexity to invalidate preconditioners but ignores the ordering that if  $\mathbf{P}$  is invalid, any  $\mathbf{P}' \succeq \mathbf{P}$  is also invalid. If such preconditioners are not already ruled out by convexity, we can find a stronger half-space by removing them, as illustrated in Figure 3c. We defer proofs to Appendix C.
Proposition 4.3 (Stronger hyperplanes). If  $\mathcal{H}_{>}(\mathbf{v})$  is a half-space given by Proposition 4.2, then  $\mathcal{H}_{>}(\mathbf{u})$  where  $\mathbf{u} := \max\{\mathbf{v}, 0\}$  element-wise is a stronger half-space in the sense that  $\mathcal{H}_{>}(\mathbf{v}) \subseteq \mathcal{H}_{>}(\mathbf{u})$ , and  $\mathcal{H}_{>}(\mathbf{u})$  contains only invalid preconditioners.
Figure 4: Minimum-volume enclosing boxes
5 Cutting-plane methods
The multidimensional backtracking method is in fact a cutting-plane method that uses separating hyperplanes (from Proposition 4.3) to search for valid preconditioners. The canonical example is the ellipsoid method (Yudin and Nemirovski, 1976; Shor, 1977), but its computational cost is  $\Omega(d^2)$  in  $\mathbb{R}^d$ . We now describe cutting-plane methods with three desirable properties: the preconditioners have good competitive ratios, the feasible set shrinks significantly when backtracking, and the computational cost is  $O(d)$ . There are many details, but the overall idea is similar to the ellipsoid method.
A simple warm-up: boxes. Consider the case when  $S_0$  consists of diagonal matrices with diagonals in the box  $\mathcal{B}(\mathbf{b}_0) = \{\mathbf{p} \in \mathbb{R}_{\geq 0}^d : \mathbf{p} \leq \mathbf{b}_0\}$ . We pick a candidate preconditioner by backtracking from the largest point in  $\mathcal{B}(\mathbf{b}_0)$  by some constant  $\gamma < 1$ , taking  $\mathbf{P} := \gamma \operatorname{Diag}(\mathbf{b}_0)$ . If  $\mathbf{P}$  satisfies the Armijo condition (4), we take a gradient step. If it does not, we compute the vector  $\mathbf{u}_0$  as in Proposition 4.3, and obtain a half-space  $\mathcal{H}_{>}(\mathbf{u}_0)$  that contains only invalid preconditioners. We then know we only need to search inside  $S_0 \cap \mathcal{H}_{\leq}(\mathbf{u}_0)$ . However, maintaining the set  $S_0 \cap \mathcal{H}_{\leq}(\mathbf{u}_0) \cap \dots \cap \mathcal{H}_{\leq}(\mathbf{u}_t)$  would be too complex to fit in  $O(d)$  time or memory. To reduce complexity, we define  $S_{t+1}$  as the box  $\mathcal{B}(\mathbf{b}_{t+1})$  of minimum volume containing  $\mathcal{B}(\mathbf{b}_t) \cap \mathcal{H}_{\leq}(\mathbf{u}_t)$ , as illustrated in Figure 4a. Due to this restriction, we might not be able to find a smaller set; the original box  $\mathcal{B}(\mathbf{b}_t)$  may already be the minimum volume box containing  $\mathcal{B}(\mathbf{b}_t) \cap \mathcal{H}_{\leq}(\mathbf{u}_t)$  if  $\mathbf{u}_t$  does not cut deep enough, as illustrated in Figure 4b. However, with enough backtracking  $(\gamma < 1/d)$ , we can show that the new box is smaller. This yields the following subroutines to fill in the gaps of Figure 2 (detailed in Appendix D)
$$
\operatorname {C A N D I D A T E} \left(\mathcal {S} _ {t}, \gamma , \mathbf {x} _ {t}\right) := \gamma \operatorname {D i a g} \left(\mathbf {b} _ {t}\right), \quad \operatorname {C U T} \left(\mathcal {S} _ {t}, \mathbf {P} _ {t}\right) := \left\{\operatorname {D i a g} (\mathbf {p}): \mathbf {p} _ {t} \in \mathcal {B} \left(\mathbf {b} _ {t + 1}\right) \right\}, \tag {7}
$$
where  $S_{t} = \mathcal{B}(\mathbf{b}_{t})$  and  $\mathbf{b}_{t + 1}\coloneqq \min \{\mathbf{b}_t,1 / \mathbf{u}_t\}$  element-wise, which give the following guarantees.
Theorem 5.1. Consider the multidimensional backtracking from Figure 2 initialized with a set  $S_0 = \{\mathrm{Diag}(\mathbf{p}) : \mathbf{p} \in \mathcal{B}(\mathbf{b}_0)\}$  containing  $\mathbf{P}_*$ , with the subroutines in Equation (7) with  $\gamma = 1/2d$ . Then: (a)  $\mathbf{P}_* \in S_t$ , (b) the candidate preconditioner  $\mathbf{P}_t$  is  $1/2d$ -competitive in  $S_t$  for any  $t$ , and
$$
(c) \quad \operatorname {V o l} (\mathcal {B} (\mathbf {b} _ {t + 1})) \leq \frac {1}{d + 1} \operatorname {V o l} (\mathcal {B} (\mathbf {b} _ {t})) \quad w h e n \mathbf {P} _ {t} f a i l s E q u a t i o n (4).
$$
In particular, CUT is not called more than  $d\log_{d + 1}(L\| \mathbf{b}_0\|_\infty)$  times.
Proof idea. To guarantee that the box shrinks, we have to guarantee that the half-space  $\mathcal{H}_{\leq}(\mathbf{u}_t)$  cuts deep enough. We know that the half-space has to exclude the query point  $\mathbf{P}_t$ , i.e.  $\langle \mathbf{p}_t, \mathbf{u}_t \rangle \geq 1$ , by Proposition 4.2 and that  $\mathbf{u}_t \geq 0$  by Proposition 4.3. Querying  $\mathbf{P}_t$  sufficiently close to the origin, by taking  $\gamma = 1/2d$ , is then enough to guarantee the decrease. To bound the total number of cuts, we note that the sets  $\mathcal{B}(\mathbf{b}_t)$  have a minimum volume  $\mathrm{Vol}_{\min}$ , as they have to contain the valid preconditioners. The number of cuts is at most  $\log_c(\mathrm{Vol}(\mathcal{B}(b_0)) / \mathrm{Vol}_{\min})$  for  $c = d + 1$ . We then bound  $\mathrm{Vol}(\mathcal{B}(b_0)) \leq \| \mathbf{b}_0\|_{\infty}^d$  and  $\mathrm{Vol}_{\min} \geq 1/L^d$  as  $(1/L)\mathbf{I}$  is a valid preconditioner.
5.1 Multidimensional Backtracking with Centered Axis-aligned Ellipsoids
We now improve the competitive ratio from  $O(1 / d)$  to  $O(1 / \sqrt{d})$  by switching from boxes to ellipsoids. Whereas general ellipsoids would require  $\Omega(d^2)$  complexity (as they involve a  $d \times d$  matrix), we consider centered, axis-aligned ellipsoids, defined by a diagonal matrix  $\mathbf{A} = \mathrm{Diag}(\mathbf{a})$ , of the form  $\mathcal{E}(\mathbf{a}) := \{\mathbf{p} \in \mathbb{R}_{\geq 0}^d : \| \mathbf{p} \|_{\mathbf{A}} \leq 1\}$ , where  $\| \mathbf{p} \|_{\mathbf{A}}^2 = \langle \mathbf{p}, \mathbf{A}\mathbf{p} \rangle$ . As preconditioners are non-negative, we consider only the positive orthant of the ellipsoid. For simplicity, we refer to those sets as ellipsoids.
Candidate preconditioner. In the box example, we selected the candidate preconditioner by backtracking from the largest preconditioner in the box. With an ellipsoid, there is no largest preconditioner. We need to choose where to backtrack from. To ensure the candidate preconditioner  $\mathbf{P}$  is competitive (Definition 3.1), we backtrack from the preconditioner that maximizes the progress  $\| \nabla f(\mathbf{x})\|_{\mathbf{P}}^2$
$$
\underset {\mathbf {p} \in \mathcal {E} (\mathbf {a})} {\arg \max } \| \nabla f (\mathbf {x}) \| _ {\mathbf {P}} ^ {2} = \frac {\mathbf {A} ^ {- 1} \nabla f (\mathbf {x}) ^ {2}}{\| \nabla f (\mathbf {x}) ^ {2} \| _ {\mathbf {A} ^ {- 1}}}, \quad \left(\text {w h e r e} \nabla f (\mathbf {x}) ^ {2} := \nabla f (\mathbf {x}) \odot \nabla f (\mathbf {x})\right). \tag {8}
$$
This lets us pick the preconditioner that makes the most progress for the current gradient, and will let us improve the competitive ratio by allowing a backtracking coefficient of  $\frac{1}{\sqrt{d}}$  instead of  $\frac{1}{d}$ .
Cutting. To complete the algorithm, we need to find a new set  $\mathcal{E}(\mathbf{b}_{t + 1})$  with smaller volume which contains the intersection of the previous set  $\mathcal{E}(\mathbf{b}_t)$  and the half-space  $\mathcal{H}_{\leq}(\mathbf{u}_t)$ . Unlike the box approach, the minimum volume ellipsoid has no closed form solution. However, if we backtrack sufficiently, by a factor of  $\gamma < 1 / \sqrt{d}$ , we can find an ellipsoid guaranteed to decrease the volume.
Lemma 5.2. Consider the ellipsoid  $\mathcal{E}(\mathbf{a})$  defined by  $\mathbf{A} = \mathrm{Diag}(\mathbf{a})$  for  $\mathbf{a} \in \mathbb{R}_{>0}^{d}$ . Let  $\mathbf{p} \in \mathcal{E}(\mathbf{a})$  be a point sufficiently deep inside the ellipsoid, such that  $\| \mathbf{p} \|_{\mathbf{A}} \leq 1 / \sqrt{2d}$ , and  $\mathcal{H}_{>}(\mathbf{u})$  be a half-space obtained from Proposition 4.3 at  $\mathbf{p}$ . The intersection  $\mathcal{E}(\mathbf{a}) \cap \mathcal{H}(\mathbf{u})_{\leq}$  is contained in the new ellipsoid
$$
\mathcal {E} \left(\mathbf {a} ^ {+} (\mathbf {a}, \mathbf {u})\right), \quad \text {w h e r e} \quad \mathbf {a} ^ {+} (\mathbf {a}, \mathbf {u}) = \lambda \mathbf {a} + (1 - \lambda) \mathbf {u} ^ {2}, \quad \lambda = \frac {\ell}{d} \frac {d - 1}{\ell - 1}, \quad \ell = \| \mathbf {u} \| _ {\mathbf {A} ^ {- 1}} ^ {2}, \tag {9}
$$
which has a smaller volume,  $\mathrm{Vol}(\mathcal{E}(\mathbf{a}^{+}(\mathbf{a},\mathbf{u}))\leq c\mathrm{Vol}(\mathcal{E}(\mathbf{a}))$  , where  $c = \sqrt[4]{\bar{\epsilon}} /\sqrt{2}\approx 0.91$
Proof idea. The new ellipsoid in (9) is a convex combination between  $\mathcal{E}(\mathbf{a})$  and the minimum volume ellipsoid containing the set  $\{\mathbf{p}\in \mathbb{R}^d:\langle \mathbf{u},|\mathbf{p}|\rangle \leq 1\}$  where  $|\mathbf{p}|$  is the element-wise absolute value of  $\mathbf{p}$ . The choice of  $\lambda$  in (9) is not optimal, but suffices to guarantee progress as long as  $\| \mathbf{p}\|_{\mathbf{A}}$  is small. A similar approach was used by Goemans et al. (2009) to approximate submodular functions, although they consider the polar problem of finding a maximum-volume enclosed ellipsoid. The full proof and discussion on the connections to the polar problem are deferred to Appendix D.
To improve the cuts, we can refine the estimate of  $\lambda$  in Lemma 5.2 by minimizing the volume numerically. We include this modification, detailed in Appendix D, in our experiments in Section 6.
Overall guarantees. We can now define the two subroutines for the ellipsoid method, and obtain the main result that we stated informally in Theorem 1.1, by combining the guarantees of the ellipsoid approach with the convergence result of Proposition 3.2.
Theorem 5.3. Consider the multidimensional backtracking from Figure 2 initialized with the set  $S_0 = \{\mathrm{Diag}(\mathbf{p}) : \mathbf{p} \in \mathcal{E}(\mathbf{a}_0)\}$  containing  $\mathbf{P}_*$ , given by some scaling  $\alpha_0 > 0$  of the uniform vector,  $\mathbf{a}_0 = \alpha_0\mathbf{1}$ . For  $S_t$ , let  $\mathbf{A}_t = \mathrm{Diag}(\mathbf{a}_t)$ . Define the subroutines
$$
\mathbf {P} _ {t} = \operatorname {C A N D I D A T E} (\mathcal {S} _ {t}, \gamma , \mathbf {x} _ {t}) := \gamma \frac {\mathbf {A} _ {t} ^ {- 1} \nabla f (\mathbf {x} _ {t}) ^ {2}}{\| \nabla f (\mathbf {x} _ {t}) ^ {2} \| _ {\mathbf {A} _ {t} ^ {- 1}}}, \operatorname {C U T} (\mathcal {S} _ {t}, \mathbf {P} _ {t}) := \left\{\operatorname {D i a g} (\mathbf {p}): \mathbf {p} \in \mathcal {E} (\mathbf {a} ^ {+} (\mathbf {a} _ {t}, \mathbf {u} _ {t})) \right\},
$$
where  $\mathbf{u}_t$  is the vector given by Proposition 4.3 when  $\mathbf{P}_t$  fails the Armijo condition at  $\mathbf{x}_t$ , and  $\mathbf{a}^+$  is computed as in (9). If  $\gamma = 1/\sqrt{2d}$ , then: (a)  $\mathbf{P}_* \in S_t$  for all  $t$ , (b) the candidate preconditioners  $\mathbf{P}_t$  are  $1/\sqrt{2d}$ -competitive in  $S_t$ , and (c) CUT is called no more than  $12d\log(L/\alpha_0)$  times.
To illustrate that multidimensional backtracking finds good preconditioners and improves over gradient descent on ill-conditioned problems even when accounting for the cost of backtracking, we run experiments on small but very ill-conditioned and large  $(d\approx 10^{6})$  problems.
As examples of adaptive gain and hypergradient methods, we include RPROP (Riedmiller and Braun, 1993) and GD with a hypergradient-tuned step-size (GD-HD, Baydin et al. 2018, multiplicative update). As examples of approximate second-order methods, we include diagonal BB (Park et al., 2020) and preconditioned GD using the diagonal of the Hessian. We use default parameters, except for the hypergradient method GD-HD, where we use  $10^{-10}$  as the initial step-size instead of  $10^{-3}$  to avoid immediate divergence. We include AdaGrad (diagonal), but augment it with a line-search as suggested by Vaswani et al. (2020), to make it competitive in the deterministic setting.
Line-searches and forward steps. For all methods that use a line-search, we include a forward step, a common heuristic in line-search procedures to allow for larger step-sizes when possible, although it
Figure 5: Multidimensional backtracking finds a good preconditioner, when there is one. Experiments on regularized linear and logistic regression on small but ill-conditioned datasets, cpusmall and breast-cancer (left, middle), and the large dataset News20 (right,  $d \approx 10^6$ ). Methods used: Gradient Descent (GD) Multidimensional Backtracking (MB) with ellipsoids, diagonal Hessian, diagonal BB, and diagonal AdaGrad—all of which use a line-search (+LS)—RPROP, and GD with hypergradient-tuned step-size (GD-HD) using the multiplicative update. Details in Appendix E.
can increase the number of backtracking steps. When a step-size or preconditioner is accepted, we increase the size of the set, allowing for larger (scalar or per-coordinate) step-sizes by a factor of 1.1.
Performance comparison. To capture the overhead of backtracking and provide a fair evaluation, Figures 1 and 5 compare performance per function and gradient evaluations (see Appendix E.1).
On a small but extremely ill-conditioned problems, our method is the only one that gets remotely close to being competitive with preconditioning with the diagonal Hessian—while only using first-order information. The diagonal Hessian is very close to the optimal preconditioner for those problems. On the cpusmall dataset, it reduces the condition number from  $\kappa \approx 5 \cdot 10^{13}$  to  $\approx 300$ , while  $\kappa_{*} \approx 150$ . All other methods struggle to make progress and stall before a reasonable solution is achieved, indicating they are not competitive with the optimal preconditioner.
On large regularized logistic regression on News20 ( $d \approx 10^6$ ), gradient descent performs relatively better, suggesting the problem is less ill-conditioned to begin with (the regularized data matrix has condition number  $\kappa \approx 10^4$ ). Despite the bound of  $O(d)$  backtracking steps, our methods find a reasonable preconditioner within 100 gradient evaluations. Despite the high dimensionality, it improves over gradient descent when measured in number of oracle calls.
Using plain gradient updates on the hyperparameters in GD-HD leads to unstable behavior, but diagonal BB and even RPROP, perform remarkably well on some problems—even outperforming preconditioning with the diagonal Hessian, which uses second-order information. However, they fail on other ill-conditioned problems, even when a good diagonal preconditioner exists. This pattern holds across other problems, as shown in Appendix E. Multidimensional backtracking demonstrates robust performance across problems, a clear advantage of having worst-case guarantees.
We designed multidimensional backtracking, an efficient algorithm to automatically find diagonal preconditioners that are competitive with the optimal diagonal preconditioner. Our work provides a definition of adaptive step-sizes that is complementary to the online learning definition. While online learning focuses on the adversarial or highly stochastic setting, we define and show how to find optimal per-coordinate step-sizes in the deterministic smooth convex setting. We show it is possible to build provably robust methods to tune a preconditioner using hypergradients. While our specific implementation uses cutting-planes, the general approach may lead to alternative algorithms, that possibly tune other hyperparameters, with similar guarantees.
The main limitation of our approach is its reliance on the convex deterministic setting. The results might transfer to the stochastic overparametrized regime using the approach of Vaswani et al. (2019), but the non-convex case seems challenging. It is not clear how to get reliable information from a cutting-plane perspective using hypergradients without convexity. As the first method to provably find competitive preconditioners, there are likely modifications that lead to practical improvements while preserving the theoretical guarantees. Possible ideas to improve practical performances include better ways to perform forward steps, using hypergradient information from accepted steps (which are currently ignored), or considering alternative structures to diagonal preconditioners.
We thank Aaron Mishkin for helpful discussions in the early stages of this work, and Curtis Fox and Si Yi (Cathy) Meng for providing comments on an early version of the manuscript, and for the feedback from anonymous reviewers. This research was partially by the Canada CIFAR AI Chair Program, the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grants RGPIN-2022-03669, and Borealis AI through the Borealis AI Global Fellowship Award.