SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang (2020). Disentangling Adaptive Gradient Methods from Learning Rates. arXiv/2002.11803.
Luís B. Almeida, Thibault Langlois, José F. M. Amaral, and Alexander Plakhov (1999). "Parameter adaptation in stochastic optimization". In: On-line learning in neural networks, pp. 111-134.
Ehsan Amid, Rohan Anil, Christopher Fifty, and Manfred K. Warmuth (2022). Step-size Adaptation Using Exponentiated Gradient Updates. Presented at the ICML Workshop Beyond First Order Methods in ML Systems.
Neculai Andrei (2019). "A diagonal quasi-Newton updating method for unconstrained optimization". In: Numerical Algorithms 81.2, pp. 575-590.
Larry Armijo (1966). "Minimization of functions having Lipschitz continuous first partial derivatives". In: Pacific Journal of mathematics 16.1, pp. 1-3.
Juhan Bae, Paul Vicol, Jeff Z. HaoChen, and Roger B. Grosse (2022). "Amortized Proximal Optimization". In: NeurIPS, pp. 8982-8997.
Andrew G. Barto and Richard S. Sutton (1981). Goal Seeking Components for Adaptive Intelligence: An Initial Assessment. Tech. rep. AFWAL-TR-81-1070. (Appendix C). Air Force Wright Aeronautical Laboratories/Avionics Laboratory.
Attilim Güneş Baydin, Robert Cornish, David Martínez-Rubio, Mark Schmidt, and Frank Wood (2018). "Online Learning Rate Adaptation with Hypergradient Descent". In: ICLR.
Albert S. Berahas, Majid Jahani, Peter Richtárik, and Martin Takáč (2022). "Quasi-Newton methods for machine learning: forget the past, just sample". In: Optimization Methods and Software 37.5, pp. 1668–1704.
Dimitri P. Bertsekas (1999). Nonlinear Programming. Athena Scientific.
Raghu Bollapragada, Dheevatsa Mudigere, Jorge Nocedal, Hao-Jun Michael Shi, and Ping Tak Peter Tang (2018). "A Progressive Batching L-BFGS Method for Machine Learning". In: ICML, pp. 619-628.
Richard H. Byrd, Samantha L. Hansen, Jorge Nocedal, and Yoram Singer (2016). "A Stochastic Quasi-Newton Method for Large-Scale Optimization". In: SIAM Journal on Optimization 26.2, pp. 1008-1031.
Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelley, and Erik Meijer (2022). "Gradient Descent: The Ultimate Optimizer". In: NeurIPS, pp. 8214-8225.
Chih-Chung Chang and Chih-Jen Lin (2011). "LIBSVM: A Library for Support Vector Machines". In: ACM Transactions on Intelligent Systems and Technology 2.3.
Aaron Defazio, Baoyu Zhou, and Lin Xiao (2022). Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method. arXiv/2206.06900.
Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier (2022). "A Simple Convergence Proof of Adam and Adagrad". In: Transactions on Machine Learning Research.
John E. Dennis Jr. and Jorge J. Moré (1977). "Quasi-Newton methods, motivation and theory". In: SIAM review 19.1, pp. 46-89.
Steven Diamond and Stephen Boyd (2016). "CVXPY: A Python-embedded modeling language for convex optimization". In: Journal of Machine Learning Research 17.83, pp. 1-5.
Dheeru Dua and Casey Graff (2017). UCI Machine Learning Repository.
Richard S. Sutton (1992a). "Adapting Bias by Gradient Descent: An Incremental Version of Delta-Bar-Delta". In: Proceedings of the 10th National Conference on Artificial Intelligence, San Jose, CA, USA, July 12-16, 1992, pp. 171-176.
Richard S. Sutton (1992b). "Gain adaptation beats least squares". In: Proceedings of the 7th Yale workshop on adaptive and learning systems, p. 166.
Pinar Tufekci (2014). "Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods". In: International Journal of Electrical Power & Energy Systems 60, pp. 126-140.
Sharan Vaswani, Frederik Kunstner, Issam H. Laradji, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien (2020). Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search). arXiv/2006.06835.
Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien (2019). "Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates". In: NeurIPS, pp. 3727-3740.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stefan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cirmrnan, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antonio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors (2020). "SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python". In: Nature Methods 17, pp. 261-272.
Rachel Ward, Xiaoxia Wu, and Léon Bottou (2019). "AdaGrad stepsizes: sharp convergence over nonconvex landscapes". In: ICML, pp. 6677-6686.
Philip Wolfe (1969). "Convergence conditions for ascent methods". In: SIAM review 11.2, pp. 226-235.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney (2021). "AdaHessian: An Adaptive Second Order Optimizer for Machine Learning". In: AAAI 35.12, pp. 10665-10673.
I.-Cheng Yeh (1998). "Modeling of strength of high-performance concrete using artificial neural networks". In: Cement and Concrete Research 28.12, pp. 1797-1808.
David B. Yudin and Arkadi S. Nemirovski (1976). "Informational complexity and effective methods for the solution of convex extremal problems". In: *Ekonom. i Mat. Metody* 12.2, pp. 357-369.
Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal (1997). "Algorithm 778: L-BFGS-B: Fortran Subroutines for Large-Scale Bound-Constrained Optimization". In: ACM Transactions on Mathematical Software 23.4, pp. 550-560.
M. Zhu, John L. Nazareth, and Henry Wolkowicz (1999). "The Quasi-Cauchy Relation and Diagonal Updating". In: SIAM Journal on Optimization 9.4, pp. 1192-1204.
A Full pseudocode of the algorithms 15
A.1 Subroutines for standard backtracking line-search 17
A.2 Separating hyperplanes used by multidimensional backtracking 17
A.3 Multidimensional backtracking using boxes 18
A.4 Multidimensional backtracking using ellipsoids 19
A.5 Implementable pseudocode 21
B Optimal preconditioners, valid preconditioners and competitive ratios 23
B.1 Defining optimal preconditioners without twice-differentiability or strong-convexity 24
B.2 Valid and optimal preconditioners with singular matrices 24
B.3 Best competitive ratio achievable by the optimal preconditioner 25
C Separating hyperplanes 27
C.1 Stronger hyperplanes 27
C.2 Separating hyperplanes for invalid preconditioners 28
D Cutting-plane methods 29
D.1 Boxes 29
D.2 Axis-aligned ellipsoids 30
E Experiments 36
E.1 Performance comparison 38
E.2 Additional results 38
Code available at https://github.com/fKunstner/multidimensional-backtracking
A Full pseudocode of the algorithms
We first give with a generic version using the subroutines INITIALIZE, CANDIDATE and CUT, to be specialized for the backtracking line-search (Figure 8), multidimensional backtracking using boxes (Figure 10), and ellipsoids (Figure 11). The generic pseudocode is written in terms of preconditioners, but also applies to the step-size version, which we can consider as looking for a preconditioner constrained to isotropic diagonal preconditioners, that is, preconditioners in the set  $\{\alpha \mathbf{I} : \alpha \in \mathbb{R}_{\geq 0}\}$ .
Although we write the pseudocode maintaining at each iteration an abstract set of preconditioners  $S$ , the only information the algorithm needs to maintain on each iteration for the implementation in the different cases is
- For the line-search:
the current maximum step-size  $\alpha_{\mathrm{max}}$  defining the interval of valid step-sizes,  $[0,\alpha_{\mathrm{max}}]$  such that the set of preconditioners is  $\mathcal{S} = \{\alpha \mathbf{I}:\alpha \in [0,\alpha_{\mathrm{max}}]\}$ ;
- For multidimensional backtracking with boxes:
the vector  $\mathbf{b}$  defining the maximum corner of the box  $\mathcal{B}(\mathbf{b}) = \{\mathbf{p} \in \mathbb{R}_{\geq 0}^d : \mathbf{p} \leq \mathbf{b}\}$  used to define the candidate diagonals preconditioners in the set  $\mathcal{S} = \{\mathrm{Diag}(\mathbf{p}) : \mathbf{p} \in \mathcal{B}(\mathbf{b})\}$ ;
- For multidimensional backtracking with ellipsoids:
the vector  $\mathbf{a}$  defining the axis-aligned ellipsoid  $\mathcal{E}(\mathbf{a}) = \{\mathbf{p} \in \mathbb{R}_{\geq 0}^{d} : \langle \mathbf{p}, \mathrm{Diag}(\mathbf{a})\mathbf{p} \rangle \leq 1\}$  used to define the candidate diagonal preconditioners in the set  $S = \{\mathrm{Diag}(\mathbf{p}) : \mathbf{p} \in \mathcal{E}(\mathbf{a})\}$ .
The pseudocode in Figure 6 updates  $(\mathbf{x}_t, S_t)$  to  $(\mathbf{x}_{t+1}, S_{t+1})$  at each iteration, and ensures that either the function value decreases,  $f(\mathbf{x}_{t+1}) < f(\mathbf{x}_t)$ , or the volume decreases,  $\mathrm{Vol}(S_{t+1}) < \mathrm{Vol}(S_t)$ .
We give an alternative pseudocode in Figure 7, which defines iterations as updates to the iterates  $\mathbf{x}_t$  that decrease the function value, and uses a while-loop to backtrack. Since it more closely resemble standard ways backtracking line-search is described, some reader may find it easier to understand. We stress, however, that this is still the same algorithm as Figure 6 but written differently.
The pseudocode in Figures 6-11, are expressed in a modular form to highlight how the algorithm works and its similarity to a line-search. In Appendix A.5, we give a more directly implementable pseudocode of multidimensional backtracking in both box and ellipsoid variants solely relying on vector notation.
Figure 6: Generic pseudocode for the line-search or multidimensional backtracking. uses the subroutines INITIALIZE, CANDIDATE, CUT defined in the later sections.
Backtracking Preconditioner Search with Sets   
Input: A starting point  $\mathbf{x}_0\in \mathbb{R}^d$  . A backtracking coefficient  $\gamma \in [0,1]$  .. A scalar  $c_{0} > 0$  larger than the optimal preconditioner, i.e., such that  $\mathbf{P}_{*}\preceq c_{0}\mathbf{I}$ $\mathcal{S}_0 = \mathrm{INITIALIZE}(c_0)$    
Iterate for  $t$  in  $0,1,\ldots ,T - 1$ $\begin{array}{rlr}{\bf P}_t = {\bf CANDIDATE}({\bf S}_t,\gamma ,\nabla f({\bf x}_t))\\ {\bf If}f({\bf x}_t - {\bf P}_t\nabla f({\bf x}_t))\leq f({\bf x}_t) - \frac{1}{2}\| \nabla f({\bf x}_t)\|_{\bf P_t}^2 & \mathrm{/ / Armijo condition~Equation~(4)}\\ ({\bf x}_{t + 1},{\cal S}_{t + 1}) = ({\bf x}_t - {\bf P}_t\nabla f({\bf x}_t),{\cal S}_t)\\ \mathrm{Otherwise},\\ ({\bf x}_{t + 1},{\cal S}_{t + 1}) = ({\bf x}_t,\mathrm{CUT}({\cal S}_t,{\bf x}_t,{\bf P}_t))\\ \end{array}$    
Output:  $\mathbf{x}_T$
Figure 7: Alternative pseudocode for the line-search or multidimensional backtracking. Uses a while-loop for backtracking and only updates the iterates  $\mathbf{x}_t$  when they lead to progress.
Backtracking Preconditioner Search with Sets - while-loop variant  
Input:  
A starting point  $\mathbf{x}_0\in \mathbb{R}^d$   
A backtracking coefficient  $\gamma \in [0,1]$   
A scalar  $c_{0} > 0$  larger than the best preconditioner, that is,  $\mathbf{P}_{*}\preceq c_{0}\mathbf{I}$ .  
Initialize the set  $S = \mathrm{INITIALIZE}(c_0)$   
Iterate for  $t$  in  $0,1,\ldots,T-1$ $\mathbf{P}_t\gets \mathrm{CANDIDATE}(\mathcal{S}_b,\gamma ,\nabla f(\mathbf{x}_t))$   
While  $f(\mathbf{x}_t - \mathbf{P}_t\nabla f(\mathbf{x}_t))\leq f(\mathbf{x}_t) - \frac{1}{2}\| \nabla f(\mathbf{x}_t)\|_{\mathbf{P}_t}^2$  // Armijo condition Equation (4)  
 $\mathcal{S}\gets \mathrm{CUT}(\mathcal{S},\mathbf{x}_t,\mathbf{P}_t)$ $\mathbf{P}_t\gets \mathrm{CANDIDATE}(\mathcal{S},\gamma ,\nabla f(\mathbf{x}_t))$ $\mathbf{x}_{t + 1} = \mathbf{x}_t - \mathbf{P}_t\nabla f(\mathbf{x}_t)$   
Output:  $\mathbf{x}_T$
A.1 Subroutines for standard backtracking line-search
Implementation of the subroutines for the standard backtracking line-search. Although written in terms of sets, the algorithm only needs to maintain the maximum step-size in the interval  $[0, \alpha_{\max}]$  at each iteration. The corresponding preconditioners are the matrices  $S = \{\alpha \mathbf{I} : \alpha \in [0, \alpha_{\max}]\}$ .
Figure 8: Specialization of the subroutines for the backtracking line-search
A.2 Separating hyperplanes used by multidimensional backtracking
Both versions of multidimensional backtracking need a direction to update the set of preconditioners in the CUT subroutine. We define the subroutine SEPARATINGHYPERPLANE in Figure 9. The description of the separating hyperplane and their properties can be found in Section 4 and Appendix C.
Figure 9: Separating hyperplane used by both variants of multidimensional backtracking.
A.3 Multidimensional backtracking using boxes
The implementation of multidimensional backtracking with boxes only needs to maintain a vector  $\mathbf{b}$ , representing the maximum step-size for each coordinate that has not been ruled out, in the box  $\mathcal{B}(\mathbf{b})$ . The associated sets of preconditioners  $\mathcal{S}$  are
$$
\mathcal {B} (\mathbf {b}) = \left\{\mathbf {p} \in \mathbb {R} _ {\geq 0} ^ {d}: \mathbf {p} \leq \mathbf {b} \right\}, \quad \mathcal {S} = \left\{\operatorname {D i a g} (\mathbf {p}): \mathbf {p} \in \mathcal {B} (\mathbf {b}) \right\}.
$$
The description of boxes and the theoretical guarantees when using them in multidimensional backtracking can be found in Section 5 and Appendix D.1. The subroutines used by the algorithm with boxes are:
- INITIALIZER: initializes  $\mathbf{b}$  to  $c_{0}\mathbf{1}$  so that the diagonal preconditioner  $c_{0}\mathbf{I}$  is in  $\mathcal{S}_0$ .
CANDIDATE: backtracks from the largest diagonal in  $\mathcal{B}(\mathbf{b})$ , returning  $\gamma$  Diag(b).
- SEPARATINGHYPERPLANE: computes the vector  $\mathbf{u}$  defining the half-space of invalid preconditioners  $\mathcal{H}_{>}(\mathbf{u})$  obtained when the preconditioner  $\mathbf{P}$  fails the Armijo condition at  $\mathbf{x}$  as described in Proposition 4.2 and Proposition 4.3.
- CUT: returns the minimum volume box  $\mathcal{B}(\mathbf{b}^{+})$  containing the intersection  $\mathcal{B}(\mathbf{b})\cap \mathcal{H}_{\leq}(\mathbf{u})$
Figure 10: Specialization of the subroutines for multidimensional backtracking with boxes
INITIALIZE  $(c_0)$    
Input: A scalar  $c_{0}$  such that  $c_{0}\mathbf{I}$  is larger than the optimal diagonal preconditioner, i.e.,  $\mathbf{P}_{*}\preceq c_{0}\mathbf{I}$    
Output: Set  $\mathcal{S} = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{B}(\mathbf{b})\}$  with  $\mathcal{B}(\mathbf{b}) = \{\mathbf{p}\in \mathbb{R}_{\geq 0}^{d}:\mathbf{p}\leq \mathbf{b}\}$  where  $\mathbf{b}\coloneqq c_0\mathbf{1}$
CANDIDATE  $(S,\gamma ,\mathbf{x})$    
Input: Set of preconditioners  $S = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{B}(\mathbf{b})\}$  with  $\mathcal{B}(\mathbf{b}) = \{\mathbf{p}\in \mathbb{R}_{\geq 0}^{d}:\mathbf{p}\leq \mathbf{b}\}$  Backtracking coefficient  $\gamma \in [0,1]$  Current iterate  $\mathbf{x}\in \mathbb{R}^d$  //Not used for the box version.   
Output: Preconditioner  $\gamma$  Diag(b)
$\mathrm{CUT}(\mathcal{S},\mathbf{x},\mathbf{P})$    
Input: Set of preconditioners  $\mathcal{S} = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{B}(\mathbf{b})\}$  with  $\mathcal{B}(\mathbf{b}) = \{\mathbf{p}\in \mathbb{R}_{\geq 0}^{d}:\mathbf{p}\leq \mathbf{b}\}$  Backtracking coefficient  $\gamma \in [0,1]$  Preconditioner  $\mathbf{P}_{\mathrm{bad}}$  that failed the Armijo condition at x.   
 $\mathbf{u} = \mathrm{SEPARATINGHYPERPLANE}(\mathbf{x}_t,\mathbf{P}_{\mathrm{bad}})$ $\mathbf{b}^{+} = \max \{\mathbf{b},1 / \mathbf{u}\}$  //Minimum volume box  $\mathcal{B}(\mathbf{b}^{+})$  containing  $\mathcal{B}(\mathbf{b})\cap \mathcal{H}_{\leq}(\mathbf{u})$    
Output: Set of diagonal preconditioners  $S = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{B}(\mathbf{b}^{+})\}$
A.4 Multidimensional backtracking using ellipsoids
The implementation only needs to maintain a vector  $\mathbf{a}$  representing the diagonal of the matrix defining the (centered, axis-alligned) ellipsoid  $\mathcal{E}(\mathbf{a})$  and the associated set of preconditioners  $S$  given by
$$
\mathcal {E} (\mathbf {a}) = \left\{\mathbf {p} \in \mathbb {R} _ {\geq 0} ^ {d}: \langle \mathbf {p}, \operatorname {D i a g} (\mathbf {a}) \mathbf {p} \rangle \leq 1 \right\}, \quad \mathcal {S} = \left\{\operatorname {D i a g} (\mathbf {p}): \mathbf {p} \in \mathcal {E} (\mathbf {a}) \right\}.
$$
The description of the ellipsoids and their properties can be found in Section 5 and Appendix D.2. The subroutines used by the algorithm with boxes are:
- INITIALIZIZE: initializes  $\mathbf{a}$  to  $(1 / dc_0^2)\mathbf{1}$  so that  $c_{0}\mathbf{1} \in \mathcal{E}(\mathbf{a})$ , implying the diagonal preconditioner  $c_{0}\mathbf{I}$  is in  $\mathcal{S}$ .
- CANDIDATE: backtracks from the diagonal preconditioner in  $S$  that maximizes the gradient norm. Let  $\mathcal{E}(\mathbf{a})$  be the set of candidate diagonals and define  $\mathbf{A} = \operatorname{Diag}(\mathbf{a})$ . The subroutine returns  $\gamma \mathbf{P}_{\max}$ , where
$$
\mathbf {P} _ {\max } := \underset {\mathbf {P} \in \mathcal {S}} {\arg \max } \| \nabla f (\mathbf {x}) \| _ {\mathbf {P}} ^ {2}.
$$
Writing this in terms of the diagonal vector  $\mathbf{p}_{\mathrm{max}}\coloneqq \mathrm{diag}(\mathbf{P}_{\mathrm{max}})$  yields
$$
\begin{array}{l} \mathbf{p}_{\max} = \operatorname *{arg  max}_{\mathbf{p}\in \mathcal{E}(\mathbf{a})}\| \nabla f(\mathbf{x})\|_{\mathrm{Diag}(\mathbf{p})}^{2}, \\ = \operatorname * {a r g   m a x} _ {\mathbf {p}} \left\langle \nabla f (\mathbf {x}) ^ {2}, \mathbf {p} \right\rangle : \| \mathbf {p} \| _ {\mathbf {A}} \leq 1 = \frac {\mathbf {A} ^ {- 1} \nabla f (\mathbf {x}) ^ {2}}{\| \nabla f (\mathbf {x}) \| _ {\mathbf {A} ^ {- 1}}}, \\ \end{array}
$$
where  $\nabla f(\mathbf{x})^2 = \nabla f(\mathbf{x})\odot \nabla f(\mathbf{x})$
- SEPARATINGHYPERPLANE: computes the vector  $\mathbf{u}$  defining the half-space of invalid preconditioners  $\mathcal{H}_{>}(\mathbf{u})$  obtained when the preconditioner  $\mathbf{P}$  fails the Armijo condition at  $\mathbf{x}$  as described in Proposition 4.2 and Proposition 4.3.
- CUT: returns an ellipsoid  $\mathcal{E}(\mathbf{a}^{+})$  containing the intersection of  $\mathcal{E}(\mathbf{a}) \cap \mathcal{H}_{\leq}(\mathbf{u})$  with guaranteed volume decrease from  $\mathcal{E}(\mathbf{a})$ . As there is no closed-form solution for the minimum volume ellipsoid, we set  $\mathbf{a}^{+}$  as a convex combination between the original ellipsoid  $\mathcal{E}(\mathbf{a})$  and the minimum volume axis-aligned ellipsoid containing  $\mathcal{H}_{\leq}(\mathbf{u})$ , given by  $\mathcal{E}(\mathbf{u}^2)$ , that is,
$$
\mathbf {a} ^ {+} := \lambda \mathbf {a} + (1 - \lambda) \mathbf {u} ^ {2}, \quad \text {w h e r e} \quad \lambda := \frac {\ell}{d} \frac {d - 1}{\ell - 1} \quad \text {a n d} \quad \ell := \| \mathbf {u} \| _ {\mathbf {A} ^ {- 1}} ^ {2}, \tag {10}
$$
where  $\mathbf{A} \coloneqq \mathrm{diag}(\mathbf{a})$ . Although the above choice of  $\lambda$  has guaranteed volume decrease, we can find a better value of  $\lambda$  by solving the minimum volume ellipsoid as a function of  $\lambda$  numerically. Namely, approximating
$$
\lambda^ {*} := \underset {0 <   \lambda <   1} {\arg \min } - \log (\det  (\lambda \operatorname {D i a g} (\mathbf {a}) + (1 - \lambda) \operatorname {D i a g} (\mathbf {u} ^ {2}))). ^ {\prime}
$$
In our experiments, we start with  $\lambda$  as in (10) and, starting from it, we solve the above minimization problem numerically using L-BFGS-B (Zhu et al., 1997) in SciPy (Virtanen et al., 2020). This preserves the theoretical guarantee while improving empirical performance.
Figure 11: Specialization of the subroutines for multidimensional backtracking with ellipsoids
INITIALIZE  $(c_0)$    
Input: A scalar  $c_{0} > 0$  such that  $c_{0}\mathbf{I}$  is larger than the optimal diagonal preconditioner, i.e.,  $\mathbf{P}_{*}\preceq c_{0}\mathbf{I}$  Output:  $\mathcal{S} = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{E}(\mathbf{a})\}$  with  $\mathcal{E}(\mathbf{a}) = \{\mathbf{p}\in \mathbb{R}_{\geq 0}^{d}:\langle \mathbf{p},\mathrm{Diag}(\mathbf{a})\mathbf{p}\rangle \leq 1\}$  for  $\mathbf{a} = \frac{1}{dc_0^2}\mathbf{1}$
CANDIDATE(S,γ,x)  
Input: A set S = {Diag(p):p ∈ E(a)} where E(a) = {p ∈ Rd≥0 : <p, Diag(a)p> ≤ 1}, and a ∈ Rd>0; Backtracking coefficient γ ∈ [0,1]; Current iterate x ∈ Rd.  
d = ∇f(x) ∩ ∇f(x)  
pmax = A-1d //Where A = Diag(a) ||d||A-1  
Output: Preconditioner γ Diag(pmax)
$\mathrm{CUT}(\mathcal{S},\mathbf{x},\mathbf{P})$    
Input: A set  $\mathcal{S} = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{E}(\mathbf{a})\}$  , where  $\mathcal{E}(\mathbf{a}) = \{\mathbf{p}\in \mathbb{R}_{\geq 0}^{d}:\langle \mathbf{p},\mathrm{Diag}(\mathbf{a})\mathbf{p}\rangle \leq 1\}$  , and  $\mathbf{a}\in \mathbb{R}_{>0}^{d}$  . Current iterate  $\mathbf{x}\in \mathbb{R}^d$  Preconditioner  $\mathbf{P}_{\mathrm{bad}}$  that failed the Armijo condition at x.   
 $\mathbf{u} = \mathrm{SEPARATINGHYPERPLANE}(\mathbf{x}_t,\mathbf{P}_{\mathrm{bad}})$ $\ell = \| \mathbf{u}\|_{\mathbf{A}^{-1}}^{2}$ $\lambda = \frac{\ell}{d}\frac{d - 1}{\ell - 1}$  (or numerically solve  $\lambda = \arg \min_{0 <   c <   1} - \log (\operatorname *{det}(c\operatorname {Diag}(\mathbf{a}) + (1 - c)\operatorname {Diag}(\mathbf{u}^2))))$ ${\bf a}^{+} = \lambda {\bf a} + (1 - \lambda){\bf u}^{2}$  //Approx.min. volume ellipsoid  $\mathcal{E}(\mathbf{a}^{+})$  containing  $\mathcal{E}(\mathbf{a})\cap \mathcal{H}_{\leq}(\mathbf{u})$    
Output: The set  $S = \{\mathrm{Diag}(\mathbf{p}):\mathbf{p}\in \mathcal{E}(\mathbf{a}^{+})\}$
A.5 Implementable pseudocode
The pseudocode in Figures 6-11 are expressed in a modular form to highlight how the algorithm works and its similarity to a line-search. In this section, we give a more directly implementable pseudocode of multidimensional backtracking, in both the box and ellipsoid variants, using mostly vector notation. Scalar operations on vectors such as  $\mathbf{u} / \mathbf{a}$ ,  $\sqrt{\mathbf{u}}$ ,  $\mathbf{u}^2$  are understood to be taken element-wise.
Multidimensional backtracking using boxes	Direct implementation
Input:
Function to optimize f: Rd → Rd;
Starting point x0 ∈ Rd;
A scalar for the scale of initial set of preconditioners c0 > 0;
Backtracking coefficient γ < 1/d.
b = c01	//Initialize box
Iterate for t in 0, 1,...
pt = γb	//Get candidate preconditioner
gt = ∇f(xt)
xt+ = xt - pt · gt	//Get candidate point
While f(xt+) > f(xt) - 1/2(gt2, pt)	//Armijo condition fails
gt+ = ∇f(xt+)
dt = (1/2gt - gt+) · gt	//Get next gradient to compute
ct = f(xt) - f(xt+) - (gt · pt, gt+)	//the separating hyperplane direction,
ut = max{dt/ct, 0} (element-wise)	//the normalization constant,
b = 1/ max{1/b, ut} (element-wise)	//and truncate it
pt = γb	//Pick next candidate preconditioner
xt+ = xt - pt · gt	//and next candidate point
xt+1 = xt+	//Accept new point
Output: xt
Multidimensional backtracking using ellipsoids	Direct implementation
Input:
Function to optimize f: Rd → R;
Starting point x0 ∈ Rd;
A scalar for the scale of initial set of preconditioners c0 > 0;
Backtracking coefficient γ < 1/√d
a = 1/(dc02)	//Initialize ellipsoid
Iterate for t in 0, 1, ...
gt = ∇f(xt)
pt = gt2/a (element-wise)	//Get candidate preconditioner
pt = γpt/||gt2/√a|| (element-wise)	//normalize it
xt+ = xt - pt o gt	//Get candidate point
While f(xt+) > f(xt) - 1/2<gt2, pt>	//Armijo condition fails
gt+ = ∇f(xt+)
dt = (1/2gt - gt+) o gt	//Get next gradient to compute
ct = f(xt) - f(xt+) - <gt o pt, gt>	//the separating hyperplane direction,
ut = max{dt/ct, 0} (element-wise)	//the normalization constant,
take λ = ℓ(d-1)/d(ℓ-1) where ℓ = ⟨u2, 1/a⟩	//Approx. min. vol. new ellipsoid
or	//
find λ by numerically minimizing φ(λ) where φ(λ) = -∑i=1d log(λa[i] + (1 - λ)ut[i]2)	//Find better approximation of min.
a = λa + (1 - λ)u2	//New ellipsoid
pt = gt2/a (element-wise)	//Get new candidate preconditioner,
pt = γpt/||gt2/√a|| (element-wise)	//normalized,
xt+ = xt - pt o gt	//and new candidate point
xt+1 = xt+	//Accept new point
Output: xt