Title: Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking
Venue: NeurIPS 2023
Accepted: yes
Keywords: line-search

gradient descent

hypergradient

adaptive methods

smooth

convex

optimization

preconditioning
Abstract: The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically optimal step-size. Many approaches have been developed to instead tune per-coordinate step-sizes, also known as diagonal preconditioners, but none of the existing methods are provably competitive with the optimal per-coordinate step-sizes. We propose multidimensional backtracking, an extension of the backtracking line-search to find good diagonal preconditioners for smooth convex problems. Our key insight is that the gradient with respect to the step-sizes, also known as hyper-gradients, yields separating hyperplanes that let us search for good preconditioners using cutting-plane methods. As black-box cutting-plane approaches like the ellipsoid method are computationally prohibitive, we develop an efficient algorithm tailored to our setting. Multidimensional backtracking is provably competitive with the best diagonal preconditioner and requires no manual tuning.

Searching for Optimal Per-Coordinate Step-sizes with Multidimensional Backtracking

Abstract

The backtracking line-search is an effective technique to automatically tune the step-size in smooth optimization. It guarantees similar performance to using the theoretically optimal step-size. Many approaches have been developed to instead tune per-coordinate step-sizes, also known as diagonal preconditioners, but none of the existing methods are provably competitive with the optimal per-coordinate stepsizes. We propose multidimensional backtracking , an extension of the backtracking line-search to find good diagonal preconditioners for smooth convex problems. Our key insight is that the gradient with respect to the step-sizes, also known as hypergradients, yields separating hyperplanes that let us search for good preconditioners using cutting-plane methods. As black-box cutting-plane approaches like the ellipsoid method are computationally prohibitive, we develop an efficient algorithm tailored to our setting. Multidimensional backtracking is provably competitive with the best diagonal preconditioner and requires no manual tuning.

1 Introduction

When training machine learning models, tuning the hyperparameters of the optimizer is often a major challenge. For example, finding a reasonable step-size hyperparameter for gradient descent typically involves trial-and-error or a costly grid search. In smooth optimization, a common approach to set the step-size without user input is a backtracking line-search: start with a large step-size, and decrease it when it is too big to make sufficient progress. For ill-conditioned problems, however, there are limits to the improvement achievable by tuning the step-size. Per-coordinate step-sizes-also known as diagonal preconditioners-can drastically improve performance. Many approaches have been developed to automatically tune per-coordinate step-sizes. Those are often described as 'adaptive' methods, but the meaning of this term varies widely, from describing heuristics that set per-coordinate step-sizes, to ensuring performance guarantees as if a particular property of the problem were known in advance. Yet, even on the simplest case of a smooth and strongly convex deterministic problem where a good fixed diagonal preconditioner exists (i.e., one that reduces the condition number), none of the existing adaptive methods are guaranteed to find per-coordinate step-sizes that improve the convergence rate. We discuss approaches to adaptive methods in the next section.

Contribution. We propose multidimensional backtracking , an extension of the standard backtracking line-search to higher dimension, to automatically find good per-coordinate step-sizes. Our method recovers the convergence rate of gradient descent with the optimal preconditioner for the problem, up to a √ 2 d factor where d is the number of coordinates. This is a direct generalization of the line-search guarantee, with a penalty depending on dimension due to the extra degrees of freedom, as expected.

1.1 Adaptive step-sizes and preconditioning methods

Adaptive and parameter-free methods in online learning are an example where adaptive methods have a well-defined meaning. AdaGrad (McMahan and Streeter, 2010; Duchi et al., 2011) and Coin Betting (Orabona and Pál, 2016; Orabona and Tommasi, 2017) can adapt to problem-specific constants without user input and have strong guarantees, even in the adversarial setting. However,

this resilience to adversaries is a double-edged sword; to satisfy this definition of adaptivity, AdaGrad uses monotonically decreasing step-sizes. While AdaGrad still converges at the desired asymptotic rate on smooth, Lipschitz functions (Ward et al., 2019; Li and Orabona, 2019), its performance can be worse than plain gradient descent. This motivated investigations of workarounds to avoid the monotonically decreasing updates, including augmenting the update with an increasing step-size schedule (Agarwal et al., 2020), a line-search (Vaswani et al., 2020), or modifying the update to the preconditioner (Defazio et al., 2022). Methods commonly used in deep learning, such as RMSProp and Adam (Hinton et al., 2012; Kingma and Ba, 2015), are often motivated as adaptive by analogy to AdaGrad, but without decreasing step-sizes (e.g., Défossez et al., 2022, §4.3). This change is crucial for their practical performance, but nullifies their online-learning adaptivity guarantees.

Adaptive gain and hypergradient heuristics. Many heuristics that tune the hyperparameters of the optimization procedure use the gradient with respect to the hyperparameters, or hypergradients (Maclaurin et al., 2015). Methods have been proposed to tune the step-size (Masse and Ollivier, 2015), a preconditioner (Moskovitz et al., 2019), any hyperparameter (Baydin et al., 2018), or to maintain a model of the objective (Bae et al., 2022). 'Stacking' such optimizers recursively has been shown to reduce the dependency on user-specified hyperparameters in practice (Chandra et al., 2022). This idea pre-dates the hypergradient nomenclature; Kesten (1958) presents a method to update the step-size based on the sign of successive gradients, and Saridis (1970) presents a control perspective for per-coordinate step-sizes, which can be cast as a hypergradient update to a diagonal preconditioner. 1 This approach has led to adaptive gain methods such as Delta-Delta and variants (Barto and Sutton, 1981; Jacobs, 1988; Silva and Almeida, 1990; Sutton, 1992a,b), and further developed using the sign of the hypergradient (Riedmiller and Braun, 1993), full-matrix updates (Almeida et al., 1999), a larger history (Plagianakos et al., 2001), updates in log-space (Schraudolph, 1999; Schraudolph et al., 2005), heuristics to adjust the outer step-size (Mahmood et al., 2012), or multiplicative weight updates (Amid et al., 2022). While showing promising practical performance in some settings, existing methods are often motivated from intuition rather than a formal definition of adaptivity, giving no guarantee that the tuned method will converge faster, if at all. Indeed, hypergradient methods are often unstable, and may require as much manual tuning as the original optimizer they are intended to tune.

Second-order methods. A classical approach to preconditioning is to use second-order information, as in Newton's method or its regularized variants (e.g., Nesterov and Polyak, 2006). To avoid the load of computing and inverting the Hessian, quasi-Newton methods (Dennis and Moré, 1977) such as L-BFGS (Liu and Nocedal, 1989) fit an approximate Hessian using the secant equation. Variants using diagonal approximations have also been proposed, framed as Quasi-Cauchy, diagonal BFGS, or diagonal Barzilai-Borwein methods (Zhu et al., 1999; Andrei, 2019; Park et al., 2020), while other methods use the diagonal of the Hessian (LeCun et al., 2012; Yao et al., 2021). Some second-order and quasi-Newton methods converge super linearly (although not the diagonal or limited memory variants used in practice), but those guarantees only hold locally when close to the minimum. To work when far from a solution, those methods require 'globalization' modifications, such as regularization or a line-search. Unfortunately, analyses of second-order methods do not capture the global benefit of preconditioning and instead lead to worse rates than gradient descent, as in the results of Byrd et al. (2016, Cor. 3.3), Bollapragada et al. (2018, Thm. 3.1), Meng et al. (2020, Thm. 1), Yao et al. (2021, Apx.), Berahas et al. (2022, Thm. 5.2), or Jahani et al. (2022, Thm. 4.9).

Line-searches. Adaptivity in smooth optimization is most closely related to line-searches. The standard guarantee for gradient descent on an L -smooth function requires a step-size of 1 /L , but L is typically unknown. The backtracking line-search based on the Armijo condition (Armijo, 1966) approximately recovers this convergence guarantee by starting with a large step-size, and backtracking; halving the step-size whenever it does not yield sufficient improvement. However, line-searches are often overlooked in the discussion of adaptive methods, as they do not provide a way to set more than a scalar step-size. While line-searches can be shown to work in the stochastic overparameterized setting and have been applied to train neural networks (Vaswani et al., 2019), improvements beyond backtracking have been limited. Additional conditions (Wolfe, 1969), non-monotone relaxations (Grippo et al., 1986), or solving the line-search to higher precision (Moré and Thuente, 1994) can improve the performance in practice, but even an exact line-search cannot improve the convergence rate beyond what is achievable with a fixed step-size (Klerk et al., 2017).

0

200

400

600

800

1000

Gradient evaluations

10

1

10 1

10 3

Optimality gap

GD+LS

*

Ellipsoid MB

1

3

5

7

9

11

13

Coordinate

10

10

10

2

10 6

T=0

T=30

T=45

T=65

Per-coordinate stepsizes

1.2 Summary of main result: adaptivity to the optimal preconditioner

Our approach is inspired by the work discussed above, but addresses the following key limitation: none of the existing methods attain better global convergence rates than a backtracking line-search. Moreover, this holds even on smooth convex problems for which a good preconditioner exists.

We generalize the backtracking line-search to handle per-coordinate step-sizes and find a good preconditioner. As in quasi-Newton methods, we build a preconditioner based on first-order information. However, instead of trying to approximate the Hessian using past gradients, our method searches for a preconditioner that minimizes the objective function at the next step. Our convergence result depends on the best rate achievable by an optimal diagonal preconditioner , similarly to how methods in online learning are competitive against the best preconditioner in hindsight. However, our notion of optimality is tailored to smooth strongly-convex problems and does not require decreasing step-sizes as in AdaGrad. Our update to the preconditioner can be interpreted as a hypergradient method, but instead of a heuristic update, we develop a cutting-plane method that uses hypergradients to guarantee a good diagonal preconditioner. Our main theoretical contribution is summarized below.

Theorem 1.1 (Informal) . On a smooth, strongly-convex function f in d dimensions, steps accepted by multidimensional backtracking guarantee the following progress

where κ ∗ is the condition number achieved by the optimal preconditioner defined in Section 2. The number of backtracking steps is at most linear in d and logarithmic in problem-specific constants.

Multidimensional backtracking finds per-coordinate step-sizes that lead to a provable improvement over gradient descent on badly conditioned problems that can be improved by diagonal preconditioning, i.e., if the condition number of f is at least √ 2 d · κ ∗ . Moreover, this guarantee is worst-case, and multidimensional backtracking can outperform the globally optimal preconditioner by finding a better local preconditioner, as illustrated on an ill-conditioned linear regression problem in Figure 1.

To find a competitive diagonal preconditioner, we view backtracking line-search as a cutting-plane method and generalize it to higher dimensions in Section 3. In Section 4 we show how to use hypergradients to find separating hyperplanes in the space of preconditioners, and in Section 5 we develop an efficient cutting-plane methods tailored to the problem. In Section 6, we illustrate the method through preliminary experiments and show it has consistent performance across problems.

Notation. We use standard font weight d , n , α for scalars, bold x , y for vectors, and capital bold P , A for matrices. We use p [ i ] for the i -th entry of p , use ⊙ for element-wise multiplication, and define p 2 := p ⊙ p . We use P = Diag( p ) to denote the diagonal matrix with diagonal p , and p = diag( P ) to denote the vector of diagonal entries of P . We say A is larger than B , A ⪰ B , if A -B is positive semidefinite. If A = Diag( a ) , B = Diag( b ) , the ordering A ⪰ B is equivalent to a [ i ] ≥ b [ i ] for all i , which we write a ≥ b . We use I for the identity matrix and 1 for the all-ones vector.

2 Optimal preconditioning and sufficient progress

Consider a twice-differentiable function f : R d → R that is L -smooth and µ -strongly convex, 2 i.e., or µ I ⪯ ∇ 2 f ( x ) ⪯ L I for all x . We measure the quality of a preconditioner P by how well P -1 approximates the Hessian ∇ 2 f ( x ) . We define an optimal diagonal preconditioner for f as

and denote by κ ∗ the optimal κ above. A related and known measure for the convergence rate of preconditioned methods is κ ( P 1 / 2 ∇ 2 f ( x ) P 1 / 2 ) (Bertsekas, 1999, §1.3.2). Moreover, (1) reduces to the definition of optimal preconditioning for linear systems (Jambulapati et al., 2020; Qu et al., 2022) when f is quadratic. Alternatively, the optimal preconditioner can be viewed as the matrix P ∗ such that f is 1 -smooth and maximally strongly-convex in the norm ∥ x ∥ 2 P -1 ∗ = 〈 x , P -1 ∗ x 〉 ,

Similar definitions of smoothness and strong-convexity relative to a matrix are common in coordinate descent methods (e.g., Qu et al., 2016; Safaryan et al., 2021), where the matrices are assumed to be known a priori. If we knew P ∗ , preconditioned gradient descent using P ∗ would converge at the rate

where x ∗ minimizes f . We do not know P ∗ and will be searching for a good approximation.

For the standard backtracking line-search on L -smooth functions, the goal is to find a step-size that works as well as 1 / L without knowledge of L . To do so, we can start with a large step-size α ≫ 1 / L and check the Armijo condition : the step-size α makes progress as if f were 1 /α -smooth, that is,

If the condition is satisfied, we take the step x -α ∇ f ( x ) . By the descent lemma, (Bertsekas, 1999, A.24), the condition is satisfied if α ≤ 1 / L . So if the condition fails, we know α is too large and can decrease α . For diagonal preconditioners, the Armijo condition checks whether the preconditioner makes sufficient progress in the norm induced by P -1 , as if f were 1-smooth in Equation (2), that is,

As with a scalar step-size, sufficient progress holds for any matrix P that satisfies ∇ 2 f ( x ) ⪯ P -1 .

3 Multidimensional Backtracking

The typical presentation of the backtracking line-search maintains a step-size and decreases it when the Armijo condition fails (e.g., Nocedal and Wright, 1999, Alg 3.1). We instead take the following non-standard view, which generalizes more naturally to high dimension; as maintaining a set containing the optimal step-size, and using bisection to narrow down the size of the set. Starting with an interval S = [0 , α max ] containing 1 / L , we pick a candidate step-size α by 'backtracking' by γ < 1 from the largest step-size in S , taking α = γα max to balance two properties;

Large progress: If the candidate step-size satisfies the Armijo condition and the step is accepted, the value of f decreases proportionally to α as in (3). To maximize the progress, γ should be large.

Volume shrinkage: If the candidate step-size fails the Armijo condition, we learn that α > 1 / L and can cut the interval to S ′ = [0 , γα max ] . To ensure the interval shrinks fast, γ should be small.

Taking γ = 1 / 2 balances both properties; α is at least 1 / 2 as large as any step-size in S , and we can halve the interval if the Armijo condition fails. We do not use α max as a candidate since, although the largest in S , it would give no information to update the interval in case it failed the Armijo condition.

For multidimensional backtracking, we can check whether a candidate preconditioner yields sufficient progress with Equation (4) instead of the Armijo condition, and replace the intervals by sets of diagonal preconditioners. The high-level pseudocode is given in Figure 2, where each iteration either leads to an improvement in function value or shrinks the sets of potential step-sizes/preconditioners.

Backtracking line-search

Multidimensional Backtracking

To complete the algorithm, we need to define the steps marked as ( † ) to select preconditioners that lead to large progress when the step is accepted, while significantly reducing the search space when the preconditioner does not yield sufficient progress. For computational efficiency, we want methods that take O ( d ) time and memory like plain gradient descent.

3.1 Guaranteed progress competitive with the optimal preconditioner

We start by formalizing the progress guarantee. If P t satisfies the Armijo condition (4) at x t , the function value decreases by at least ∥∇ f ( x t ) ∥ 2 P t . If we can guarantee that ∥∇ f ( x t ) ∥ 2 P t ≥ γ ∥∇ f ( x t ) ∥ 2 P ∗ for some γ > 0 , we can recover the convergence rate of gradient descent preconditioned with P ∗ up to a factor of γ . However, we do not know P ∗ , but know a set S t that contains preconditioners we have not yet ruled out, including P ∗ . To guarantee that P t is competitive with P ∗ , we can enforce that P t is competitive with all the preconditioners in S t , as captured by the following definition.

If P t is γ -competitive, then it is competitive with P ∗ as max Q ∈S t ∥∇ f ( x t ) ∥ 2 Q ≥ ∥∇ f ( x t ) ∥ 2 P ∗ . However, this is a strong requirement. To illustrate what competitive ratios are attainable, we show in Appendix B that even the optimal preconditioner P ∗ might only be 1 / d -competitive, as other preconditioners can lead to more local progress depending on ∇ f ( x t ) , whereas P ∗ is a fixed global optimal preconditioner. This also suggests that selecting a preconditioner that guarantees more local progress may lead to better performance, which we take advantage of to ensure a γ = 1 / √ 2 d competitive ratio.

Definition 3.1 ( γ -competitive candidate preconditioners) . A matrix P t ∈ S t is γ -competitive in S t , for a gradient ∇ f ( x t ) , if ∥∇ f ( x t ) ∥ 2 P t ≥ γ ∥∇ f ( x t ) ∥ 2 Q for any Q ∈ S t .

To see how to ensure a competitive ratio, consider the case where S contains diagonal preconditioners whose diagonals come from the box B ( b ) := { p ∈ R d ≥ 0 : p ≤ b } . To select a candidate preconditioner that is γ -competitive in S , we can backtrack from the largest vector in B ( b ) by some constant γ < 1 , and take P = γ Diag( b ) . While a large γ leads to more progress when the step is accepted, we will see that we need a small γ to ensure the volume shrinks when the step is rejected.

We can obtain the convergence rate of Theorem 1.1 depending on γ and the optimal preconditioned condition number κ ∗ if we ensure P ∗ ∈ S t and that P t is γ -competitive for all t .

Proposition 3.2. Let P ∗ , κ ∗ be an optimal preconditioner and condition number for f (1) . If the set S t from the algorithm in Figure 2 contains P ∗ , and P t ∈ S t is γ -competitive (Definition 3.1), then whenever the candidate step leads to sufficient progress and is accepted.

Proof. The proof relies on three inequalities. (a) The iterate x t +1 yields sufficient progress (Eq. 4), (b) any accepted preconditioner P t is γ -competitive in S t and thus with P ∗ , and (c) f is 1 / κ ∗ -strongly convex in ∥·∥ P -1 ∗ , which implies κ ∗ 1 2 ∥∇ f ( x t ) ∥ 2 P ∗ ≥ f ( x t ) -f ( x ∗ ) . Combining those yields

Subtracting f ( x ∗ ) on both sides yields the contraction guarantee.

.

P

.

P

∗

0

(a) Failing the Armijo condition cuts the interval in half in one dimension, but only removes 1 / 2 d of the volume in d dimensions.

.

P

.

P

∗

v

0

.

P

.

P

∗

v

u

0

4 Separating hyperplanes in higher dimensions

In one dimension, if the step-size α does not satisfy the sufficient progress condition (3), we know α > 1 / L and can rule out any α ′ ≥ α . We are looking for a generalization to higher dimensions: if the queried preconditioner fails the sufficient progress condition, we should be able to discard all larger preconditioners. The notion of valid preconditioners formalizes this idea.

Definition 4.1 (Valid preconditioner) . A preconditioner P is valid if P 1 / 2 ∇ 2 f ( x ) P 1 / 2 ⪯ I for all x , which guarantees that P satisfies the sufficient progress (4) condition, and invalid otherwise.

Validity is a global property: a preconditioner P might lead to sufficient progress locally but still be invalid. Using the partial order, if P is invalid then any preconditioner P ′ ⪰ P is also invalid. However, this property alone only discards an exceedingly small portion of the feasible region in high dimensions. Consider the example illustrated in Figure 3a: if the diagonals are in a box B ( b ) , the fraction of volume discarded in this way if ( 1 / 2 ) Diag( b ) is invalid is only 1 / 2 d .

To efficiently search for valid preconditioners, we show that if f is convex, then the gradient of the sufficient progress condition gives a separating hyperplane for valid preconditioners. That is, it gives a vector u ∈ R d such that if p ∈ R d ≥ 0 satisfies ⟨ u , p ⟩ > 1 , then Diag( p ) is invalid, as illustrated in Figure 3b. We use the following notation to denote normalized half-spaces:

Proposition 4.2 (Separating hyperplane in preconditioner space) . Suppose Q = Diag( q ) ≻ 0 does not lead to sufficient progress (4) at x , and let h ( q ) be the gap in the sufficient progress condition,

Then Diag( p ) for any p in the following half-space satisfies h ( p ) > 0 and is also invalid,

This half-space is equal to H > ( v ) with v given by v = ∇ h ( q ) / ( ⟨∇ h ( q ) , q ⟩-h ( q )) , or

Proof idea. If f is convex, then h also is. Convexity guarantees that h ( p ) ≥ h ( q ) + ⟨∇ h ( q ) , p -q ⟩ for any p . A sufficient condition for h ( p ) > 0 , which means p is invalid, is whether h ( q ) + ⟨∇ h ( q ) , p -q ⟩ > 0 holds. Reorganizing yields Equation (5), and Equation (6) expresses the halfspace in normalized form, H > ( v ) , expanding h in terms of f , its gradients, and Q .

The half-space in Proposition 4.2 is however insufficient to find good enough cutting-planes, as it uses convexity to invalidate preconditioners but ignores the ordering that if P is invalid, any P ′ ⪰ P is also invalid. If such preconditioners are not already ruled out by convexity, we can find a stronger half-space by removing them, as illustrated in Figure 3c. We defer proofs to Appendix C.

Proposition 4.3 (Stronger hyperplanes) . If H > ( v ) is a half-space given by Proposition 4.2, then H > ( u ) where u := max { v , 0 } element-wise is a stronger half-space in the sense that H > ( v ) ⊆ H > ( u ) , and H > ( u ) contains only invalid preconditioners.

.

B

(

b

t

+1

)

.

.

P

t

u

t

B

(

b

t

)

.

.

P

t

u

t

B

(

b

t

)

(a) Minimum-volume box containing the intersection. We maintain sets of low-complexity by computing the minimumvolume box B ( b t +1 ) containing the intersection of the initial box B ( b t ) and the half-space H ≤ ( u t ) obtained from Proposition 4.3 when the preconditioner P t fails to yield sufficient decrease.

(b) Need sufficient backtracking. If the candidate preconditioner P t selected inside the initial box B ( b t ) is not close enough to 0, there might not be a box smaller than B ( b t ) that contains the intersection.

5 Cutting-plane methods

The multidimensional backtracking method is in fact a cutting-plane method that uses separating hyperplanes (from Proposition 4.3) to search for valid preconditioners. The canonical example is the ellipsoid method (Yudin and Nemirovski, 1976; Shor, 1977), but its computational cost is Ω( d 2 ) in R d . We now describe cutting-plane methods with three desirable properties: the preconditioners have good competitive ratios, the feasible set shrinks significantly when backtracking, and the computational cost is O ( d ) . There are many details, but the overall idea is similar to the ellipsoid method.

A simple warm-up: boxes. Consider the case when S 0 consists of diagonal matrices with diagonals in the box B ( b 0 ) = { p ∈ R d ≥ 0 : p ≤ b 0 } . We pick a candidate preconditioner by backtracking from the largest point in B ( b 0 ) by some constant γ < 1 , taking P := γ Diag( b 0 ) . If P satisfies the Armijo condition (4), we take a gradient step. If it does not, we compute the vector u 0 as in Proposition 4.3, and obtain a half-space H > ( u 0 ) that contains only invalid preconditioners. We then know we only need to search inside S 0 ∩ H ≤ ( u 0 ) . However, maintaining the set S 0 ∩ H ≤ ( u 0 ) ∩ · · · ∩ H ≤ ( u t ) would be too complex to fit in O ( d ) time or memory. To reduce complexity, we define S t +1 as the box B ( b t +1 ) of minimum volume containing B ( b t ) ∩ H ≤ ( u t ) , as illustrated in Figure 4a. Due to this restriction, we might not be able to find a smaller set; the original box B ( b t ) may already be the minimum volume box containing B ( b t ) ∩ H ≤ ( u t ) if u t does not cut deep enough, as illustrated in Figure 4b. However, with enough backtracking ( γ < 1 / d ), we can show that the new box is smaller. This yields the following subroutines to fill in the gaps of Figure 2 (detailed in Appendix D)

Theorem 5.1. Consider the multidimensional backtracking from Figure 2 initialized with a set S 0 = { Diag( p ) : p ∈ B ( b 0 ) } containing P ∗ , with the subroutines in Equation (7) with γ = 1 / 2 d . Then: (a) P ∗ ∈ S t , (b) the candidate preconditioner P t is 1 / 2 d -competitive in S t for any t , and where S t = B ( b t ) and b t +1 := min { b t , 1 / u t } element-wise, which give the following guarantees.

In particular, CUT is not called more than d log d +1 ( L ∥ b 0 ∥ ∞ ) times.

Proof idea. To guarantee that the box shrinks, we have to guarantee that the half-space H ≤ ( u t ) cuts deep enough. We know that the half-space has to exclude the query point P t , i.e. ⟨ p t , u t ⟩ ≥ 1 , by Proposition 4.2 and that u t ≥ 0 by Proposition 4.3. Querying P t sufficiently close to the origin, by taking γ = 1 / 2 d , is then enough to guarantee the decrease. To bound the total number of cuts, we note that the sets B ( b t ) have a minimum volume Vol min , as they have to contain the valid preconditioners. The number of cuts is at most log c (Vol( B ( b 0 )) / Vol min ) for c = d +1 . We then bound Vol( B ( b 0 )) ≤ ∥ b 0 ∥ d ∞ and Vol min ≥ 1 / L d as ( 1 / L ) I is a valid preconditioner.

5.1 Multidimensional Backtracking with Centered Axis-aligned Ellipsoids

We now improve the competitive ratio from O ( 1 / d ) to O ( 1 / √ d ) by switching from boxes to ellipsoids. Whereas general ellipsoids would require Ω( d 2 ) complexity (as they involve a d × d matrix), we consider centered, axis-aligned ellipsoids , defined by a diagonal matrix A = Diag( a ) , of the form E ( a ) := { p ∈ R d ≥ 0 : ∥ p ∥ A ≤ 1 } , where ∥ p ∥ 2 A = ⟨ p , Ap ⟩ . As preconditioners are non-negative, we consider only the positive orthant of the ellipsoid. For simplicity, we refer to those sets as ellipsoids .

Candidate preconditioner. In the box example, we selected the candidate preconditioner by backtracking from the largest preconditioner in the box. With an ellipsoid, there is no largest preconditioner. We need to choose where to backtrack from. To ensure the candidate preconditioner P is competitive (Definition 3.1), we backtrack from the preconditioner that maximizes the progress ∥∇ f ( x ) ∥ 2 P ,

This lets us pick the preconditioner that makes the most progress for the current gradient , and will let us improve the competitive ratio by allowing a backtracking coefficient of 1 / √ d instead of 1 / d .

Cutting. To complete the algorithm, we need to find a new set E ( b t +1 ) with smaller volume which contains the intersection of the previous set E ( b t ) and the half-space H ≤ ( u t ) . Unlike the box approach, the minimum volume ellipsoid has no closed form solution. However, if we backtrack sufficiently, by a factor of γ < 1 / √ d , we can find an ellipsoid guaranteed to decrease the volume.

Lemma 5.2. Consider the ellipsoid E ( a ) defined by A = Diag( a ) for a ∈ R d > 0 . Let p ∈ E ( a ) be a point sufficiently deep inside the ellipsoid, such that ∥ p ∥ A ≤ 1 / √ 2 d , and H > ( u ) be a half-space obtained from Proposition 4.3 at p . The intersection E ( a ) ∩ H ( u ) ≤ is contained in the new ellipsoid

Proof idea. The new ellipsoid in (9) is a convex combination between E ( a ) and the minimum volume ellipsoid containing the set { p ∈ R d : ⟨ u , | p |⟩ ≤ 1 } where | p | is the element-wise absolute value of p . The choice of λ in (9) is not optimal, but suffices to guarantee progress as long as ∥ p ∥ A is small. A similar approach was used by Goemans et al. (2009) to approximate submodular functions, although they consider the polar problem of finding a maximum-volume enclosed ellipsoid. The full proof and discussion on the connections to the polar problem are deferred to Appendix D.

To improve the cuts, we can refine the estimate of λ in Lemma 5.2 by minimizing the volume numerically. We include this modification, detailed in Appendix D, in our experiments in Section 6.

Overall guarantees. We can now define the two subroutines for the ellipsoid method, and obtain the main result that we stated informally in Theorem 1.1, by combining the guarantees of the ellipsoid approach with the convergence result of Proposition 3.2.

Theorem 5.3. Consider the multidimensional backtracking from Figure 2 initialized with the set S 0 = { Diag( p ) : p ∈ E ( a 0 ) } containing P ∗ , given by some scaling α 0 > 0 of the uniform vector, a 0 = α 0 1 . For S t , let A t = Diag( a t ) . Define the subroutines

where u t is the vector given by Proposition 4.3 when P t fails the Armijo condition at x t , and a + is computed as in (9) . If γ = 1 / √ 2 d , then: (a) P ∗ ∈ S t for all t , (b) the candidate preconditioners P t are 1 / √ 2 d -competitive in S t , and (c) CUT is called no more than 12 d log( L/α 0 ) times.

6 Experiments

To illustrate that multidimensional backtracking finds good preconditioners and improves over gradient descent on ill-conditioned problems even when accounting for the cost of backtracking, we run experiments on small but very ill-conditioned and large ( d ≈ 10 6 ) problems.

As examples of adaptive gain and hypergradient methods, we include RPROP (Riedmiller and Braun, 1993) and GD with a hypergradient-tuned step-size (GD-HD, Baydin et al. 2018, multiplicative update). As examples of approximate second-order methods, we include diagonal BB (Park et al., 2020) and preconditioned GD using the diagonal of the Hessian. We use default parameters, except for the hypergradient method GD-HD, where we use 10 -10 as the initial step-size instead of 10 -3 to avoid immediate divergence. We include AdaGrad (diagonal), but augment it with a line-search as suggested by Vaswani et al. (2020), to make it competitive in the deterministic setting.

Line-searches and forward steps. For all methods that use a line-search, we include a forward step, a common heuristic in line-search procedures to allow for larger step-sizes when possible, although it

0

500

1000

1500

# f/grad evals

10

2

10 0

10 2

10 4

Optimality gap

Linear - cpusmall

0

500

1000

# f/grad evals

10

5

10

3

10

1

10 1

Logistic - Breast-cancer

0

500

1000

1500

# f/grad evals

10

5

10

3

10

1

10 1

Logistic - News20

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

can increase the number of backtracking steps. When a step-size or preconditioner is accepted, we increase the size of the set, allowing for larger (scalar or per-coordinate) step-sizes by a factor of 1 . 1 .

Performance comparison. To capture the overhead of backtracking and provide a fair evaluation, Figures 1 and 5 compare performance per function and gradient evaluations (see Appendix E.1).

On a small but extremely ill-conditioned problems , our method is the only one that gets remotely close to being competitive with preconditioning with the diagonal Hessian-while only using firstorder information. The diagonal Hessian is very close to the optimal preconditioner for those problems. On the cpusmall dataset, it reduces the condition number from κ ≈ 5 · 10 13 to ≈ 300 , while κ ∗ ≈ 150 . All other methods struggle to make progress and stall before a reasonable solution is achieved, indicating they are not competitive with the optimal preconditioner.

On large regularized logistic regression on News20 ( d ≈ 10 6 ) , gradient descent performs relatively better, suggesting the problem is less ill-conditioned to begin with (the regularized data matrix has condition number κ ≈ 10 4 ). Despite the bound of O ( d ) backtracking steps, our methods finds a reasonable preconditioner within 100 gradient evaluations. Despite the high dimensionality, it improves over gradient descent when measured in number of oracle calls.

Using plain gradient updates on the hyperparameters in GD-HD leads to unstable behavior, but diagonal BB and even RPROP, perform remarkably well on some problems-even outperforming preconditioning with the diagonal Hessian, which uses second-order information. However, they fail on other ill-conditioned problems, even when a good diagonal preconditioner exists. This pattern holds across other problems, as shown in Appendix E. Multidimensional backtracking demonstrates robust performance across problems, a clear advantage of having worst-case guarantees.

7 Conclusion

We designed multidimensional backtracking , an efficient algorithm to automatically find diagonal preconditioners that are competitive with the optimal diagonal preconditioner. Our work provides a definition of adaptive step-sizes that is complementary to the online learning definition. While online learning focuses on the adversarial or highly stochastic setting, we define and show how to find optimal per-coordinate step-sizes in the deterministic smooth convex setting. We show it is possible to build provably robust methods to tune a preconditioner using hypergradients. While our specific implementation uses cutting-planes, the general approach may lead to alternative algorithms, that possibly tune other hyperparameters, with similar guarantees.

The main limitation of our approach is its reliance on the convex deterministic setting. The results might transfer to the stochastic overparametrized regime using the approach of Vaswani et al. (2019), but the non-convex case seems challenging. It is not clear how to get reliable information from a cutting-plane perspective using hypergradients without convexity. As the first method to provably find competitive preconditioners, there are likely modifications that lead to practical improvements while preserving the theoretical guarantees. Possible ideas to improve practical performances include better ways to perform forward steps, using hypergradient information from accepted steps (which are currently ignored), or considering alternative structures to diagonal preconditioners.

Acknowledgments and Disclosure of Funding

We thank Aaron Mishkin for helpful discussions in the early stages of this work, and Curtis Fox and Si Yi (Cathy) Meng for providing comments on an early version of the manuscript, and for the feedback from anonymous reviewers. This research was partially by the Canada CIFAR AI Chair Program, the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grants RGPIN-2022-03669, and Borealis AI through the Borealis AI Global Fellowship Award.

References

Naman Agarwal, Rohan Anil, Elad Hazan, Tomer Koren, and Cyril Zhang (2020). Disentangling Adaptive Gradient Methods from Learning Rates . arXiv/2002.11803.

Luís B. Almeida, Thibault Langlois, José F. M. Amaral, and Alexander Plakhov (1999). 'Parameter adaptation in stochastic optimization'. In: On-line learning in neural networks , pp. 111-134.

Ehsan Amid, Rohan Anil, Christopher Fifty, and Manfred K. Warmuth (2022). Step-size Adaptation Using Exponentiated Gradient Updates . Presented at the ICML Workshop Beyond First Order Methods in ML Systems.

Neculai Andrei (2019). 'A diagonal quasi-Newton updating method for unconstrained optimization'. In: Numerical Algorithms 81.2, pp. 575-590.

Larry Armijo (1966). 'Minimization of functions having Lipschitz continuous first partial derivatives'. In: Pacific Journal of mathematics 16.1, pp. 1-3.

Juhan Bae, Paul Vicol, Jeff Z. HaoChen, and Roger B. Grosse (2022). 'Amortized Proximal Optimization'. In: NeurIPS , pp. 8982-8997.

Andrew G. Barto and Richard S. Sutton (1981). Goal Seeking Components for Adaptive Intelligence: An Initial Assessment. Tech. rep. AFWAL-TR-81-1070. (Appendix C). Air Force Wright Aeronautical Laboratories/Avionics Laboratory.

Atılım Güne¸ s Baydin, Robert Cornish, David Martínez-Rubio, Mark Schmidt, and Frank Wood (2018). 'Online Learning Rate Adaptation with Hypergradient Descent'. In: ICLR .

Albert S. Berahas, Majid Jahani, Peter Richtárik, and Martin Takáˇ c (2022). 'Quasi-Newton methods for machine learning: forget the past, just sample'. In: Optimization Methods and Software 37.5, pp. 1668-1704.

Dimitri P. Bertsekas (1999). Nonlinear Programming . Athena Scientific.

Raghu Bollapragada, Dheevatsa Mudigere, Jorge Nocedal, Hao-Jun Michael Shi, and Ping Tak Peter Tang (2018). 'A Progressive Batching L-BFGS Method for Machine Learning'. In: ICML , pp. 619628.

Richard H. Byrd, Samantha L. Hansen, Jorge Nocedal, and Yoram Singer (2016). 'A Stochastic Quasi-Newton Method for Large-Scale Optimization'. In: SIAM Journal on Optimization 26.2, pp. 1008-1031.

Kartik Chandra, Audrey Xie, Jonathan Ragan-Kelley, and Erik Meijer (2022). 'Gradient Descent: The Ultimate Optimizer'. In: NeurIPS , pp. 8214-8225.

Chih-Chung Chang and Chih-Jen Lin (2011). 'LIBSVM: A Library for Support Vector Machines'. In: ACM Transatctions on Intelligent Systems and Technology 2.3.

Aaron Defazio, Baoyu Zhou, and Lin Xiao (2022). Grad-GradaGrad? A Non-Monotone Adaptive Stochastic Gradient Method . arXiv/2206.06900.

Alexandre Défossez, Leon Bottou, Francis Bach, and Nicolas Usunier (2022). 'A Simple Convergence Proof of Adam and Adagrad'. In: Transactions on Machine Learning Research .

John E. Dennis Jr. and Jorge J. Moré (1977). 'Quasi-Newton methods, motivation and theory'. In: SIAM review 19.1, pp. 46-89.

Steven Diamond and Stephen Boyd (2016). 'CVXPY: A Python-embedded modeling language for convex optimization'. In: Journal of Machine Learning Research 17.83, pp. 1-5.

Dheeru Dua and Casey Graff (2017). UCI Machine Learning Repository .

John C. Duchi, Elad Hazan, and Yoram Singer (2011). 'Adaptive Subgradient Methods for Online Learning and Stochastic Optimization'. In: Journal of Machine Learning Research 12, pp. 21212159.

Gary William Flake and Steve Lawrence (2002). 'Efficient SVM Regression Training with SMO'. In: Mach. Learn. 46.1-3, pp. 271-290.

Michel X. Goemans, Nicholas J. A. Harvey, Satoru Iwata, and Vahab Mirrokni (2009). 'Approximating submodular functions everywhere'. In: ACM-SIAM SODA 2009 , pp. 535-544.

Luigi Grippo, Francesco Lampariello, and Stephano Lucidi (1986). 'A Nonmonotone Line Search Technique for Newton's Method'. In: SIAM Journal on Numerical Analysis 23.4, pp. 707-716.

Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky (2012). Neural Networks for Machine Learning . lecture 6.

Robert A. Jacobs (1988). 'Increased rates of convergence through learning rate adaptation'. In: Neural Networks 1.4, pp. 295-307.

Majid Jahani, Sergey Rusakov, Zheng Shi, Peter Richtárik, Michael W. Mahoney, and Martin Takac (2022). 'Doubly Adaptive Scaled Algorithm for Machine Learning Using Second-Order Information'. In: ICLR .

Arun Jambulapati, Jerry Li, Christopher Musco, Aaron Sidford, and Kevin Tian (2020). Fast and Near-Optimal Diagonal Preconditioning . arXiv/2008.01722.

Hamed Karimi, Julie Nutini, and Mark Schmidt (2016). 'Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition'. In: ECML , pp. 795-811.

Sathiya Keerthi and Dennis DeCoste (2005). 'A Modified Finite Newton Method for Fast Solution of Large Scale Linear SVMs'. In: Journal of Machine Learning Research 6.12, pp. 341-361.

Kelley Pace and Ronald Barry (1997). 'Sparse spatial autoregressions'. In: Statistics & Probability Letters 33.3, pp. 291-297.

Harry Kesten (1958). 'Accelerated Stochastic Approximation'. In: The Annals of Mathematical Statistics 29.1, pp. 41-59.

Diederik P. Kingma and Jimmy Ba (2015). 'Adam: A Method for Stochastic Optimization'. In: ICLR .

Etienne de Klerk, François Glineur, and Adrien B. Taylor (2017). 'On the worst-case complexity of the gradient method with exact line search for smooth strongly convex functions'. In: Optimization Letters 11.7, pp. 1185-1199.

Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller (2012). 'Efficient BackProp'. In: Neural Networks: Tricks of the Trade - Second Edition . Vol. 7700, pp. 9-48.

David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li (2004). 'RCV1: A New Benchmark Collection for Text Categorization Research'. In: Journal of Machine Learning Research 5, pp. 361-397.

Xiaoyu Li and Francesco Orabona (2019). 'On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes'. In: AISTATS 2019 , pp. 983-992.

Dong C. Liu and Jorge Nocedal (1989). 'On the limited memory BFGS method for large scale optimization'. In: Mathematical programming 45.1-3, pp. 503-528.

Łojasiewicz (1963). 'Une propriété topologique des sous-ensembles analytiques réels'. In: Les Équations aux Dérivées Partielles , pp. 87-89.

Dougal Maclaurin, David Duvenaud, and Ryan P. Adams (2015). 'Gradient-based Hyperparameter Optimization through Reversible Learning'. In: ICML , pp. 2113-2122.

Ashique Rupam Mahmood, Richard S. Sutton, Thomas Degris, and Patrick M. Pilarski (2012). 'Tuning-free step-size adaptation'. In: ICASSP , pp. 2121-2124.

Pierre-Yves Masse and Yann Ollivier (2015). Speed learning on the fly . arXiv/1511.02540.

Brendan McMahan and Matthew J. Streeter (2010). 'Adaptive Bound Optimization for Online Convex Optimization'. In: COLT , pp. 244-256.

Si Yi Meng, Sharan Vaswani, Issam Hadj Laradji, Mark Schmidt, and Simon Lacoste-Julien (2020). 'Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation'. In: AISTATS , pp. 1375-1386.

Jorge J. Moré and David J. Thuente (1994). 'Line search algorithms with guaranteed sufficient decrease'. In: ACM Transactions on Mathematical Software (TOMS) 20.3, pp. 286-307.

Ted Moskovitz, Rui Wang, Janice Lan, Sanyam Kapoor, Thomas Miconi, Jason Yosinski, and Aditya Rawal (2019). First-Order Preconditioning via Hypergradient Descent . arXiv/1910.08461.

Yurii E. Nesterov and Boris T. Polyak (2006). 'Cubic regularization of Newton method and its global performance'. In: Mathematical Programming 108.1, pp. 177-205.

Jorge Nocedal and Stephen J. Wright (1999). Numerical Optimization . Springer.

Francesco Orabona and Dávid Pál (2016). 'Coin Betting and Parameter-Free Online Learning'. In: NeurIPS , pp. 577-585.

Francesco Orabona and Tatiana Tommasi (2017). 'Training Deep Networks without Learning Rates Through Coin Betting'. In: NeurIPS , pp. 2160-2170.

Youngsuk Park, Sauptik Dhar, Stephen P. Boyd, and Mohak Shah (2020). 'Variable Metric Proximal Gradient Method with Diagonal Barzilai-Borwein Stepsize'. In: ICASSP , pp. 3597-3601.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala (2019). 'PyTorch: An Imperative Style, High-Performance Deep Learning Library'. In: NeurIPS , pp. 8024-8035.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay (2011). 'Scikit-learn: Machine Learning in Python'. In: Journal of Machine Learning Research 12, pp. 2825-2830.

Vassilis P. Plagianakos, George D. Magoulas, and Michael N. Vrahatis (2001). 'Learning rate adaptation in stochastic gradient descent'. In: Advances in Convex Analysis and Global Optimization: Honoring the Memory of C. Caratheodory (1873-1950) , pp. 433-444.

Boris T. Polyak (1963). 'Gradient methods for minimizing functionals'. In: Ž. Vyˇ cisl. Mat i Mat. Fiz. 3, pp. 643-653.

Zhaonan Qu, Wenzhi Gao, Oliver Hinder, Yinyu Ye, and Zhengyuan Zhou (2022). Optimal Diagonal Preconditioning: Theory and Practice . arXiv/2209.00809.

Zheng Qu, Peter Richtárik, Martin Takác, and Olivier Fercoq (2016). 'SDNA: Stochastic Dual Newton Ascent for Empirical Risk Minimization'. In: ICML , pp. 1823-1832.

Martin A. Riedmiller and Heinrich Braun (1993). 'A direct adaptive method for faster backpropagation learning: the RPROP algorithm'. In: ICNN , pp. 586-591.

Mher Safaryan, Filip Hanzely, and Peter Richtárik (2021). 'Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization'. In: NeurIPS , pp. 25688-25702.

George N. Saridis (1970). 'Learning Applied to Successive Approximation Algorithms'. In: IEEE Transactions on Systems Science and Cybernetics 6.2, pp. 97-103.

Nicol N. Schraudolph (1999). 'Local gain adaptation in stochastic gradient descent'. In: ICANN , 569-574 vol.2.

Nicol N. Schraudolph, Douglas Aberdeen, and Jin Yu (2005). 'Fast Online Policy Gradient Learning with SMD Gain Vector Adaptation'. In: NeurIPS , pp. 1185-1192.

Naum Z. Shor (1977). 'Cut-off method with space extension in convex programming problems'. In: Cybernetics 13.1, pp. 94-96.

Fernando M. Silva and Luís B. Almeida (1990). 'Acceleration techniques for the backpropagation algorithm'. In: Neural Networks , pp. 110-119.

Richard S. Sutton (1992a). 'Adapting Bias by Gradient Descent: An Incremental Version of DeltaBar-Delta'. In: Proceedings of the 10th National Conference on Artificial Intelligence, San Jose, CA, USA, July 12-16, 1992 , pp. 171-176.

Richard S. Sutton (1992b). 'Gain adaptation beats least squares'. In: Proceedings of the 7th Yale workshop on adaptive and learning systems , p. 166.

Pınar Tüfekci (2014). 'Prediction of full load electrical power output of a base load operated combined cycle power plant using machine learning methods'. In: International Journal of Electrical Power &Energy Systems 60, pp. 126-140.

Sharan Vaswani, Frederik Kunstner, Issam H. Laradji, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien (2020). Adaptive Gradient Methods Converge Faster with Over-Parameterization (and you can do a line-search) . arXiv/2006.06835.

Sharan Vaswani, Aaron Mishkin, Issam H. Laradji, Mark Schmidt, Gauthier Gidel, and Simon Lacoste-Julien (2019). 'Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates'. In: NeurIPS , pp. 3727-3740.

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors (2020). 'SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python'. In: Nature Methods 17, pp. 261-272.

Rachel Ward, Xiaoxia Wu, and Léon Bottou (2019). 'AdaGrad stepsizes: sharp convergence over nonconvex landscapes'. In: ICML , pp. 6677-6686.

Philip Wolfe (1969). 'Convergence conditions for ascent methods'. In: SIAM review 11.2, pp. 226235.

Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney (2021). 'AdaHessian: An Adaptive Second Order Optimizer for Machine Learning'. In: AAAI 35.12, pp. 10665-10673.

I.-Cheng Yeh (1998). 'Modeling of strength of high-performance concrete using artificial neural networks'. In: Cement and Concrete Research 28.12, pp. 1797-1808.

David B. Yudin and Arkadi S. Nemirovski (1976). 'Informational complexity and effective methods for the solution of convex extremal problems'. In: Èkonom. i Mat. Metody 12.2, pp. 357-369.

Ciyou Zhu, Richard H. Byrd, Peihuang Lu, and Jorge Nocedal (1997). 'Algorithm 778: L-BFGS-B: Fortran Subroutines for Large-Scale Bound-Constrained Optimization'. In: ACM Transactions on Mathematical Software 23.4, pp. 550-560.

Zhu, John L. Nazareth, and Henry Wolkowicz (1999). 'The Quasi-Cauchy Relation and Diagonal Updating'. In: SIAM Journal on Optimization 9.4, pp. 1192-1204.

Contents

E.2

Additional results

Code available at

Supplementary Material

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

38

https://github.com/fKunstner/multidimensional-backtracking

A Full pseudocode of the algorithms

We first give with a generic version using the subroutines INITIALIZE, CANDIDATE and CUT, to be specialized for the backtracking line-search (Figure 8), multidimensional backtracking using boxes (Figure 10), and ellipsoids (Figure 11). The generic pseudocode is written in terms of preconditioners, but also applies to the step-size version, which we can consider as looking for a preconditioner constrained to isotropic diagonal preconditioners, that is, preconditioners in the set { α I : α ∈ R ≥ 0 } .

Although we write the pseudocode maintaining at each iteration an abstract set of preconditioners S , the only information the algorithm needs to maintain on each iteration for the implementation in the different cases is the current maximum step-size α max defining the interval of valid step-sizes, [0 , α max ] such that the set of preconditioners is = α I : α [0 , α ] ;

· For multidimensional backtracking with boxes: the vector b defining the maximum corner of the box B ( b ) = { p ∈ R d ≥ 0 : p ≤ b } used to define the candidate diagonals preconditioners in the set S = { Diag( p ) : p ∈ B ( b ) } ;

For the line-search: S { ∈ max }

· For multidimensional backtracking with ellipsoids: the vector a defining the axis-aligned ellipsoid E ( a ) = { p ∈ R d ≥ 0 : ⟨ p , Diag( a ) p ⟩ ≤ 1 } used to define the candidate diagonal preconditioners in the set S = { Diag( p ) : p ∈ E ( a ) } .

We give an alternative pseudocode in Figure 7, which defines iterations as updates to the iterates x t that decrease the function value, and uses a while -loop to backtrack. Since it more closely resemble standard ways backtracking line-search is described, some reader may find it easier to understand. We stress, however, that this is still the same algorithm as Figure 6 but written differently.

The pseudocode in Figure 6 updates ( x t , S t ) to ( x t +1 , S t +1 ) at each iteration, and ensures that either the function value decreases, f ( x t +1 ) < f ( x t ) , or the volume decreases, Vol( S t +1 ) < Vol( S t ) .

The pseudocode in Figures 6-11, are expressed in a modular form to highlight how the algorithm works and its similarity to a line-search. In Appendix A.5, we give a more directly implementable pseudocode of multidimensional backtracking in both box and ellipsoid variants solely relying on vector notation.

A.1 Subroutines for standard backtracking line-search

Implementation of the subroutines for the standard backtracking line-search. Although written in terms of sets, the algorithm only needs to maintain the maximum step-size in the interval [0 , α max ] at each iteration. The corresponding preconditioners are the matrices S = { α I : α ∈ [0 , α max ] } .

INITIALIZE ( c 0 ) Input: A scaling c 0 larger than the optimum step-size, that is, such that 1 / L ≤ c 0 . Output: Set of preconditioners S = { α I : α ∈ [0 , c 0 ] } .

CANDIDATE ( S , γ, x ) Input: Set of scalar preconditioners S = { α I : α ∈ [0 , α max ] } ; Backtracking coefficient γ ∈ [0 , 1] ; Current iterate x ∈ R d . / / Not used for the step-size version. Output: Preconditioner γα max I

CUT ( S , x , P ) Input: Set of scalar preconditioners S = { α I : α ∈ [0 , α max ] } ; Current iterate x ∈ R d ; / / Not used for the step-size version. Preconditioner P = α bad I that failed the Armijo condition at x . Output: Set of scalar preconditioners with reduced interval, S = { α I : α ∈ [0 , α bad ] }

A.2 Separating hyperplanes used by multidimensional backtracking

Both versions of multidimensional backtracking need a direction to update the set of preconditioners in the CUT subroutine. We define the subroutine SEPARATINGHYPERPLANE in Figure 9. The description of the separating hyperplane and their properties can be found in Section 4 and Appendix C.

SEPARATINGHYPERPLANE ( x , P ) for diagonal preconditioners

Input: Current iterate x ∈ R d ; Diagonal preconditioner P = Diag( p ) that failed the Armijo condition at x . x + = x -P ∇ f ( x ) g = ∇ f ( x ) g + = ∇ f ( x + ) v = ( 1 2 g -g + ) ⊙ g f ( x ) -f ( x + ) -⟨ g , Pg + ⟩ / / Separating hyperplane from Proposition 4.2 Output: u = max { v , 0 } element-wise / / Stronger hyperplane from Proposition 4.3

A.3 Multidimensional backtracking using boxes

The implementation of multidimensional backtracking with boxes only needs to maintain a vector b , representing the maximum step-size for each coordinate that has not been ruled out, in the box B ( b ) . The associated sets of preconditioners S are

The description of boxes and the theoretical guarantees when using them in multidimensional backtracking can be found in Section 5 and Appendix D.1. The subroutines used by the algorithm with boxes are:

INITIALIZE: initializes b to c 0 1 so that the diagonal preconditioner c 0 I is in S 0 .

CANDIDATE: backtracks from the largest diagonal in B ( b ) , returning γ Diag( b ) .

SEPARATINGHYPERPLANE: computes the vector u defining the half-space of invalid preconditioners H > ( u ) obtained when the preconditioner P fails the Armijo condition at x as described in Proposition 4.2 and Proposition 4.3.

CUT: returns the minimum volume box B ( b + ) containing the intersection B ( b ) ∩ H ≤ ( u ) .

INITIALIZE ( c 0 )

Input:

A scalar c 0 such that c 0 I is larger than the optimal diagonal preconditioner, i.e., P ∗ ⪯ c 0 I .

Output: Set S = { Diag( p ) : p ∈ B ( b ) } with B ( b ) = { p ∈ R d ≥ 0 : p ≤ b } where b := c 0 1

CANDIDATE ( S , γ, x )

Input: Set of preconditioners S = { Diag( p ) : p ∈ B ( b ) } with B ( b ) = { p ∈ R d ≥ 0 : p ≤ b } ; Backtracking coefficient γ ∈ [0 , 1] ; Current iterate x ∈ R d . / / Not used for the box version. Output: Preconditioner γ Diag( b )

CUT ( S , x , P ) Input: Set of preconditioners S = { Diag( p ) : p ∈ B ( b ) } with B ( b ) = { p ∈ R d ≥ 0 : p ≤ b } ; Backtracking coefficient γ ∈ [0 , 1] ; Preconditioner P bad that failed the Armijo condition at x . u = SEPARATINGHYPERPLANE ( x t , P bad ) b + = max { b , 1 / u } / / Minimum volume box B ( b + ) containing B ( b ) ∩ H ≤ ( u ) Output: Set of diagonal preconditioners S = { Diag( p ) : p ∈ B ( b + ) } .

A.4 Multidimensional backtracking using ellipsoids

The implementation only needs to maintain a vector a representing the diagonal of the matrix defining the (centered, axis-alligned) ellipsoid E ( a ) and the associated set of preconditioners S given by

The description of the ellipsoids and their properties can be found in Section 5 and Appendix D.2. The subroutines used by the algorithm with boxes are:

INITIALIZE: initializes a to ( 1 / dc 2 0 ) 1 so that c 0 1 ∈ E ( a ) , implying the diagonal preconditioner c 0 I is in S .

CANDIDATE: backtracks from the diagonal preconditioner in S that maximizes the gradient norm. Let E ( a ) be the set of candidate diagonals and define A = Diag( a ) . The subroutine returns γ P max , where

Writing this in terms of the diagonal vector p max := diag( P max ) yields

where ∇ f ( x ) 2 = ∇ f ( x ) ⊙∇ f ( x ) .

SEPARATINGHYPERPLANE: computes the vector u defining the half-space of invalid preconditioners H > ( u ) obtained when the preconditioner P fails the Armijo condition at x as described in Proposition 4.2 and Proposition 4.3.

CUT: returns an ellipsoid E ( a + ) containing the intersection of E ( a ) ∩H ≤ ( u ) with guaranteed volume decrease from E ( a ) . As there is no closed-form solution for the minimum volume ellipsoid, we set a + as a convex combination between the original ellipsoid E ( a ) and the minimum volume axis-aligned ellipsoid containing H ≤ ( u ) , given by E ( u 2 ) , that is,

where A := diag( a ) . Although the above choice of λ has guaranteed volume decrease, we can find a better value of λ by solving the minimum volume ellipsoid as a function of λ numerically. Namely, approximating

In our experiments, we start with λ as in (10) and, starting from it, we solve the above minimization problem numerically using L-BFGS-B (Zhu et al., 1997) in SciPy (Virtanen et al., 2020). This preserves the theoretical guarantee while improving empirical performance.

INITIALIZE ( c 0 )

Input:

A scalar c 0 > 0 such that c 0 I is larger than the optimal diagonal preconditioner, i.e., P ∗ ⪯ c 0 I .

CANDIDATE ( S , γ, x )

Input:

Backtracking coefficient γ ∈ [0 , 1] ;

Current iterate x ∈ R d .

Output:

Preconditioner γ Diag( p max )

CUT ( S , x , P )

Input:

Current iterate x ∈ R d ;

Preconditioner P bad that failed the Armijo condition at x .

u = SEPARATINGHYPERPLANE ( x t , P bad )

2

Output: +

The set S = { Diag( p ) : p ∈ E ( a ) }

A.5 Implementable pseudocode

The pseudocode in Figures 6-11 are expressed in a modular form to highlight how the algorithm works and its similarity to a line-search. In this section, we give a more directly implementable pseudocode of multidimensional backtracking, in both the box and ellipsoid variants, using mostly vector notation. Scalar operations on vectors such as u / a , √ u , u 2 are understood to be taken element-wise.

2

1

0

1

2

2

1

0

1

2

Function

0.0

0.4

0.8

1.2

1.6

2.0

0.0

0.4

0.8

1.2

1.6

2.0

Valid Preconditioners

1/L

B Optimal preconditioners, valid preconditioners and competitive ratios

In Section 2, we defined the optimal preconditioner P ∗ as the preconditioner that is the best overall approximation to the inverse Hessian. Formally, we define the optimal diagonal preconditioner P ∗ as

One way to interpret this definition is that P -1 ∗ is the tightest diagonal approximation to ∇ 2 f ( x ) .

We remark that we do not need f to be (strongly-)convex to define the theoretically optimal step-size of 1 / L for gradient descent. Thus, one may wonder why we need strong-convexity (although we relax this to requiring f to be PL in Appendix B.1) to define what an optimal preconditioner is in (1).

The main difference between the scalar step-size and per-coordinate step-sizes settings is whether the 'largest' step-size or preconditioner is well-defined. In the scalar setting, the largest step-size that is guaranteed to lead to progress everywhere (i.e., a step-size that satisfies the Armijo condition (3) for all x ) is well-defined and equal to α ∗ := 1 / L for L -smooth function f . Equivalently,

where λ max ( ∇ 2 f ( x )) is the largest eigenvalue of ∇ 2 f ( x ) . But in the case of preconditioners, the ordering on positive definite matrices is not complete, so there is no single 'largest' preconditioner P that satisfies ∇ 2 f ( x ) ⪯ P -1 . We can still describe 'good' preconditioners, that are guaranteed to satisfy the Armijo condition (Equation (4)) everywhere; this is the notion of valid preconditioners defined in Definition 4.1, which in set notation is V := { P ≻ 0 : ∇ 2 f ( x ) ⪯ P -1 } . With this definition, we can consider the set of valid preconditioners P for which there are no bigger valid preconditioners, that is, P := { P ∈ V : ̸ ∃ P ′ ∈ V s.t. P ≺ P ′ } . However, P contains incomparable preconditioners, that is, distinct matrices A , B ∈ P that neither A ⪰ B nor A ⪯ B hold.

Let us look at an example with a quadratic function (illustrated in Figure 12)

There are many preconditioners that are valid, 3 for example using the per-coordinate step-sizes

The preconditioner P L corresponds to the 1 / L step-size, P 1 and P 2 take the largest possible stepsize in each coordinate, and P ∗ is the optimal preconditioner according to Equation (1). Those preconditioners are not comparable to each other, as neither P L ≺ P ∗ nor P ∗ ≺ P L hold. Instead of looking at the matrices themselves, we use in (1) the condition number 4 of P 1 / 2 ∇ 2 f ( x ) P 1 / 2 as a measure of quality of P . This allows for a well-defined optimal preconditioner as this condition number can be maximized.

B.1 Defining optimal preconditioners without twice-differentiability or strong-convexity

Although we used twice-differentiability of f to define the optimal preconditioner, this is not necessary. If f is not twice-differentiable but still strongly-convex, the definition in Equation (1) can be replaced by Equation (2), as finding the P -norm under which the function is most strongly-convex.

--⟨∇ -⟩ ≤ 2 ∥ -∥ P To avoid strong-convexity, we can instead use the PL inequality. A function f is µ -PL if

This property is implied by µ -strong convexity. We refer to the work of Karimi et al. (2016) for the properties of PL functions and its relation to other assumptions. To adapt Equation (12) to our results, we can measure the PL constant µ in the norm induced by P , and say that f is µ -PL in ∥·∥ P if

We use this inequality in the convergence proof in Proposition 3.2 since it is a consequence of strong-convexity. As this property is the only property of strong-convexity needed for our results, we can adapt our results to be competitive with the optimal preconditioner defined using the PL inequality, using the definition

If f is µ -PL and L -smooth, Equation (14) has a feasible solution at P = 1 / L I number κ = L / µ . The constraint based on the µ -PL condition in Equation (14) is weaker than the definition using strong-convexity, as strong-convexity implies the PL inequality. The optimal preconditioner defined using the PL inequality (14) might thus achieve a lower condition number than the one using strongconvexity (1). For example, the quadratic f ( x ) = ( 1 / 2 ) ⟨ x , Ax ⟩ with a positive semi-definite A is not strongly convex if the smallest eigenvalue of A is 0. The optimal preconditioner in Equation (1) is ill-defined (or has condition number κ ∗ = ∞ ). In contrast, the optimal preconditioner defined using the PL inequality in Equation (14) has a finite condition number, as P = 1 / L I is a feasible solution with condition number κ = L / λ + min ( A ) where λ + min ( A ) is the smallest non-zero eigenvalue of A . As our proofs only use the properties guaranteed by Equation (14), our results also apply to PL functions.

B.2 Valid and optimal preconditioners with singular matrices

In the main text, we defined valid preconditioners (Definition 4.1) only for positive definite matrices for ease of presentation. The notion of valid preconditioners can be extended to general positive semidefinite matrices. In the diagonal case, the convention 1 / 0 = + ∞ is a useful mental model but can cause inconsistencies (such as ∞· 0 ). To extend the notion of valid preconditioners to general positive semidefinite matrices, we can use the definition

The above is well-defined for all positive semidefinite matrices. An alternative to arrive at a definition closer to Definition 4.1 is to consider the projection matrix Π P onto the image of P , given by Π P = P 1 / 2 ( P 1 / 2 ) † where P † is the Moore-Penrose pseudo-inverse of P . Using that, one can show that P is valid (according to Definition B.1) if and only if

An example of a valid preconditioner that is covered by Definition B.1 but not 4.1 is the all-zeroes matrix. Definition B.1 can seamlessly replace 4.1, and all the results follow similarly. Moreover, notice that the optimization problem defining the optimal preconditioner (1) may not attain its minima on positive definite matrices when f is not strongly convex. In this case, we can define an optimal

2

1

0

1

2

2

1

0

1

2

Function

0.00

0.25

0.50

0.75

1.00

1.25

0.00

0.25

0.50

0.75

1.00

1.25

Valid Preconditioners

1/L

preconditioner as a limit point of a sequence that attains in the limit the value in (1) by replacing the minimum with an infimum. In this case, an optimal preconditioner may be singular, but the results in the main body also follow seamlessly using this definition. We decided to restrict our attention to non-singular preconditioners in the main paper for ease of exposition, since when f is strongly-convex, an optimal preconditioner is always non-singular.

B.3 Best competitive ratio achievable by the optimal preconditioner

In Section 3, we mentioned that the optimal preconditioner P ∗ could be only 1 / d -competitive. In fact, the competitive ratio of P ∗ can be arbitrarily bad. The reason for this is that the competitive ratio γ does not compare against P ∗ , but rather against any P in the set S of potentially valid preconditioners. Moreover, this definition only takes into account the norm ∥∇ f ( x ) ∥ P at a fixed x , while the optimal preconditioner needs to have large norm for all x .

For example, consider the scalar step-size case. If our current interval of candidate step-sizes to try is S = [0 , 1] but the optimal step-size α ∗ is small, let us say α ∗ = 1 / 10 , then α ∗ is only 1 / 10 -competitive in S . The motivation for this definition of competitive ratio is that we cannot check whether α is large compared to α ∗ (as we do not know α ∗ ) but we can more easily ensure that a candidate step-size α is γ -competitive in S (for example α = 1 / 2 is 1 / 2 -competitive in [0 , 1] ).

In the previous example, the bad competitive ratio of α ∗ in S was mostly due to the fact that S was large and that, for some x , step sizes larger than α ∗ could satisfy the Armijo condition (3). Even if α ∗ is globally optimal, we could make more progress by using a larger step-size if they were to be accepted, and we have not yet ruled out those step-sizes. However, as S shrinks, it may eventually converge to the interval [0 , 1] , in which case the optimal step-size α ∗ would be 1 -competitive.

In high dimensions however, the optimal preconditioner can have a competitive ratio of 1 / d even when comparing only against valid preconditioners. 5 This is because the competitive ratio is defined using the P -norm of the gradient, and we need to take the direction of the gradient into account. For example, consider the quadratic function (illustrated in Figure 13)

with eigenvalues { 2 , 0 } as A = [ -1 , 1] T [ -1 , 1] . The following three preconditioners are all valid:

The preconditioner P 1 takes the largest possible step-size in the first coordinate and ignores the second, while P 2 does the opposite. They are not good global preconditioners, as each ignores one coordinate. Yet, they can make much more progress (i.e., the objective value may decrease more) than the optimal preconditioner P ∗ if the gradient is very skewed towards one coordinate. This implies that

P ∗ may be only 1 / 2 -competitive in { P 1 , P 2 } for some x since

The preconditioner P ∗ is still a better choice globally (i.e, for all x ) since it ensures optimal worstcase linear rate in preconditioned gradient descent. But there are better preconditioners that depend on the current gradient. We exploit this in the ellipsoid variant of multidimensional backtracking to improve our competitive ratio. We backtrack from the preconditioner that maximizes the local progress guarantees to ensure a 1 / √ d competitive ratio, while ensuring volume shrinkage of the set of candidate preconditioners when we call CUT, if the preconditioner fails the Armijo condition.

C Separating hyperplanes

In this section, we prove Propositions 4.2 and 4.3 on existence and strengthening of separating hyperplanes for valid preconditioners.

General idea. Let us start with a summary of the separating hyperplanes used to search for good preconditioners as discussed in Sections 3 and 4. The goal of the separating hyperplanes is to give us ways to shrink the initial set of potential preconditioners S to narrow in on valid preconditioners using the cutting-plane methods in Section 5. At each iteration we are looking for preconditioners P that satisfy the Armijo condition at x given by

If P fails the Armijo condition, we conclude that P is invalid. To obtain more information, we look at the condition as a function of the (diagonal of the) preconditioner, and define the gap function at x ,

Then, h ( p ) ≤ 0 if P = Diag( p ) satisfies the Armijo condition at x , and h ( p ) > 0 otherwise. Any preconditioner Diag( q ) such that h ( q ) > 0 is guaranteed to be invalid. We can use the gradient of h at p and convexity to find a half-space such that one side contains only preconditioners with h ( p ) > 0 . In this section, we show how to construct such half-space, and strengthen them using the partial order on matrices, which is needed to ensure volume shrinkage of our cutting plane methods.

C.1 Stronger hyperplanes

In the main body we presented the strengthening of separating hyperplanes via truncation (Proposition 4.3) after the result of existence of separating hyperplanes (Proposition 4.2). Here, we prove a more general lemma on strengthening half-spaces of invalid preconditioners first, as it is useful in simplifying the proof of Proposition 4.2. Proposition 4.3 follows directly from the following lemma.

Lemma C.1. Let H v ,α be the intersection of the non-negative orthant R d ≥ 0 and the half-space defined by the vector v ∈ R d and coefficient α > 0 ,

Define u := max { v , 0 } and let H u ,α be defined similarly as above, that is,

If H v ,α only contains diagonals of invalid preconditioners, that is, Diag( p ) is invalid for any p ∈ H v , Then H v ,α ⊆ H u ,α and H u ,α only contains diagonals of invalid preconditioners.

Proof. Inclusion H v ,α ⊆ H u ,α . We have that ⟨ p , v ⟩ > α implies ⟨ p , u ⟩ > α for any p ∈ R d ≥ 0 since

H u ,α only contains invalid diagonals. Let p u ∈ H u ,α . We can show that Diag( p u ) is invalid by finding p v ∈ H v ,α such that Diag( p v ) ⪯ Diag( p u ) . Since Diag( p v ) is invalid by assumption, this would imply that Diag( p u ) is also invalid. To find p v , we can truncate the entries of p u as

Then p v ∈ H v ,α since α < ⟨ u , p u ⟩ = ⟨ u , p v ⟩ = ⟨ v , p v ⟩ . 6 and Diag( p u ) ⪰ Diag( p v ) , as desired.

C.2 Separating hyperplanes for invalid preconditioners

We are now in position to prove Proposition 4.2.

Proof of Proposition 4.2. Throughout the proof, we shall denote by P the matrix Diag( p ) . If f is convex, then h also is since the map p ∈ R d ≥ 0 ↦→ f ( x -P ∇ f ( x )) is the composition of an affine transformation and a convex function, and ∥∇ f ( x ) ∥ 2 P = ⟨∇ f ( x ) , Diag( p ) ∇ f ( x ) ⟩ is linear in p . Convexity of h yields the inequality

This implies that if p is such that h ( q ) + ⟨∇ h ( q ) , p -q ⟩ > 0 , then h ( p ) > 0 , which implies that Diag( p ) is an invalid preconditioner. Rearranging we conclude that Diag( p ) is invalid for all p in the set in (5), i.e., in

We express the above half-space as

Yet, for H > ( v ) to be equivalent to the set in (16) or even to be well-defined, we need to ensure ⟨∇ h ( q ) , q ⟩ -h ( q ) > 0 . To see that this holds, note first that by convexity of h and that fact that h (0) = 0 we have

Finally, we can write v in terms of f and Q . To do so, first define x + := x -Q ∇ f ( x ) , and the gradients of f at different points by g := ∇ f ( x ) and g + := ∇ f ( x + ) . Then, by the chain-rule,

To show that the last inequality is strict, assume that ⟨∇ h ( q ) , q -0 ⟩ -h ( q ) = 0 for the sake of contradiction. By Lemma C.1, the half-space H := { p ∈ R d ≥ 0 : 〈 [ ∇ h ( x )] + , p 〉 > 0 } contains only diagonals of invalid preconditioners, where [ ∇ h ( x )] + := max {∇ h ( x ) , 0 } entry wise. However, ( 1 / L ) 1 ∈ H as [ ∇ h ( x )] + ≥ 0 and should be invalid, which is a contradiction since f is L -smooth and 1 / L I is valid. Therefore, ⟨∇ h ( q ) , q -0 ⟩ -h ( q ) > 0 .

which implies

Plugging these equations in the definition of v yields

Remark on assumptions of Proposition 4.2. One may have noticed that we never use the assumption that Q fails the Armijo condition (i.e., that h ( q ) > 0 ) in the proof of the proposition. In fact, the proposition holds for any q ∈ R d ≥ 0 . However, and crucially for our application, we have that q is in the half-space H > ( u ) of invalid diagonals from Proposition 4.2. In multidimensional backtracking, q is the diagonal of a preconditioner Diag( q ) that failed the Armijo condition h ( q ) > 0 . Since q is close to the origin in multidimensional backtracking, we can ensure the half-space H > ( u ) contains a significant portion of our current set of candidate preconditioners, leading to significant shrinkage of the set of candidate preconditioners whenever CUT is invoked.

D Cutting-plane methods

D.1 Boxes

Given a box B ( b ) for some b ∈ R d ≥ 0 and a vector u ∈ R d ≥ 0 , our cutting plane method needs to find a box B ( b + ) that contains B ( b ) ∩ H > ( u ) which, hopefully, has smaller volume than B ( b ) .

The next lemma gives a formula for the minimum volume box for any u , which is used in the main text to define CUT in Equation (7). Moreover, we show that if the half-space H > ( u ) is close enough to the origin (since otherwise we might have b + = b ), then we have a significant volume decrease.

Lemma D.1. Let b ∈ R d ≥ 0 and q ∈ B ( b ) . Let u ∈ R d ≥ 0 . Then the box B ( b + ) with minimum volume that contains B ( b ) ∩ H ≤ ( u ) is given by (using the convention that 1 / u [ i ] = + ∞ if u [ i ] = 0 )

Moreover, if ( 1 / 2 d ) · b is excluded by the half-space, that is, b ∈ H > ( u ) , then Vol( B ( b + )) ≤ ( 1 / ( d +1) ) Vol( B ( b + )) .

Proof. Formula for b + . Finding the minimum volume box containing B ( b ) ∩ H ≤ ( u ) ,

is equivalent to finding the solution to the following optimization problem:

As the constraints separate over the coordinates, the minimization can be done for each coordinate separately. As the function is increasing in c [ i ] , the minimum is achieved by making all the constraints tight, which giver the formula for b + in the statement of the lemma.

Volume decrease. Let us prove the second part of the statement. Thus, assume for the remainder of the proof that ( 1 / 2 d ) · b ∈ H > ( u ) . We first show that Vol( B ( b + )) ≤ ( 1 / ( d +1) ) Vol( B ( b + )) if we assume that the update from B ( b ) to B ( b + ) shrinks the box in only one coordinate, i.e.,

̸

Assume the above holds and I = { j } . Then, as ( 1 / 2 d ) · b ∈ H > ( u ) implies ⟨ u , ( 1 / 2 d ) b ⟩ > 1 ,

This together with the fact that b + [ i ] = b [ i ] for all i = j and b + [ j ] = 1 / u [ j ] yields

̸

̸

To complete the proof, we only need to show we may assume (18) holds. Assume the opposite, that is, that there are two distinct coordinates that shrink from b + to b . We will show that the volume shrinks more, meaning the above bound also applies. Formally, assume there are j, k ∈ I that are distinct. For this part, it will be useful to denote by b + ( u ) the point defined in Equation (17) for a given vector u . We will show we can construct u ′ ∈ R d ≥ 0 such that Vol( b + ( u )) ≤ Vol( b + ( u ′ )) while maintaining the property ( 1 / 2 d ) b ∈ H > ( u ′ ) and such that b + ( u ′ )[ i ] = b [ i ] for all i ∈ I \ { j } , which makes (18) follow by induction. Indeed, define u ′ ∈ R d ≥ 0 by

First, note that ( 1 / 2 d ) b ∈ H > ( u ′ ) since

̸

and, thus, 1 < ⟨ u , ( 1 / 2 d ) b ⟩ = ⟨ u ′ , ( 1 / 2 d ) b ⟩ . Let us now show that Vol( B ( b + ( u ))) ≤ Vol( B ( b + ( u ′ ))) . Since b + ( u )[ i ] = b + ( u ′ )[ i ] for i ̸∈ { j, k } , we have

To get that Vol( b + ( u )) ≤ Vol( b + ( u ′ )) , we can show that last line is bounded by < 1 . Using the substitution α := b [ j ] u [ j ] and β := b [ k ] u [ k ] , we want to show that

This holds if α > 1 and β > 1 , is implied by j, k ∈ I since α = b [ j ] u [ j ] > 1 and β = b [ k ] u [ k ] > 1 . A simple induction shows we may assume (18) holds. To see that ( α + β -1) /αβ < 1 , note that

Equipped with the above lemma, we are in position to prove Theorem 5.1.

Proof of Theorem 5.1. Property (a), holds by induction because, for any u t used in a call to CUT, we have P ∗ ∈ H ≤ ( u t ) since P ∗ is valid and since by Proposition 4.2 the half-space H ≤ ( u t ) contains only diagonals of invalid preconditioners. For (b), fix t ∈ { 1 , . . . , T } and recall that in this case we have S t = { Diag( p ) : p ∈ B ( b t ) } and P t = ( 1 / 2 d ) · Diag( b t ) . The competitive ratio of 1 / 2 d follows since Diag( b t ) is the preconditioner that maximizes ∥∇ f ( x t ) ∥ P for P ∈ B ( b t ) . Finally, for (c) by Lemma D.1 we have that every call to CUT makes the volume of the set decrease by 1 /c := 1 / ( d +1) . Moreover, one can easily verify that b t [ i ] ≥ min { 1 / L , b 0 [ i ] } for all i ∈ { 1 , . . . , d } since B (( 1 / L ) ✶ ) contains only diagonals of valid preconditioners. Therefore, for b min [ i ] := min { 1 / L , b 0 [ i ] } , the volume of B ( b t ) cannot be smaller than B ( b min ) for all iteration t . Therefore, the number of times CUT is invoked is no more than

D.2 Axis-aligned ellipsoids

We now analyze the cutting-plane method using axis-aligned ellipsoids. Interestingly, the results that we prove in this sections are connected to some of the results from Goemans et al. (2009) via polarity theory. We defer a discussion on this connection to the end of this section.

Different from the main body, it will be helpful for the analysis of the method and proofs of the results to not restrict ellipsoids to the non-negative orthant, as was done in the main text for ease of exposition. For any symmetric positive definite matrix A ∈ R d × d , define the ellipsoid given by A by

When A is diagonal, we say that E ( A ) is axis-aligned . Moreover, we may slightly overload our notation by defining E ( a ) := E (Diag( a )) .

General ellipsoids. Although we are ultimately interested in working solely with ellipsoids defined by diagonal matrices, we will start by looking at more general ellipsoids, and then exploit symmetry in our case to derive the result in Lemma 5.2. We start with an ellipsoid E ( A ) where A is a positive definite matrix. Then, given a vector u ∈ R d , we are interested in finding an ellipsoid the intersection

of E ( A ) with the half-spaces defined by u and -u that contain the origin, that is, the set

The following theorem shows how to find an ellipsoid that contains the above intersection, and how to guarantee its volume is smaller than E ( A ) if u is large enough. Interestingly, note that

The set E ( uu T ) is a degenerate ellipsoid, in the sense that it is not a compact set, and any p orthogonal to u is contained in E ( uu ⊤ ) . Still, the next theorem shows how to find a convex combination of E ( A ) and E ( uu T ) -which always contains E ( A ) ∩E ( uu T ) -that is guaranteed to have volume smaller than E ( A ) if u is large enough. The following result can be seen as the polar result of Goemans et al. (2009, Lemma 2).

Theorem D.2. Let A ∈ R d × d be positive definite and let u ∈ R d . Let λ ∈ (0 , 1) and define

Then E ( A ) ∩ E ( uu T ) ⊆ E ( L ( A , u )) and

In particular, if ℓ := ∥ u ∥ 2 A -1 > d and then λ ∈ (0 , 1) and Vol( E ( L ( A , u ))) = ν d ( u ) Vol( E ( A )) where

Proof. First, note that for any p ∈ E ( A ) ∩ E ( uu T ) and any λ ∈ (0 , 1) we have

Thus, E ( L ( A , u )) ⊆ E ( A ) ∩ E ( uu T ) . For the volume decrease, recall that for ellipsoids E ( A ) we have Vol( E ( A )) = V d / √ det( A ) where V d is the volume of the unit sphere in R d . By the matrixdeterminant lemma, we have

Therefore,

Finally, for λ defined as in (20) we have

which yields the desired formula for ν d ( u ) .

On the norm of u . The above theorem has a requirement on the norm of the vector u that defines the half-space H ≤ ( u ) . However, in our cutting plane method we obtain u from Proposition 4.2 and Proposition 4.3, which do not have any guarantees on the norm of u explicitly. Crucially, at any given iteration t of multidimensional backtracking with ellipsoids, we select a candidate preconditioner P t = Diag( p t ) such that ∥ p t ∥ A = 1 / √ 2 d . Then, if it fails the Armijo condition in (4) and u t is as given by Proposition 4.2, then we have p t ∈ H > ( u t ) , that is, the separating hyperplane excludes p t . As we will show, this implies that ∥ u ∥ A -1 is large.

Lemma D.3. Let A ∈ R d × d be positive definite and p ∈ R d ≥ 0 be such that ∥ p ∥ A ≤ γ for some γ > 0 . Let u ∈ R d ≥ 0 be such that p ∈ H > ( u ) . Then ∥ u ∥ A -1 > 1 / γ .

Proof. For the sake of contradiction, assume ∥ u ∥ A -1 ≤ 1 / γ . Then ∥ u ∥ A -1 · ∥ p ∥ A ≤ 1 . Thus, by the Cauchy-Schwartz inequality,

This is a contradiction since p ∈ H > ( u ) and, therefore, ⟨ u , p ⟩ > 1 .

On the volume decrease. Although the formula ν d ( u ) in Equation (21) can be hard to interpret, we show a simple bound when ∥ u ∥ 2 A -1 ≥ 2 d .

Lemma D.4. Let A ∈ R d × d be a positive definite matrix and u ∈ R d be such that ∥ u ∥ 2 A -1 > d . For c := d/ℓ ∈ (0 , 1) we have ν d ( u ) ≤ √ c · e 1 -c , where ν d is defined as in (21) . In particular, if ∥ u ∥ 2 A -1 > d , then ν d ( u ) ≤ 4 √ e / √ 2 .

Proof. Define ℓ := ∥ u ∥ 2 A -1 > d and c := d / ℓ ∈ (0 , 1) . Then, where the last inequality follows since 1 + x ≤ e x for all x ∈ R . In particular, note that c ∈ (0 , 1) ↦→ c · e 1 -c is increasing since the derivative of the mapping is positive on (0 , 1) . Thus, if ∥ u ∥ A -1 ≥ 2 d , then c ≤ 1 2 and c · e 1 -c ≤ ( 1 / 2 ) · e 1 / 2 .

Exploiting symmetry. Let us now exploit symmetry to avoid using non-diagonal matrices in our ellipsoids. We use the notion of axis-aligned sets in the next few results. A set X ⊆ R d is axis-aligned if for any point p ∈ X , the reflections of p along the axes are also contained in X . Formally, for any s ∈ {± 1 } d , we have that if p ∈ X , then Diag( s ) p ∈ X . Furthermore, with a slight abuse of notation define Diag( A ) := Diag(diag( A )) . That is, Diag( A ) is the diagonal matrix whose diagonal entries match those of A . The idea is that the set { p ∈ R d ≥ 0 : Diag( p ) is valid } of diagonals of valid preconditioners is contained in the non-negative orthant. Yet, we can extend it by reflecting it over each of the axes. Although this may seem counter-intuitive, this translates the structure of our problem

into symmetry among all orthant, and this can be exploited elegantly. Formally, the set of diagonals of valid preconditioners reflected over each axis is given by set

where | p | is the entry-wise absolute value of p ∈ R d . The following lemma shows that when looking for low volume ellipsoids that contain an axis-aligned set, we can restrict out attention to axis-aligned ellipsoids, defined by a diagonal matrix. The following lemma can be seen as the polar statement of Goemans et al. (2009, Proposition 3.1), with the benefit of not requriring any matrix inversions.

Lemma D.5. Let X ⊂ R d be an axis-aligned convex set and let A ∈ R d × d be positive definite matrix such that X ⊆ E ( A ) . Then X ⊆ E (Diag( A )) and Vol( E (Diag( A ))) ≤ Vol( E ( A )) .

Proof. Let us start by showing that X ⊆ E (Diag( A )) . We use the notation Diag( v ) · X to denote the set Diag( v ) · X := { Diag( v ) · x : x ∈ X} . Since X is axis-aligned, we have

Therefore, X is contained in each of the 2 d ellipsoids of the form E (Diag( s ) A Diag( s )) . Thus,

where the last inclusion follows since, for any set of positive definite matrices M , one may verify that ∩ M ∈M E ( M ) ⊆ E (( 1 / |M| ) ∑ M ∈M M ) . Finally, note that

Indeed, let i, j ∈ { 1 , · · · , d } . If i = j , then (Diag( s ) A Diag( s )) i,j = A i,j for any s ∈ {± 1 } d . If i = j , then

̸

Let us now show that Vol( E (Diag( A ))) ≤ Vol( E ( A )) . Note that log(Vol( E ( A ))) = log(Vol( E ( I ))) -1 2 log det( A ) . Since log det( · ) is concave over positive definite matrices, we have

Therefore,

which implies that Vol( E (Diag( A ))) ≤ Vol( E ( A ))

We are now in position to prove Lemma 5.2, which follows directly from the previous two results.

Proof of Lemma 5.2. By the assumptions in Proposition 4.2 we have that P := Diag( p ) fails the Armijo condition 4 condition and, thus, p ∈ H > ( u ) . This together with the assumption that ∥ p ∥ A ≤ 1 / √ 2 d imply via Lemma D.3 that ∥ u ∥ A -1 ≥ √ 2 d . This allows us to use Theorem D.2 to find a new ellipsoid containing E ( a ) ∩ H ≤ ( u ) with the required volume decrease by Lemma D.4.

̸

Yet, this ellipsoid may not be axis-aligned. We shall exploit the symmetry described in Lemma D.5 to show that the axis-aligned ellipsoid E ( a + ( a , u )) enjoys the same guarantees.

Formally, we need E ( a + ( a , u )) to contain E ( a ) ∩ H ≤ ( u ) . Since u ≥ 0 , we have

Thus, it suffices for E ( a + ( a , u )) to contain E ( a ) ∩S ( u ) . From Theorem D.2 we know that E ( a ) ∩S ( u ) is contained in the ellipsoid given by the matrix λ Diag( a ) + (1 -λ ) uu T for any λ , in particular for λ as in (20) since ∥ u ∥ A -1 > √ d . Since S ( u ) is axis-aligned, we can exploit symmetry using Lemma D.5, which tells that E ( a ) ∩ S ( u ) is contained in the ellipsoid given by the matrix

as desired. Finally, the bound on the volume follows by Theorem D.2 and the bound on ν d ( u ) given by Lemma D.4 since ∥ u ∥ A -1 ≥ √ 2 d .

Finally, we are in position to prove Theorem 5.3, which follows almost directly from Lemma 5.2.

Proof of Theorem 5.3. Note that (a) holds by induction and since, by Proposition 4.2, we have diag( P ∗ ) ∈ H ≤ ( u t ) for any u t used in a call to CUT. For (b), fix t ∈ { 1 , . . . , T } and recall that in this case we have S t = { Diag( p ) : p ∈ E ( a t ) } . As described in (8), one may verify that Diag( q ∗ t ) for q ∗ t given by

maximizes ∥∇ f ( x t ) ∥ P for P ∈ S t . Since

we conclude that P t is 1 / √ 2 d -competitive. For (c), first note that we may assume ( 1 / L ) ✶ ∈ E ( α 0 I ) . To see that, assume ( 1 / L ) ✶ ̸∈ E ( α 0 I ) , implying α 0 d > L 2 . In this case, any candidate preconditioner computed by CANDIDATE is always valid and, thus, we never call CUT. To see this, let A 0 := α 0 I be the matrix defining the initial ellipsoid. Then, by the definition of CANDIDATE for ellipsoids we have that P 0 = Diag( p 0 ) is such that

Therefore, p 0 [ i ] ≤ 1 / L for all i ∈ { 1 , . . . , d } , which implies that P 0 is valid since P 0 ⪯ 1 L I .

Let us look now at the case ( 1 / L ) ✶ ∈ E ( α 0 I ) . Therefore, B ( 1 / L ✶ ) ⊆ E ( a t ) for all iterations t . Since the minimum volume ellipsoid containing the box B (( 1 / L ) ✶ ) is the unit sphere of radius 1 / L , that is, E (( L 2 / d ) ✶ ) . Therefore, Vol( E ( a t )) ≥ Vol( E (( L 2 / d ) ✶ )) . Moreover, every time we call cut the volume of the ellipsoid goes down by 1 /c := 4 √ e/ √ 2 . Therefore, the total number of calls to CUT is no more than since log( c ) ≥ 1 / 12

Refining the choice of λ . Although we have shown in Lemma 5.2 a choice a λ that guarantees volume decrease, it may be sub-optimal. The choice of λ in Equation (20) is inherited from the non-symmetric case in Theorem D.2. Although Lemma 5.2 and Theorem D.2 match when u has only one non-zero entry, we should expect better choices of λ , leading to more volume shrinkage, to be possible in Lemma 5.2. Although we have not found a choice of λ that is dependent on u that generically improves upon (20), in practice we can solve for a better λ numerically, by directly minimizing the volume of the resulting ellipsoid,

As the problem is one-dimensional, numerical solvers can often find near-optimal solutions. By warm-starting a numerical solver with the λ defined in (20), we can guarantee that the resulting ellipsoid leads to a smaller volume and we do not lose our worst-case theoretical guarantees.

Connection to the polar problem and related work. Our results have an interesting connection to some of the results from Goemans et al. (2009), via the use of polarity theory. Here we give a quick overview of their work and the connection to our cutting plane methods. Goemans et al. (2009) shows techniques to approximate some polyhedron P ⊆ R d (a polymatroid being one of the main examples) from inside by some ellipsoid E ( A ) . Their algorithm maintains an ellipsoid E ( A ) ⊆ P and tries to iteratively enlarge it. They assume access to an oracle such that, at each iteration, either finds a point u ∈ P that is sufficiently far from E ( A ) , meaning ∥ u ∥ A > √ d + ϵ for some ϵ > 0 , or guarantees that E ( A ) 'approximates well' P from inside in the sense that ∥ u ∥ A ≤ ( √ n + ϵ ) /α for all u ∈ P , where α > 0 is some approximation factor. In their algorithm, when the oracle finds a point u ∈ P such that ∥ u ∥ A > √ d + ϵ the algorithm needs to find an ellipsoid E ( A + ) such that where conv( D ) is the convex hull of D . Interestingly, the polar problem is exactly what we need for out cutting plane method. More precisely, the polar set X ∗ of a set X is given by X ∗ := { z ∈ R d : ⟨ z, x ⟩ ≤ 1 } . Then, by taking polars and using that E ( A ) ∗ = E ( A -1 ) , we have that P ∗ ⊆ E ( A -1 ) . Moreover, taking polar on both sides of (22) yields that an equivalent problem is finding ( A + ) -1 such that

That is, the problem is the one of finding a smaller ellipsoid E (( A + ) -1 ) that contains E ( A -1 ) ∩ { z : |⟨ u , z ⟩| ≤ 1 } , which is broadly the goal of the subroutine CUT.

E Experiments

Objective functions

We use L 2 -regularized linear regression L LINEAR and L 2 -regularized logistic regression L LOGISTIC ( w ) , with a regularization coefficient of 1 . Given a data matrix X ∈ R n × d , target y ∈ R n for regression tasks and y ∈ { 0 , 1 } n for classification tasks, and parameters w ∈ R d ,

where x i is the i th row of X and σ is the sigmoid function, σ ( z ) = 1 / 1+exp( -z ) . For all datasets, we add a bias term by prepending a feature column of ones to X .

Datasets

We use the datasets listed in Table 1, made available by LIBSVM (Chang and Lin, 2011), Scikit-Learn (Pedregosa et al., 2011) and the UCI repository (Dua and Graff, 2017).

Data rescaling

We do not rescale, standardize or otherwise change any of the datasets beyond adding a bias term, as our goal is to check whether preconditioned methods can handle badly scaled data.

Initializations

We consider two types of initializations. The first approximates a 'best-case' scenario where we start from an estimate with a reasonable loss value despite the bad scaling of the data. We set w [ i ] = 0 except for the bias term w [0] which is set at the MLE of the non-regularized problem,

w [0] = ¯ y where ¯ y = n ∑ i =1 y [ i ] for linear regression , w [0] = log ( ¯ y 1 -¯ y ) where ¯ y = 1 n ∑ n i =1 y [ i ] for logistic regression.

The results in the main text use this initialization. The second initialization takes w ∼ N (0 , I ) , giving a starting point with potentially large loss. We give results using both initializations in the appendix.

Optimizers used

For the small linear regression problems, we use preconditioned gradient descent with the optimal preconditioner, pre-computed using the semidefinite formulation of Qu et al. (2022), solved using CVXPY (Diamond and Boyd, 2016) based on the Matlab implementation of Qu et al.

Gradient descent with a backtracking line-search with backtracking parameter γ = 1 / 2 .

RPROP (Riedmiller and Braun, 1993) following the implementation and default hyperparameters in PyTorch (Paszke et al., 2019) (starting step-size of 10 -1 , increase step-size factor η + = 1 . 2 , decreasing step-size factor η -= 0 . 5 , minimum step-size of 10 -6 and maximum step-size of 50 ).

https://github.com/pytorch/pytorch/blob/v2.0.1/torch/optim/rprop.py

Hypergradient descent to set the step-size, using (S)GD-HD (the multiplicative variant, Baydin et al., 2018). The hypergradient step-size is set to the default β = 0 . 02 (Baydin et al., 2018, footnote 3). The initial step-size is set to α 0 = 10 -10 , as otherwise most runs diverged immediately.

The diagonal Barzilai-Borwein method of Park et al. (2020), using their non-monotonic line-search. We use the default parameters suggested; a starting step-size of 10 -6 , regularization factor on the previous diagonal approximation µ = 10 -6 , a backtracking factor of 1 / 2 for the backtracking line-search and a window of 15 steps for the non-monotone line-search. This line-search does not use a forward step as the update can increase the preconditioner.

Preconditioned gradient descent using the diagonal Hessian, with a backtracking line-search.

AdaGrad (Duchi et al., 2011) but augmented with a backtracking line-search as suggested by Vaswani et al. (2020) to make it competitive in the deterministic setting, following the PyTorch (Paszke et al., 2019) implementation.

https://github.com/pytorch/pytorch/blob/v2.0.1/torch/optim/adagrad.py

Line-search and forward steps

For all methods, the backtracking line-search is augmented by a forward step. When a step-size is accepted, it is increased by a factor of 1 . 1 for the next step. For multidimensional backtracking, we increase the set uniformly, taking b ′ = 1 . 1 · b for the box and a ′ = a / √ 1 . 1 for the ellipsoid. The ellipsoid uses a slightly smaller increase factor. 7

Hyperparameters for the line-search and multidimensional backtracking

For the backtracking line-searches used in gradient descent, preconditioned gradient descent and used to augment the other algorithms, we start the search at an initial step-size of 10 10 and backtrack by a factor of 1 / 2 when failing the Armijo condition, implemented generically as

For multidimensional backtracking, we initialize the sets such that the first preconditioner is on the order of 10 10 I . Using the notation of Appendix A, we use the scaling factor c 0 = d · 10 10 for the box variant and c 0 = √ d · 10 10 for the ellipsoid variant. The first preconditioner tried by the box variant with backtracking factor γ = 1 / 2 d is then 1 / 2 · 10 10 I , and the first preconditioner tried by the ellipsoid variant (assuming the gradient is uniform, ∇ f ( x 0 ) ∝ 1 ) is 1 / √ 2 · 10 10 I .

E.1 Performance comparison

The algorithms we compare do not have a unified notion of an 'iteration'. For example, a backtracking line-search and multidimensional backtracking do more work than plain gradient descent in between updates to the parameters. To make the performance comparison fair, the results in Figures 1 and 5 account for the overhead of backtracking by comparing the performance against the number of oracle calls, counting the number function and gradient evaluations.

This metric is a good proxy for the work required by multidimensional backtracking. The majority of the computation cost of the 'backtracking' part of the algorithm comes from computing the gradient at the next point (to compute the hypergradient). The overhead due to other operations such as the CUT method are minimal compared to gradient computations. Indeed, beyond the gradient computation, multidimensional backtracking only involves a few vector operations (see Figures 10 and 11). Even solving for the best convex combination numerically to obtain tighter ellipsoids (as in the CUT method in Figure 11) is faster than computing gradients. To illustrate this point, we provide the running times for subsets of a backtracking update from multidimensional backtracking in Table 2.

E.2 Additional results

Figures 15-20 give additional results on small linear and logistic regression problems and large logistic regression problems. Multidimensional backtracking has a consistent performance across problems and does not suffer from the extremely bad conditioning of cpusmall or california-housing (linear regression) or australian, breast-cancer, diabetes and heart (logistic regression).

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10

7

10

2

10 3

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10

7

10

2

10 3

cpusmall

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10

2

10 0

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10

2

10 0

California-housing

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10 0

10 2

10 4

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10 0

10 2

10 4

Power-plant

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10

1

10 1

10 3

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10

1

10 1

10 3

Concrete

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10

6

10

3

10 0

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10

6

10

3

10 0

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

Optimality gap

Optimality gap

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10 2

10 7

10 12

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10 2

10 7

10 12

cpusmall

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10 0

10 3

10 6

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10 0

10 3

10 6

California-housing

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10 1

10 4

10 7

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10 1

10 4

10 7

Power-plant

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10 1

10 4

10 7

Optimality gap

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10 1

10 4

10 7

Concrete

0

500

1000

1500

2000

2500

3000

3500

4000

Oracle calls (Function + Gradient eval.)

10

4

10

1

10 2

0

200

400

600

800

1000

1200

1400

Iterations (Changes in

)

10

4

10

1

10 2

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

Optimality gap

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

11

10

6

10

1

0

100

200

300

400

500

Iterations (Changes in

)

10

11

10

6

10

1

Breast-cancer

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

7

10

4

10

1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

7

10

4

10

1

Australian

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

4

10

2

10 0

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

4

10

2

10 0

Heart

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

5

10

2

10 1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

5

10

2

10 1

Diabetes

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

9

10

5

10

1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

9

10

5

10

1

Ionosphere

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

5

10 0

10 5

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

5

10 0

10 5

Breast-cancer

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

4

10 0

10 4

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

4

10 0

10 4

Australian

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

1

10 1

10 3

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

1

10 1

10 3

Heart

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

3

10 0

10 3

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

3

10 0

10 3

Diabetes

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

9

10

5

10

1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

9

10

5

10

1

Ionosphere

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

News20

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

4

10

1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

4

10

1

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

5

10

2

10 1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

5

10

2

10 1

RCV1

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

3

10 0

10 3

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

3

10 0

10 3

News20

0

200

400

600

800

1000

1200

1400

Oracle calls (Function + Gradient eval.)

10

5

10

2

10 1

Optimality gap

0

100

200

300

400

500

Iterations (Changes in

)

10

5

10

2

10 1

RCV1

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

GD+LS

Ellipsoid MB

Diag. Hessian+LS

Diag. AdaGrad+LS

RPROP

GD-HD (mult.)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors suggest incremental updates of $\\mathbf{x}$ for finding the minimum of strongly-convex function $f$ that guarantee decreasing $f(\\mathbf{x}_{t})-f(\\mathbf{x}_\\ast)$ based on only 1st-order gradient information. \n\nTheir idea is in each step,\n- choose a candidate matrix $\\mathbf{P}_t$ based on set $\\mathcal{S}_t$, and\n- check the condition (4), that guarantee sufficient decrease of the $f$, and\n- if the condition is satisfied:\n    - apply update $\\mathbf{x}_{t+1} = \\mathbf{x}_t - \\mathbf{P}_t\\nabla f(\\mathbf{x}_t)$\n    - $\\mathcal{S}_{t+1} = \\mathcal{S}_t$\n- else\n    - $\\mathbf{x}_{t+1} = \\mathbf{x}_t$\n    - update $\\mathcal{S}_{t+1} = \\text{cut}(\\mathcal{S}_t, \\mathbf{x}_t, \\mathbf{P}_t)$\n\nThey provide proofs for\n- approaching the optimal in Proposition 3.2\n- how to choose candidate and cut algorithm in Theorem 5.3, and its maximum number of calls.\n\nThey conduct some simple experiments in Section 6, and show the proposed algorithm's efficiency.\n\nSTRENGTHS:\noriginality\n- considering backtracking using preconditioned matrix $\\mathbf{P}_t$ would be novel idea. But I'm not an expert of this field, and not so sure on the originality.\n\nquality\n- The proposed algorithm is supported by some proofs, and it shows good empirical results.\n\nclarity\n- Basically, the manuscript is readable.\n\nsignificance\n- The proposed algorithm seems be better than other baselines except for Diag. Hessian+LS, which uses information of 2nd order derivatives, i.e. Hessian, even the proposed algorithm uses 1st-order derivatives of $f$. It would be significant.\n\nWEAKNESSES:\n- The manuscript contains some typos.\n- I have a concern on Figure 5. The horizontal axis shows number of f/grad evals, but I guess the number of CUT calls should be also taken into account.\n\nQUESTIONS:\n- In line 129, the representation $\\nabla^2 f$ appears. Does it mean Hessian? or Laplacian?\n- Between line 197 and 198, the final inequality may be not $\\overset{(3)}{\\leq}$ but $\\overset{(4)}{\\leq}$ ?\n- In the caption of Figure 3(b), $\\mathcal{H}\\_{>}(\\mathbf{u})$ should be replaced by $\\mathcal{H}_>(\\mathbf{v})$ ?\n- In Figure 5, do the authors take the number of CUT into account?\n\nLIMITATIONS:\nThey address the limitation, their method is only supported for convex deterministic setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper provides a backtracking approach for smooth convex optimization on a per-coordinate basis with a theoretical analysis that show the gain with respect to classical backtracking line-search and that compare to the optimal per-coordinate conditioners.

STRENGTHS:
This paper is super well written and organized. The contribution is also significant as it is a building block of many problems in machine learning.
In general, further improving the "adaptivity" of optimization algorithms is essential to seamlessly apply theoretical results (i.e., optimal per coordinate step sizes) to operational purposes.

WEAKNESSES:
The only drawback might be focusing on smooth and strongly convex problems, but it is still a significant first step.

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

comments: SUMMARY:
This paper extends backtracking to multi-dimension. The authors propose a cutting plane method to find optimal per-coordinate step-sizes (in other words, to find an optimal preconditioner) for smooth convex optimisation. Experiments on ill-conditioned logistic regression problems show that the proposed algorithm can find good preconditioner and improve over vanilla gradient descent.

STRENGTHS:
This paper fills a potential gap in the optimization literature by proposing multidimensional backtracking. The proposed method is technically sound and seems to work well in practice.

WEAKNESSES:
I do not see any major issues with the paper, except maybe that it is a bit hard to follow and understand (even though the English is good). Maybe because I don't have enough background on the topic. I'm really sorry for the short review.

QUESTIONS:
None.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-15

comments: SUMMARY:
This paper presents a generalized backtracking line-search method, which estimates coordinate-wise stepsizes referred to as 'preconditioner' of gradient descent. Stemmed from the observation that any existing methods do not exceed the performance of backtracking line-search method, this paper designs a generalized backtracking line-search technique which is realized as a cutting plane method, whose separating hyperplane comes from the hypergradient, i.e., gradient with respect to hyperparameter of the algorithm, which is a stepsize in this case. Followed by the worst-case convergence analysis for smooth strongly convex function, the writers also provide experimental results illustrating the competitiveness of this method for ill-conditioned problems and robustness among problem classes.

STRENGTHS:
Section 4 contains the key insight of this work: that a failed preconditioner (defined as one that violates a Armijo-type condition) provides a cutting plane on the set of valid preconditioners. This is a very nice idea that is, as far as I know, novel, and I expect this work to lead to a lot of follow-up work. This is a new type of result and I think it is valuable.

WEAKNESSES:
.

QUESTIONS:
The 'CUT' subroutine, which is a 'backtracking' phase of this algorithm, can be called up to the number of iterations linear in dimension $d$, which can be large in considering large-scale problems. It has been illustrated in the experimental result that for large problems it recovers preconditioner quite fast, but I'm also curious on how much total overhead is caused by the subroutine `CUT'.


 (p.3 line 121) It seems there is a typo on notation regarding $d, n, \\alpha$.

LIMITATIONS:
.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-21