Title: Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: convex optimization

quasi-Newton methods

Monteiro-Svaiter acceleration

Nesterov's accelerated gradient

online learning
Abstract: In this paper, we propose an accelerated quasi-Newton proximal extragradient method for solving unconstrained smooth convex optimization problems. With access only to the gradients of the objective, we prove that our method can achieve a convergence rate of $\mathcal{O}\bigl(\min\\{\frac{1}{k^2}, \frac{\sqrt{d\log k}}{k^{2.5}}\\}\bigr)$, where $d$ is the problem dimension and $k$ is the number of iterations. In particular, in the regime where $k = \mathcal{O}(d)$, our method matches the _optimal rate_ of $\mathcal{O}(\frac{1}{k^2})$ by Nesterov's accelerated gradient (NAG). Moreover, in the the regime where $k = \Omega(d \log d)$, it outperforms NAG and converges at a _faster rate_ of $\mathcal{O}\bigl(\frac{\sqrt{d\log k}}{k^{2.5}}\bigr)$. To the best of our knowledge, this result is the first to demonstrate a provable gain for a quasi-Newton-type method over NAG in the convex setting.  To achieve such results, we build our method on a recent variant of the Monteiro-Svaiter acceleration framework and adopt an online learning perspective to update the Hessian approximation matrices, in which we relate the convergence rate of our method to the dynamic regret of a specific online convex optimization problem in the space of matrices.

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

Abstract

In this paper, we present an accelerated quasi-Newton proximal extragradient method for solving unconstrained smooth convex optimization problems. With access only to the gradients of the objective function, we prove that our method can achieve a convergence rate of O ( min { 1 k 2 , √ d log k k 2 . 5 } ) , where d is the problem dimension and k is the number of iterations. In particular, in the regime where k = O ( d ) , our method matches the optimal rate of O ( 1 k 2 ) by Nesterov's accelerated gradient (NAG). Moreover, in the the regime where k = Ω( d log d ) , it outperforms NAG and converges at a faster rate of O ( √ d log k k 2 . 5 ) . To the best of our knowledge, this result is the first to demonstrate a provable gain for a quasi-Newton-type method over NAG in the convex setting. To achieve such results, we build our method on a recent variant of the Monteiro-Svaiter acceleration framework and adopt an online learning perspective to update the Hessian approximation matrices, in which we relate the convergence rate of our method to the dynamic regret of a specific online convex optimization problem in the space of matrices.

1 Introduction

In this paper, we consider the following unconstrained convex minimization problem

where the objective function f : R d → R is convex and differentiable. We are particularly interested in quasi-Newton methods, which are among the most popular iterative methods for solving the problem in (1) [1-8]. Like gradient descent and other first-order methods, quasi-Newton methods require only the objective's gradients to update the iterates. On the other hand, they can better exploit the local curvature of f by constructing a Hessian approximation matrix and using it as a preconditioner, leading to superior convergence performance. In particular, when the objective function in (1) is strictly convex or strongly convex, it has long been proved that quasi-Newton methods achieve an asymptotic superlinear convergence rate [7-15], which significantly improves the linear convergence rate obtained by first-order methods. More recently, there has been progress on establishing a local non-asymptotic superlinear rate of the form O ((1 / √ k ) k ) for classical quasiNewton methods and their variants [16-22].

However, all of the results above only apply under the restrictive assumption that the objective function f is strictly or strongly convex. In the more general setting where f is merely convex, to the best of our knowledge, there is no result that demonstrates any form of convergence improvement by quasi-Newton methods over first-order methods. More precisely, it is well known that Nesterov's accelerated gradient (NAG) [23] can achieve a convergence rate of O (1 /k 2 ) if f is convex and has Lipschitz gradients. On the other hand, under the same setting, asymptotic convergence of

classical quasi-Newton methods has been shown in [13, 24] but no explicit rate is given. With certain conditions on the Hessian approximation matrices, the works in [25-27] presented quasi-Newton-type methods with convergence rates of O (1 /k ) and O (1 /k 2 ) , respectively, which are no better than the rate of NAG and, in fact, can be even worse in terms of constants. This gap raises the following fundamental question:

Can we design a quasi-Newton-type method that achieves a convergence rate faster than O (1 /k 2 ) for the smooth convex minimization problem in (1) ?

At first glance, this may seem impossible, as for any first-order method that has access only to a gradient oracle, one can construct a 'worst-case' instance and establish a lower bound of Ω(1 /k 2 ) on the optimality gap [28, 29]. It is worth noting that while such a lower bound is typically shown under a 'linear span' assumption, i.e., the methods only query points in the span of the gradients they observe, this assumption is in fact not necessary and can be removed by the technique of resisting oracle (see, [30, Section 3.3]). In particular, this Ω(1 /k 2 ) lower bound applies for any iterative method that only queries gradients of the objective, including quasi-Newton methods. On the other hand, this lower bound is subject to a crucial assumption: it only works in the high-dimensional regime where the problem dimension d exceeds the number of iterations k . As such, it does not rule out the possibility of a faster rate than O (1 /k 2 ) when the number of iterations k is larger than d . Hence, ideally, we are looking for a method that attains the optimal rate of O (1 /k 2 ) in the regime that k = O ( d ) and surpasses this rate in the regime that k = Ω( d ) .

Contributions. In this paper, we achieve the above goal by presenting an accelerated quasi-Newton proximal extragradient (A-QNPE) method. Specifically, under the assumptions that f in (1) is convex and its gradient and Hessian are Lipschitz, we prove the following guarantees:

From any initialization, A-QNPE can attain a global convergence rate of O ( min { 1 k 2 , √ d log k k 2 . 5 } ) . In particular, this implies that our method matches the optimal rate of O ( 1 k 2 ) when k = O ( d ) , while it converges at a faster rate of O ( √ d log k k 2 . 5 ) when k = Ω( d log d ) . Alternatively, we can bound the number of iterations required to achieve an ϵ -accurate solution by N ϵ = O ( min { 1 ϵ 0 . 5 , d 0 . 2 ϵ 0 . 4 (log d ϵ 2 ) 0 . 2 } ) .

In terms of computational cost, we show that the total number of gradient queries after N iterations can be bounded by 3 N , i.e., on average no more than 3 per iteration. Moreover, the number of matrix-vector products to achieve an ϵ -accurate solution can be bounded by ˜ O ( min { d 0 . 25 ϵ 0 . 5 , 1 ϵ 0 . 625 } ) .

Combining the two results above, we conclude that A-QNPE requires O ( min { 1 ϵ 0 . 5 , d 0 . 2 ϵ 0 . 4 (log d ϵ 2 ) 0 . 2 } ) gradient queries to reach an ϵ -accurate solution, which is at least as good as NAG and is further superior when ϵ = O ( 1 d 2 log 2 ( d ) ) . To the best of our knowledge, this is the first result that demonstrates a provable advantage of a quasi-Newton-type method over NAG in terms of gradient oracle complexity in the smooth convex setting.

To obtain these results, we significantly deviate from the classical quasi-Newton methods such as BFGS and DFP. Specifically, instead of mimicking Newton's method as in the classical updates, our A-QNPE method is built upon the celebrated Monteiro-Svaiter (MS) acceleration framework [31, 32], which can be regarded as an inexact version of the accelerated proximal point method [33, 34]. Another major difference lies in the update rule of the Hessian approximation matrix. Classical quasiNewton methods typically perform a low-rank update of the Hessian approximation matrix while enforcing the secant condition. On the contrary, our update rule is purely driven by our convergence analysis of the MS acceleration framework. In particular, inspired by [35], we assign certain loss functions to the Hessian approximation matrices and formulate the Hessian approximation matrix update as an online convex optimization problem in the space of matrices. Therefore, we propose to update the Hessian approximation matrices via an online learning algorithm.

Related work. The authors in [32] proposed a refined MS acceleration framework, which simplifies the line search subroutine in the original MS method [31]. By instantiating it with an adaptive second-order oracle , they presented an accelerated second-order method that achieves the optimal rate of O ( 1 k 3 . 5 ) . The framework in [32] serves as a basis for our method, but we focus on the setting where we have access only to a gradient oracle and we consider a quasi-Newton-type update. Another closely related work is [35], where the authors proposed a quasi-Newton proximal extragradient method with a global non-asymptotic superlinear rate. In particular, our Hessian approximation update is inspired by the online learning framework in [35]. On the other hand, the major difference is that the authors in [35] focused on the case where f is strongly convex and presented a global

superlinear rate, while we consider the more general convex setting where f is only convex (may not be strongly convex). Moreover, we further incorporate the acceleration mechanism into our method, which greatly complicates the convergence analysis; see Remark 2 for more discussions.

Another class of optimization algorithms with better gradient oracle complexities than NAG are cutting plane methods [36-44], which are distinct from quasi-Newton methods we study in this paper. In particular, in the regime where ϵ = ˜ O ( 1 d 2 ) , they can achieve the optimal gradient oracle complexity of O ( d log 1 ϵ ) [38]. On the other hand, in the regime where ϵ = Ω( 1 d 2 ) , the complexity of the cutting plane methods is worse than NAG, while our proposed method matches the complexity of NAG.

2 Preliminaries

Next, we formally state the required assumptions for our main results.

Assumption 1. The function f is twice differentiable, convex, and L 1 -smooth. As a result, we have 0 ⪯ ∇ 2 f ( x ) ⪯ L 1 I for any x ∈ R d , where I ∈ R d × d is the identity matrix.

Assumption 2. The Hessian of f is L 2 -Lipschitz, i.e., we have ∥∇ 2 f ( x ) -∇ 2 f ( y ) ∥ op ≤ L 2 ∥ x -y ∥ 2 for any x , y ∈ R d , where ∥ A ∥ op ≜ sup x : ∥ x ∥ 2 =1 ∥ Ax ∥ 2 .

We note that both assumptions are standard in the optimization literature and are satisfied by various loss functions such as the logistic loss and the log-sum-exp function (see, e.g., [16]).

Remark 1 . We note that the additional assumption of Lipschitz Hessian does not alter the lower bound of Ω(1 /k 2 ) that we discussed in the introduction. Indeed, this lower bound is established by a worst-case quadratic function, whose Hessian is constant (Lipschitz continuous with L 2 = 0 ). Therefore, Assumption 2 does not eliminate this worst-case construction from the considered problem class, and thus the lower bound also applies to our setting.

Monteiro-Svaiter acceleration. As our proposed method uses ideas from the celebrated MonteiroSvaiter (MS) acceleration algorithm [31], we first briefly recap this method. MS acceleration, also known as accelerated hybrid proximal extragradient (A-HPE), consists of intertwining sequences of iterates { x k } , { y k } , { z k } , scalar variables { a k } and { A k } as well as step sizes { η k } . The algorithm has three main steps. In the first step, we compute the auxiliary iterate y k according to

In the second step, an inexact proximal point step x k +1 ≈ y k -η k ∇ f ( x k +1 ) is performed. To be precise, given a parameter σ ∈ [0 , 1) , we find x k +1 that satisfies

Then in the third step, the iterate z is updated by following the update

Finally, we update the scalar A k +1 by A k +1 = A k + a k . The above method has two implementation issues. First, to perform the update in (3) directly, one needs to solve the nonlinear system of equations x -y k + η k ∇ f ( x ) = 0 to a certain accuracy, which could be costly in general. To address this issue, a principled approach is to replace the gradient operator ∇ f ( x ) with a simpler approximation function P ( x ; y k ) and select x k +1 as the (approximate) solution of the equation:

For instance, we can use P ( x ; y k ) = ∇ f ( y k ) and accordingly (4) is equivalent to x k +1 = y k -η k ∇ f ( y k ) , leading to the accelerated first-order method in [31]. If we further have access to the Hessian oracle, we can use P ( x ; y k ) = ∇ f ( y k ) + ∇ 2 f ( y k )( x -y k ) and (4) becomes x k +1 = y k -η k ( I + η k ∇ 2 f ( y k )) -1 ∇ f ( y k ) , leading to the second-order method in [31].

However, approximating ∇ f ( x ) by P ( x ; y k ) leads to a second issue related to finding a proper step size η k . More precisely, one needs to first select η k , compute y k from (2), and then solve the system in (4) exactly or approximately to obtain x k +1 . However, these three variables, i.e., x k +1 , y k and η k may not satisfy the condition in (3) due to the gap between ∇ f ( x ) and P ( x ; y k ) . If that happens, we need to re-select η k and recalculate both y k and x k +1 until the condition in (3) is satisfied. To

address this issue, several bisection subroutines have been proposed in the literature [31, 45-47] and they all incur a computational cost of log(1 /ϵ ) per iteration.

Optimal Monteiro-Svaiter acceleration. A recent paper [32] refines the MS acceleration algorithm by separating the update of y k from the line search subroutine. In particular, in the first stage, we use η k to compute a k and then y k from (2), which will stay fixed throughout the line search scheme. In the second stage, we aim to find a pair ˆ x k +1 and ˆ η k such that they satisfy

To find that pair, we follow a similar line search scheme as above, with the key difference that y k is fixed and ˆ η k can be different from η k that is used to compute y k . More precisely, for a given ˆ η k , we find the solution of (4) denoted by ˆ x k +1 and check whether it satisfies (5) or not. If it does not, then we adapt the step size and redo the process until (5) is satisfied. Then given the values of these two parameters η k and ˆ η k , the updates for x and z would change as we describe next:

If ˆ η k ≥ η k , we update x k +1 = ˆ x k +1 , A k +1 = A k + a k and z k +1 = z k -a k ∇ f (ˆ x k +1 ) . Moreover, we increase the next tentative step size by choosing η k +1 = η k /β for some β ∈ (0 , 1) .

Otherwise, if ˆ η k < η k , the authors in [32] introduced a momentum damping mechanism . Define γ k = ˆ η k /η k < 1 . We then choose x k +1 = (1 -γ k ) A k A k + γ k a k x k + γ k ( A k + a k ) A k + γ k a k ˆ x k +1 , which is a convex combination of x k and ˆ x k +1 . Moreover, we update A k +1 = A k + γ k a k and z k +1 = z k -γ k a k ∇ f (ˆ x k +1 ) . Finally, we decrease the next tentative step size by choosing η k +1 = βη k .

This approach not only simplifies the procedure by separating the update of { y k } from the line search scheme, but it also shaves a factor of log(1 /ϵ ) from the computational cost of the algorithm, leading to optimal first and second-order variants of the MS acceleration method. Therefore, as we will discuss in the next section, we build our method upon this more refined MS acceleration framework.

3 Accelerated Quasi-Newton Proximal Extragradient

In this section, we present our accelerated quasi-Newton proximal extragradient (A-QNPE) method. An informal description of our method is provided in Algorithm 1. On a high level, our method can be viewed as the quasi-Newton counterpart of the adaptive Monteiro-Svaiter-Newton method proposed in [32]. In particular, we only query a gradient oracle and choose the approximation function in (4) as P ( x ; y k ) = ∇ f ( y k ) + B k ( x -y k ) , where B k is a Hessian approximation matrix obtained only using gradient information. Moreover, another central piece of our method is the update scheme of B k . Instead of following the classical quasi-Newton updates such as BFGS or DFP, we use an online learning framework, where we choose a sequence of matrices B k to achieve a small dynamic regret for an online learning problem defined by our analysis; more details will be provided later in Section 3.2. We initialize our method by choosing x 0 , z 0 ∈ R d and setting A 0 = 0 and η 0 = σ 0 , where σ 0 is a user-specified parameter. Our method can be divided into the following four stages:

In the first stage , we compute the scalar a k and the auxiliary iterate y k according to (2) using the step size η k . Note that y k is then fixed throughout the k -th iteration.

In the second stage , given the Hessian approximation matrix B k and the iterate y k , we use a line search scheme to find the step size ˆ η k and the iterate ˆ x k +1 such that

where α 1 ∈ [0 , 1) and α 2 ∈ (0 , 1) are user-specified parameters with α 1 + α 2 < 1 . The first condition in (6) requires that ˆ x k +1 inexactly solves the linear system of equations ( I +ˆ η k B k )( x -y k ) + ˆ η k ∇ f ( y k ) = 0 , where α 1 ∈ [0 , 1) controls the accuracy. As a special case, we have ˆ x k +1 = y k -( I + ˆ η k B k ) -1 ∇ f ( y k ) when α 1 = 0 . The second condition in (7) directly comes from (5) in the optimal MS acceleration framework, which ensures that we approximately follow the proximal point step ˆ x k +1 = y k -ˆ η k ∇ f (ˆ x k +1 ) . To find the pair (ˆ η k , ˆ x k +1 ) satisfying both (6) and (7), we implement a backtracking line search scheme. Specifically, for some β ∈ (0 , 1) , we iteratively try ˆ η k = η k β i for i ≥ 0 and solve ˆ x k +1 from (6) until the condition in (7) is satisfied. The line search scheme will be discussed in more detail in Section 3.1.

In the third stage , we update the variables x k +1 , z k +1 , A k +1 and set the step size η k +1 in the next iteration. Specifically, the update rule we follow depends on the outcome of the line search scheme. In the first case where ˆ η k = η k , i.e., the line search scheme accepts the initial trial step size, we let

as in the original MS acceleration framework. Moreover, this also suggests our choice of the step size η k may be too conservative. Therefore, we increase the step size in the next iteration by η k +1 = ˆ η k /β . In the second case where ˆ η k < η k , i.e., the line search scheme backtracks, we adopt the momentum damping mechanism in [32]:

where γ k = ˆ η k /η k < 1 . Accordingly, we decrease the step size in the next iteration by letting η k +1 = ˆ η k (note that ˆ η k < η k ).

In the fourth stage , we update the Hessian approximation matrix B k +1 . Inspired by [35], we depart from the classical quasi-Newton methods and instead let the convergence analysis guide our update scheme. As we will show in Section 3.2, the convergence rate of our method is closely related to the cumulative loss ∑ k ∈B ℓ k ( B k ) incurred by our choices of { B k } , where B = { k : ˆ η k < η k } denotes the indices where the line search scheme backtracks. Moreover, the loss function has the form ℓ k ( B k ) ≜ ∥ w k -B k s k ∥ 2 ∥ s k ∥ 2 , where w k ≜ ∇ f (˜ x k ) -∇ f ( x k ) , s k ≜ ˜ x k -x k and ˜ x k is an auxiliary iterate returned by our line search scheme. Thus, this motivates us to employ an online learning algorithm to minimize the cumulative loss. Specifically, in the first case where ˆ η k = η k (i.e., k / ∈ B ), the current Hessian approximation matrix B k does not contribute to the cumulative loss and thus we keep it unchanged (cf. Line 9). Otherwise, we follow an online learning algorithm in the space of matrices. The details will be discussed in Section 3.2.

Finally, we provide a convergence result in the following Proposition for Algorithm 1, which serves as the basis for our convergence analysis. We note that the following results do not require additional conditions on B k other than the ones in (6) and (7). The proof is available in Appendix A.1.

Proposition 1. Let { x k } N k =0 be the iterates generated by Algorithm 1. If f is convex, we have

Proposition 1 characterizes the convergence rate of Algorithm 1 by the quantity A N , which can be further lower bounded in terms of the step sizes { ˆ η k } . Moreover, we can observe that larger step sizes will lead to a faster convergence rate. On the other hand, the step size ˆ η k is constrained by the condition in (7), which, in turn, depends on our choice of the Hessian approximation matrix B k . Thus, the central goal of our line search scheme and the Hessian approximation update is to make the step size ˆ η k as large as possible, which we will describe next.

3.1 Line Search Subroutine

In this section, we specify our line search subroutine to select the step size ˆ η k and the iterate ˆ x k +1 in the second stage of A-QNPE. For simplicity, denote ∇ f ( y k ) by g and drop the subscript k in y k and B k . In light of (6) and (7), our goal in the second stage is to find a pair (ˆ η, ˆ x + ) such that

As mentioned in the previous section, the condition in (10) can be satisfied by solving the linear system ( I + ˆ η B )(ˆ x + -y ) = -ˆ η g to a desired accuracy. Specifically, we let

where the oracle LinearSolver is defined as follows.

Definition 1. The oracle LinearSolver ( A , b ; α ) takes a matrix A ∈ S d + , a vector b ∈ R d and α ∈ (0 , 1) as input, and returns an approximate solution s + satisfying ∥ As + -b ∥ ≤ α ∥ s + ∥ .

The most direct way to implement LinearSolver ( A , b ; α ) is to compute s + = A -1 b , which however costs O ( d 3 ) arithmetic operations. Alternatively, we can implement the oracle more efficiently by using the conjugate residual method [48], which only requires computing matrix-vector products and thus incurs a cost of O ( d 2 ) . The details are discussed in Appendix E.1. We characterize the total number of required matrix-vector products for this oracle in Theorem 2.

Now we are ready to describe our line search scheme with the LinearSolver oracle (see also Subroutine 1 in Appendix B). Specifically, we start with the step size η and then reduce it by a factor β until we find a pair (ˆ η, ˆ x + ) that satisfies (11). It can be shown that the line search scheme will terminate in a finite number of steps and return a pair (ˆ η, ˆ x + ) satisfying both conditions in (10) and (11) (see Appendix B.1). Regarding the output, we distinguish two cases: (i) If we pass the test in (11) on our first attempt, we accept the initial step size η and the corresponding iterate ˆ x + (cf. Line 10 in Subroutine 1). (ii) Otherwise, along with the pair (ˆ η, ˆ x + ) , we also return an auxiliary iterate ˜ x + that we compute from (12) using the rejected step size ˆ η/β (cf. Line 12 in Subroutine 1). As we shall see in Lemma 1, the iterate ˜ x + is used to derive a lower bound on ˆ η , which will be the key to our convergence analysis and guide our update of the Hessian approximation matrix. For ease of notation, let B be the set of iteration indices where the line search scheme backtracks, i.e., B ≜ { k : ˆ η k < η k } .

Lemma 1. For k / ∈ B we have ˆ η k = η k , while for k ∈ B we have

Lemma1provides a lower bound on the step size ˆ η k in terms of the approximation error ∥∇ f (˜ x k +1 ) -∇ f ( y k ) -B k (˜ x k +1 -y k ) ∥ . Hence, a better Hessian approximation matrix B k leads to a larger step size, which in turn implies faster convergence. Also note that the lower bound uses the auxiliary iterate ˜ x k +1 that is not accepted as the actual iterate. Thus, the second inequality in (13) will be used to relate ∥ ˜ x k +1 -y k ∥ with ∥ ˆ x k +1 -y k ∥ . Finally, we remark that to fully characterize the computational cost, we need to upper bound the total number of line search steps, each of which requires a call to LinearSolver and a call to the gradient oracle. This will be discussed in Theorem 2.

3.2 Hessian Approximation Update via Online Learning with Dynamic Regret

In this section, we discuss how to update the Hessian approximation matrix B k in the fourth stage of A-QNPE. As mentioned earlier, instead of following the classical quasi-Newton updates, we directly motivate our update policy for B k from the convergence analysis. The first step is to connect the

convergence rate of A-QNPE with the Hessian approximation matrices { B k } . By Proposition 1, if we define the absolute constant C 1 ≜ 2(2 - √ β ) 2 (1 - √ β ) 2 , then we can write

where the last inequality follows from Hölder's inequality. Furthermore, we can establish an upper bound on ∑ N -1 k =0 1 ˆ η 2 k in terms of the Hessian approximation matrices { B k } , as we show next.

Lemma 2. Let { ˆ η k } N -1 k =0 be the step sizes in Algorithm 1 using Subroutine 1. Then we have

where w k ≜ ∇ f (˜ x k +1 ) -∇ f ( y k ) and s k ≜ ˜ x k +1 -y k for k ∈ B .

The proof of Lemma 2 is given in Appendix C.1. On a high level, for those step sizes ˆ η k with k ∈ B , we can apply Lemma 1 and directly obtain a lower bound in terms of B k . On the other hand, for k / ∈ B , we have ˆ η k = η k and our update rule in Lines 8 and 13 of Algorithm 1 allows us to connect the sequence { η k } N -1 k =0 with the backtracked step sizes { ˆ η k : k ∈ B} . As a result, we note that the sum in (15) only involves the Hessian approximation matrices { B k : k ∈ B} .

In light of (14) and (15), our update for B k aims to make the right-hand side of (15) as small as possible. To achieve this, we adopt the online learning approach in [35] and view the sum in (15) as the cumulative loss incurred by our choice of { B k } . To formalize, define the loss at iteration k by

and consider the following online learning problem: (i) At the k -th iteration, we choose B k ∈ Z where Z ≜ { B ∈ S d + : 0 ⪯ B ⪯ L 1 I } ; (ii) We receive the loss function ℓ k ( B ) defined in (16); (iii) We update our Hessian approximation matrix to B k +1 . Therefore, we propose to employ an online learning algorithm to update the Hessian approximation matrices { B k } , and the task of proving a convergence rate for our A-QNPE algorithm boils down to analyzing the performance of our online learning algorithm. In particular, an upper bound on the cumulative loss ∑ N -1 k =0 ℓ k ( B k ) will directly translate into a convergence rate for A-QNPE by using (14) and (15).

Naturally, the first idea is to update B k by following projected online gradient descent [49]. While this approach would indeed serve our purpose, its implementation could be computationally expensive. Specifically, like other projection-based methods, it requires computing the Euclidean projection onto the set Z in each iteration, which in our case amounts to performing a full d × d matrix eigendecomposition and would incur a cost of O ( d 3 ) (see Appendix C.2). Inspired by the recent work in [50], we circumvent this issue by using a projection-free online learning algorithm, which relies on an approximate separation oracle for Z instead of a projection oracle. For simplicity, we first translate and rescale the set Z via the transform ˆ B = 2 L 1 ( B -L 1 2 I ) to obtain ˆ Z ≜ { ˆ B ∈ S d : ∥ ˆ B ∥ op ≤ 1 } . The approximate separation oracle SEP ( W ; δ, q ) is then defined as follows.

Definition 2. The oracle SEP ( W ; δ, q ) takes a symmetric matrix W ∈ S d , δ > 0 , and q ∈ (0 , 1) as input and returns a scalar γ > 0 and a matrix S ∈ S d with one of the following possible outcomes:

Case I: γ ≤ 1 , which implies that, with probability at least 1 -q , W ∈ ˆ Z ;

Case II: γ > 1 , which implies that, with probability at least 1 -q , W /γ ∈ ˆ Z , ∥ S ∥ F ≤ 3 and ⟨ S , W -ˆ B ⟩ ≥ γ -1 -δ for any ˆ B such that ˆ B ∈ ˆ Z .

To sum up, SEP ( W ; δ, q ) has two possible outcomes: with probability 1 -q , either it certifies that W ∈ ˆ Z , or it produces a scaled version of W that belongs to ˆ Z and an approximate separation hyperplane between W and the set ˆ Z . As we show in Appendix E.2, implementing this oracle

requires computing the two extreme eigenvectors and eigenvalues of the matrix W inexactly, which can be implemented efficiently by the randomized Lanczos method [51].

Building on the SEP ( W ; δ, q ) oracle, we design a projection-free online learning algorithm adapted from [35, Subroutine 2]. Since the algorithm is similar to the one proposed in [35], we relegate the details to Appendix C but sketch the main steps in the analysis. To upper bound the cumulative loss ∑ N -1 k =0 ℓ k ( B k ) , we compare the performance of our online learning algorithm against a sequence of reference matrices { H k } N -1 k =0 . Specifically, we aim to control the dynamic regret [49, 52, 53] defined by D -Reg N ( { H k } N -1 k =0 ) ≜ ∑ N -1 k =0 ( ℓ k ( B k ) -ℓ k ( H k )) , as well as the the cumulative loss ∑ N -1 k =0 ℓ k ( H k ) by the reference sequence. In particular, in our analysis we show that the choice of H k ≜ ∇ 2 f ( y k ) for k = 0 , . . . , N -1 allows us to upper bound both quantities.

Remark 2 . While our online learning algorithm is similar to the one in [35], our analysis is more challenging due to the lack of strong convexity. Specifically, since f is assumed to be strongly convex in [35], the iterates converge to x ∗ at least linearly, resulting in less variation in the loss functions { ℓ k } . Hence, the authors in [35] let H k = H ∗ ≜ ∇ 2 f ( x ∗ ) for all k and proved that ∑ N -1 k =0 ℓ k ( H ∗ ) remains bounded. In contrast, without linear convergence, we need to use a time-varying sequence { H k } to control the cumulative loss. This in turn requires us to bound the variation ∑ N -2 k =0 ∥ H k +1 -H k ∥ F , which involves a careful analysis of the stability property of the sequence { y k } in Algorithm 1.

4 Complexity Analysis of A-QNPE

In this section, we present our main theoretical results: we establish the convergence rate of A-QNPE (Theorem 1) and characterize its computational cost in terms of gradient queries and matrix-vector product evaluations (Theorem 2). The proofs are provided in Appendices D and E.3.

Theorem 1. Let { x k } be the iterates generated by Algorithm 1 using the line search scheme in Section 3.1, where α 1 , α 2 ∈ (0 , 1) with α 1 + α 2 < 1 and β ∈ (0 , 1) , and using the Hessian approximation update in Section 3.2 (the hyperparameters are given in Appendix D). Then with probability at least 1 -p , the following statements hold, where C i ( i = 4 , . . . , 10 ) are absolute constants only depending on α 1 , α 2 and β .

(a) For any k ≥ 0 , we have

(b) Furthermore, for any k ≥ 0 ,

where we define log + ( x ) ≜ max { log( x ) , 0 } and the quantity M is given by

Both results in Theorem 1 are global, as they are valid for any initial points x 0 , z 0 and any initial matrix B 0 . Specifically, Part (a) of Theorem 1 shows that A-QNPE converges at a rate of O (1 /k 2 ) , matching the rate of NAG [23] that is known to be optimal in the regime where k = O ( d ) [28, 29]. Furthermore, Part (b) of Theorem 1 presents a convergence rate of O ( √ d log( k ) /k 2 . 5 ) . To see this, note that since we have 0 ⪯ B 0 ⪯ L 1 I and 0 ⪯ ∇ 2 f ( z 0 ) ⪯ L 1 I , in the worst case ∥ B 0 -∇ 2 f ( z 0 ) ∥ 2 F in the expression of (19) can be upper bounded by L 2 1 d . Thus, assuming that L 1 , L 2 and ∥ z 0 -x ∗ ∥ are on the order of O (1) , we have M = O ( d ) and the convergence rate in (18) can be simplified to O ( √ d log( k ) /k 2 . 5 ) . Notably, this rate surpasses the O (1 /k 2 ) rate when k = Ω( d log d ) . To the best of our knowledge, this is the first work to show a convergence rate faster than O (1 /k 2 ) for a quasi-Newton-type method in the convex setting, thus establishing a provable advantage over NAG.

Remark 3 (Iteration complexity) . Based on Theorem 1, we can find A-QNPE's iteration complexity. Define N ϵ as the number of iterations required by A-QNPE to find an ϵ -accurate solution, i.e., f ( x ) -f ( x ∗ ) ≤ ϵ . When √ ϵ > 1 d log d , the rate in (17) is better and we have N ϵ = O ( 1 ϵ 0 . 5 ) . Conversely, when √ ϵ < 1 d log d , the rate in (18) is the better one, resulting in N ϵ = O (( d ϵ 2 log d ϵ 2 ) 0 . 2 ) . Hence, to achieve an ϵ -accurate solution A-QNPE requires O (min { 1 ϵ 0 . 5 , d 0 . 2 ϵ 0 . 4 (log d ϵ 2 ) 0 . 2 } ) iterations.

Remark 4 (Special case) . If the initial point z 0 is close to an optimal solution x ∗ and the initial Hessian approximation matrix B 0 is chosen properly, the dependence on d in the convergence rate of (18) can be eliminated. Specifically, if ∥ z 0 -x ∗ ∥ = O ( 1 d ) and we set B 0 = ∇ 2 f ( z 0 ) , then we have M = O (1) and this leads to a local dimension-independent rate of O ( √ log k/k 2 . 5 ) .

Recall that in each iteration of Algorithm 1, we need to execute a line search subroutine (Section 3.1) and a Hessian approximation update subroutine (Section 3.2). Thus, to fully characterize the computational cost of Algorithm 1, we need to upper bound the total number of gradient queries as well as the total number of matrix-vector product evaluations, which is the goal of Theorem 2.

Theorem 2. Recall that N ϵ denotes the minimum number of iterations required by Algorithm 1 to find an ϵ -accurate solution according to Theorem 1. Then, with probability at least 1 -p :

(a) The total number of gradient queries is bounded by 3 N ϵ +log 1 /β ( σ 0 L 1 α 2 ) .

(b) The total number of matrix-vector product evaluations in the LinearSolver oracle is bounded by N ϵ + C 11 √ σ 0 L 1 + C 12 √ L 1 ∥ z 0 -x ∗ ∥ 2 2 ϵ , where C 11 and C 12 are absolute constants.

(c) The total number of matrix-vector product evaluations in the SEP oracle is bounded by O ( N 1 . 25 ϵ (log N ϵ ) 0 . 5 log ( √ dN ϵ p )) .

If the initial step size is chosen as σ 0 = α 2 L 1 , Theorem 2(a) implies that A-QNPE requires no more than 3 gradient queries per iteration on average. Thus, the gradient oracle complexity of A-QNPE is the same as the iteration complexity, i.e., O (min { 1 ϵ 0 . 5 , d 0 . 2 ϵ 0 . 4 (log d ϵ 2 ) 0 . 2 } ) . On the other hand, the complexity in terms of matrix-vector products is worse. More precisely, by using the expression of N ϵ in Remark 3, Parts (b) and (c) imply that the total number of matrix-vector product evaluations in the LinearSolver and SEP oracles can be bounded by O ( 1 ϵ 0 . 5 ) and ˜ O (min { d 0 . 25 ϵ 0 . 5 , 1 ϵ 0 . 625 } ) , respectively.

For easier comparison, we summarize the detailed computational costs of NAG and our method A-QNPE to achieve an ϵ -accuracy in Table 1. We observe that A-QNPE outperforms NAG in terms of gradient query complexity: It makes equal or fewer gradient queries especially when ϵ ≪ 1 d 2 . On the other hand, A-QNPE requires additional matrix-vector product computations to implement the LinearSolver and SEP oracles. While this is a limitation of our method, in some cases, gradient evaluations are the main bottleneck and can be more expensive than matrix-vector products. As a concrete example, consider the finite-sum minimization problem f ( x ) = 1 n ∑ n i =1 f i ( x ) . In this case, one gradient query typically costs O ( nd ) , while one matrix-vector product costs O ( d 2 ) . Thus, the total computational cost of NAG and A-QNPE can be bounded by O ( nd ϵ 0 . 5 ) and ˜ O ( nd 1 . 2 ϵ 0 . 4 + d 2 . 25 ϵ 0 . 5 ) , respectively. In particular, our method incurs a lower computational cost when ϵ ≪ 1 d 2 and n ≫ d 1 . 25 .

5 Experiments

In this section, we compare the numerical performance of our proposed A-QNPE method with NAG and the classical BFGS quasi-Newton method. For fair comparison, we also use a line search

0

500

1000

1500

2000

2500

10 -15

10 -10

10 -5

10 0

10 5

0

500

1000

1500

2000

2500

10 -15

10 -10

10 -5

10 0

10 5

0.6

0.5

0.4

0.3

0.2

0.1

0

2

3

4

5

6

7

Gradient queries

(a) Convergence by iteration

(b) Convergence by gradient queries

(c) Histogram of gradient queries

0

0.5

1

1.5

2

2.5

3

10 4

10 -15

10 -10

10 -5

10 0

10 5

(a) Convergence by iteration

0

0.5

1

1.5

2

2.5

3

10 4

10 -15

10 -10

10 -5

10 0

10 5

(b) Convergence by gradient queries

(c) Histogram of gradient queries

0.6

0.5

0.4

0.3

0.2

0.1

0

2

Gradient queries

4

6

8

scheme in NAG and BFGS to obtain their best performance [54, 15]. We would like to highlight that our paper mainly focuses on establishing a provable gain for quasi-Newton methods with respect to NAG, and our experimental results are presented to numerically verify our theoretical findings. In the first experiment, we focus on a logistic regression problem with the loss function f ( x ) = 1 n ∑ n i =1 log(1 + e -y i ⟨ a i , x ⟩ ) , where a 1 , . . . , a n ∈ R d are feature vectors and y 1 , . . . , y n ∈ {-1 , 1 } are binary labels. We perform our numerical experiments on a synthetic dataset and the data generation process is described in Appendix F. In the second experiment, we consider the log-sum-exp function f ( x ) = log( ∑ n i =1 e ⟨ a i , x ⟩-b i ) , where we generate the dataset { ( a i , b i ) } n i =1 following a similar procedure as in [16] (more details in Appendix F). As we observe in Fig. 1(a) and Fig. 2(a), our proposed A-QNPE method converges in much fewer iterations than NAG, while the best performance is achieved by BFGS. Due to the use of line search, we also compare these algorithms in terms of the total number of gradient queries. Moreover, additional plots in terms of the running time are included in Appendix F. As illustrated in Fig. 1(b) and Fig. 2(b), A-QNPE still outperforms NAG but the relative gain becomes less substantial. This is because the line search scheme in NAG only queries the function value at the new point, and thus it only requires one gradient per iteration. On the other hand, we should add that the number of gradient queries per iteration for A-QNPE is still small as guaranteed by our theory. In particular, the histogram of gradient queries in Fig. 1(c) and Fig. 2(c) shows that most of the iterations of A-QNPE require 2-3 gradient queries with an average of less than 3 . Finally, although there is no theoretical guarantee showing a convergence gain for BFGS with respect to NAG, we observe that BFGS outperforms all the other considered methods in our experiments. Hence, studying the convergence behavior of BFGS (with line search) in the convex setting is an interesting research direction to explore.

6 Conclusions

We proposed a quasi-Newton variant of the accelerated proximal extragradient method for solving smooth convex optimization problems. We established two global convergence rates for our A-QNPE method, showing that it requires ˜ O (min { 1 ϵ 0 . 5 , d 0 . 2 ϵ 0 . 4 } ) gradient queries to find an ϵ -accurate solution. In particular, in the regime where ϵ = Ω( 1 d 2 ) , A-QNPE achieves a gradient oracle complexity of O ( 1 ϵ 0 . 5 ) , matching the complexity of NAG. Moreover, in the regime where ϵ = ˜ O ( 1 d 2 ) , it outperforms NAG and improves the complexity to ˜ O ( d 0 . 2 ϵ 0 . 4 ) . To the best of our knowledge, this is the first result showing a provable gain for a quasi-Newton-type method over NAG in the convex setting.

Acknowledgments and Disclosure of Funding

The research of R. Jiang and A. Mokhtari is supported in part by NSF Grants 2007668, 2019844, and 2112471, ARO Grant W911NF2110226, the Machine Learning Lab (MLL) at UT Austin, the Wireless Networking and Communications Group (WNCG) Industrial Affiliates Program, and the NSF AI Institute for Foundations of Machine Learning (IFML). The authors would also like to thank Yair Carmon and the anonymous reviewers for their comments on the first draft of the paper.

References

W. C. Davidon. Variable metric method for minimization. Techinical Report ANL-5990, Argonne National Laboratory, Argonne, IL, 1959.

Roger Fletcher and Michael JD Powell. A rapidly convergent descent method for minimization. The Computer Journal , 6(2):163-168, 1963.

Charles G Broyden. The convergence of single-rank quasi-newton methods. Mathematics of Computation , 24(110):365-382, 1970.

Roger Fletcher. A new approach to variable metric algorithms. The computer journal , 13(3):317322, 1970.

Donald Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of computation , 24(109):23-26, 1970.

David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of computation , 24(111):647-656, 1970.

Andrew R Conn, Nicholas IM Gould, and Ph L Toint. Convergence of quasi-newton matrices generated by the symmetric rank one update. Mathematical programming , 50(1):177-195, 1991.

H Fayez Khalfan, Richard H Byrd, and Robert B Schnabel. A theoretical and experimental study of the symmetric rank-one update. SIAM Journal on Optimization , 3(1):1-24, 1993.

Charles George Broyden, John E Dennis Jr, and Jorge J Moré. On the local and superlinear convergence of quasi-newton methods. IMA Journal of Applied Mathematics , 12(3):223-245, 1973.

John E Dennis and Jorge J Moré. A characterization of superlinear convergence and its application to quasi-newton methods. Mathematics of computation , 28(126):549-560, 1974.

MJD Powell. On the convergence of the variable metric algorithm. IMA Journal of Applied Mathematics , 7(1):21-36, 1971.

Laurence Charles Ward Dixon. Variable metric algorithms: necessary and sufficient conditions for identical behavior of nonquadratic functions. Journal of Optimization Theory and Applications , 10(1):34-40, 1972.

Richard H Byrd, Jorge Nocedal, and Ya-Xiang Yuan. Global convergence of a class of quasinewton methods on convex problems. SIAM Journal on Numerical Analysis , 24(5):1171-1190, 1987.

Richard H Byrd, Humaid Fayez Khalfan, and Robert B Schnabel. Analysis of a symmetric rank-one trust region method. SIAM Journal on Optimization , 6(4):1025-1039, 1996.

Jorge Nocedal and Stephen J. Wright. Numerical Optimization . Springer Science+Business Media, LLC, 2006.

Anton Rodomanov and Yurii Nesterov. Greedy quasi-newton methods with explicit superlinear convergence. SIAM Journal on Optimization , 31(1):785-811, 2021.

Anton Rodomanov and Yurii Nesterov. Rates of superlinear convergence for classical quasinewton methods. Mathematical Programming , 2021.

Anton Rodomanov and Yurii Nesterov. New results on superlinear convergence of classical quasi-newton methods. Journal of Optimization Theory and Applications , 188(3):744-769, 2021.

Qiujiang Jin and Aryan Mokhtari. Non-asymptotic superlinear convergence of standard quasinewton methods. Mathematical Programming , 2022.

Qiujiang Jin, Alec Koppel, Ketan Rajawat, and Aryan Mokhtari. Sharpened quasi-Newton methods: Faster superlinear rate and larger local convergence neighborhood. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 10228-10250. PMLR, 2022.

Dachao Lin, Haishan Ye, and Zhihua Zhang. Greedy and random quasi-newton methods with faster explicit superlinear convergence. Advances in Neural Information Processing Systems , 34:6646-6657, 2021.

Haishan Ye, Dachao Lin, Xiangyu Chang, and Zhihua Zhang. Towards explicit superlinear convergence rate for sr1. Mathematical Programming , pages 1-31, 2022.

Yurii Nesterov. A method of solving a convex programming problem with convergence rate O (1 /k 2 ) . Soviet Mathematics Doklady , pages 372-376, 1983.

M.J.D. Powell. Some properties of the variable metric algorithm. Numerical methods for nonlinear optimization , pages 1-17, 1972.

Katya Scheinberg and Xiaocheng Tang. Practical inexact proximal quasi-newton method with global complexity analysis. Mathematical Programming , 160:495-529, 2016.

Hiva Ghanbari and Katya Scheinberg. Proximal quasi-newton methods for regularized convex optimization with linear and accelerated sublinear convergence rates. Computational Optimization and Applications , 69:597-627, 2018.

Dmitry Kamzolov, Klea Ziu, Artem Agafonov, and Martin Takáˇ c. Accelerated adaptive cubic regularized quasi-newton methods. arXiv preprint arXiv:2302.04987 , 2023.

A. Nemirovski and D Yudin. Problem Complexity and Method Efficiency in Optimization . Wiley, 1983.

Yurii Nesterov. Lectures on Convex Optimization . Springer International Publishing, 2018.

Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming , 184(1-2):71-120, 2020.

Renato DC Monteiro and Benar Fux Svaiter. An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM Journal on Optimization , 23(2):1092-1125, 2013.

Yair Carmon, Danielle Hausler, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Optimal and adaptive monteiro-svaiter acceleration. In Advances in Neural Information Processing Systems , 2022.

Osman Güler. New proximal point algorithms for convex minimization. SIAM Journal on Optimization , 2(4):649-664, 1992.

Saverio Salzo and Silvia Villa. Inexact and accelerated proximal point algorithms. Journal of Convex analysis , 19(4):1167-1192, 2012.

Ruichen Jiang, Qiujiang Jin, and Aryan Mokhtari. Online learning guided curvature approximation: A quasi-newton method with global non-asymptotic superlinear convergence. arXiv preprint arXiv:2302.08580 , 2023.

D. Newman. Location of the maximum on unimodal surfaces. Journal of the ACM , 12(3):395398, 1965.

A. Levin. On an algorithm for the minimization of convex functions. Soviet Mathematics Doklady , 160:1244-1247, 1965.

David B Yudin and Arkadii S Nemirovski. Evaluation of the information complexity of mathematicalprogramming problems. Ekonomika i Matematicheskie Metody , 12:128-142, 1976.

Naum Z Shor. Cut-off method with space extension in convex programming problems. Cybernetics and systems analysis , 13(1):94-96, 1977.

Leonid G Khachiyan. Polynomial algorithms in linear programming. USSR ComputationalMathematics and Mathematical Physics , 20(1):53-72, 1980.

Pravin M Vaidya. A new algorithm for minimizing convex functions over convex sets. Mathematical programming , 73(3):291-341, 1996.

David S Atkinson and Pravin M Vaidya. A cutting plane algorithm for convex programming that uses analytic centers. Mathematical programming , 69(1-3):1-43, 1995.

Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 1049-1065. IEEE, 2015.

Haotian Jiang, Yin Tat Lee, Zhao Song, and Sam Chiu-wai Wong. An improved cutting plane method for convex optimization, convex-concave games, and its applications. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 944-953, 2020.

Bo Jiang, Haoyue Wang, and Shuzhong Zhang. An optimal high-order tensor method for convex optimization. Mathematics of Operations Research , 46(4):1390-1412, 2021.

Sébastien Bubeck, Qijia Jiang, Yin Tat Lee, Yuanzhi Li, and Aaron Sidford. Near-optimal method for highly smooth convex optimization. In Conference on Learning Theory , pages 492-507. PMLR, 2019.

Alexander Gasnikov, Pavel Dvurechensky, Eduard Gorbunov, Evgeniya Vorontsova, Daniil Selikhanovych, and César A Uribe. Optimal tensor methods in smooth convex and uniformly convexoptimization. In Conference on Learning Theory , pages 1374-1391. PMLR, 2019.

Yousef Saad. Iterative Methods for Sparse Linear Systems . Society for Industrial and Applied Mathematics, second edition, 2003.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936, 2003.

Zakaria Mhammedi. Efficient projection-free online convex optimization with membership oracle. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 5314-5390. PMLR, 2022.

J. Kuczy´ nski and H. Wo´ zniakowski. Estimating the Largest Eigenvalue by the Power and Lanczos Algorithms with a Random Start. SIAM Journal on Matrix Analysis and Applications , 13(4):1094-1122, 1992.

Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 7195-7201. IEEE, 2016.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. Advances in Neural Information Processing Systems , 33:12510-12520, 2020.

Amir Beck. First-Order Methods in Optimization . Society for Industrial and Applied Mathematics, 2017.

Alexandre d'Aspremont, Damien Scieur, and Adrien Taylor. Acceleration methods. Foundations and Trends® in Optimization , 5(1-2):1-245, 2021.

Arkadi Nemirovski. Information-based complexity of convex programming. Lecture notes , 834, 1995.

Yousef Saad. Numerical methods for large eigenvalue problems: revised edition . SIAM, 2011.

Alp Yurtsever, Joel A Tropp, Olivier Fercoq, Madeleine Udell, and Volkan Cevher. Scalable semidefinite programming. SIAM Journal on Mathematics of Data Science , 3(1):171-200, 2021.

Appendix

A Optimal Monteiro-Svaiter Acceleration Framework

In this section, we present some general results that hold for the optimal MS Acceleration framework. In particular, in the first part of this section (Section A.1), we present the proof of Proposition 1. In the second part (Section A.2), we further provide some useful additional lemmas.

A.1 Proof of Proposition 1

To begin with, we establish a potential function for Algorithm 1, as shown in Proposition 2. The result is similar to Proposition 1 in [32], but for completeness we present its proof loosely following the strategy in [55, Theorem 5.3]. To simplify the notations, we use f ∗ to denote the optimal f ( x ∗ ) .

Proposition 2. Consider the iterates generated by Algorithm 1. If f is convex, then

Moreover, let σ = α 1 + α 2 and we have

Proof. Since f is convex, it holds that

By summing up the two inequalities with weights a k and A k respectively, we get

Let ˜ z k +1 = ˆ x k +1 + A k a k (ˆ x k +1 -x k ) . By rearranging the terms, (22) can be rewritten as

Moreover, note that the update rule for z k +1 in both (8) and (9) can be written as

Also, since we also have z k = y k + A k a k ( y k -x k ) from (2), we can write

where we used the fact that ( A k + a k ) η k = a 2 k in the last equality (cf. (2)). Hence, combining (24) and (25) leads to

where we used (7) in the last inequality. In the following, we distinguish two cases depending on ˆ η k = η k or ˆ η k < η k . In both cases, we shall prove that

If this is true, then Proposition 2 immediately follows. Indeed, since σ < 1 , the last term in the right-hand side of (27) is negative, which implies (20). Moreover, (21) follows from summing the inequality in (27) from k = 0 to N -1 .

where we used (25) and (26) in the last inequality. This immediately leads to (27) after rearranging the terms.

Case II: ˆ η k < η k . Since 0 < γ k < 1 and x k +1 = (1 -γ k ) A k A k + γ k a k x k + γ k ( A k + a k ) A k + γ k a k ˆ x k +1 according to (9), by Jensen's inequality we have ( A k + γ k a k ) f ( x k +1 ) ≤ γ k ( A k + a k ) f (ˆ x k +1 ) + (1 -γ k ) A k f ( x k ) , which further implies that

Moreover, since A k +1 = A k + γ k a k by (9), together with (23) we obtain

which also leads to (27).

Next, we prove a lower bound on A N . Recall that B denotes the set of iteration indices where the line search scheme backtracks, i.e., B ≜ { k : ˆ η k < η k } .

Lemma 3. For any N ≥ 0 , it holds that

Proof. To begin with, according to the update rule of A k +1 in (8) and (9) and the expression of a k in (2), the sequence { A k } follows the dynamic:

Since we initialize A 0 = 0 , we have a 0 = η 0 . We further have A 1 = ˆ η 0 , since we get A 1 = A 0 + a 0 = ˆ η 0 if 0 / ∈ B , while we get A 1 = A 0 + γ 0 a 0 = ˆ η 0 η 0 η 0 = ˆ η 0 if 0 ∈ B . Moreover:

In Case I where k / ∈ B , we have

which further implies that √ A k +1 ≥ √ A k + √ η k 2 = √ A k + √ ˆ η k 2 .

In Case II where k ∈ B , we have A k +1 = A k + γ k a k ≥ A k , which implies that √ A k +1 ≥ √ A k .

Lemma 3 provides a lower bound on A N in terms of the step sizes ˆ η k in those iterations where the line search scheme does not backtrack, i.e., k / ∈ B . The following lemma shows how we can further prove a lower bound in terms of all the step sizes { ˆ η k } N -1 k =0 .

As a corollary, we have

Proof. When the line search scheme backtracks, i.e., k ∈ B , we have ˆ η k ≤ βη k . Therefore,

Moreover, in the update of Algorithm 1, we have η k +1 = ˆ η k /β if k / ∈ B (cf. Line 8) and η k +1 = ˆ η k otherwise (cf. Line 13). This implies that η 1 ≤ ˆ η 0 /β and we further have

We combine (32) and (33) to get

By rearranging the terms and simple algebraic manipulation, we obtain (30) as desired. Finally, (31) follows by adding √ ˆ η 0 + ∑ 1 ≤ k ≤ N -1 ,k/ ∈B √ ˆ η k to both sides of (30).

Now we are ready to prove Proposition 1.

Proof of Proposition 1. By Proposition 2, the potential function ϕ k ≜ A k ( f ( x k ) -f ∗ )+ 1 2 ∥ z k -x ∗ ∥ 2 is non-increasing in each iteration. Hence, via a recursive augment we have A N ( f ( x N ) -f ∗ ) ≤ ϕ N ≤ · · · ≤ ϕ 0 = 1 2 ∥ z 0 -x ∗ ∥ 2 , which yields f ( x N ) -f ∗ ≤ ∥ z 0 -x ∗ ∥ 2 2 A N . Moreover, combining Lemma 3 and (31) in Lemma 4 leads to the second inequality in Proposition 1.

A.2 Additional Supporting Lemmas

A crucial part of our analysis is to bound the path length of the sequence { y k } N k =0 . This is done in Lemma 8. To achieve this goal we first present the results in Lemmas 5-7, which provide the required ingredients for proving the claim in Lemma 8. In our first intermediate result, we establish uniform upper bounds for the error terms ∥ z k -x ∗ ∥ and ∥ x k -x ∗ ∥ .

Proof. To begin with, it follows from (20) in Proposition 2 that

Hence, we get ∥ z k -x ∗ ∥ ≤ ∥ z 0 -x ∗ ∥ for any k ≥ 0 . To show the second inequality, we distinguish two cases and in both cases we will prove that

Case I : ˆ η k = η k . Recall that in the proof of Proposition 2 we defined ˜ z k +1 = ˆ x k +1 + A k a k (ˆ x k +1 -x k ) .

Since x k +1 = ˆ x k +1 , we have x k +1 = A k A k + a k x k + a k A k + a k ˜ z k +1 and by Jensen's inequality

Furthermore, we have

where we used (26) in the last inequality. By combining the above two inequalities, we obtain

which leads to (34) (note that A k +1 = A k + a k in Case I ).

have

Similarly, by Jensen's inequality we have

Combining this inequality with (35), we obtain

which leads to (34) (note that A k +1 = A k + γ k a k in Case II ).

Now by summing (34) over k = 0 , . . . , N -1 , we get

A key term appearing in several of our bounds is a k +1 A k +1 + a k +1 . In the next lemma, we establish an upper bound for this ratio based on a factor of its previous value, for both cases of our algorithm.

Lemma 6. Without loss of generality assume β > 1 / 5 . In Case I we have a k +1 A k +1 + a k +1 ≤ 1 √ β a k A k + a k . Otherwise, in Case II we have a k +1 A k +1 + a k +1 ≤ 2 √ β √ β +1 a k A k + a k .

Proof. By the choice of a k in (2) we have η k ( A k + a k ) = a 2 k for all k ≥ 0 . As a result, we have

and similarly

In Case I , we have η k +1 = η k /β and A k +1 ≥ A k . Hence, it implies that A k +1 /η k +1 ≥ βA k /η k , which leads to

where the second inequality follows from the fact that β ≤ 1 .

In Case II , we have η k +1 = ˆ η k = γ k η k and A k +1 = A k + γ k a k . Since we also have a k ≥ η k and γ k ≤ β , we obtain A k +1 /η k +1 ≥ A k / ( γ k η k ) + 1 ≥ A k / ( βη k ) + 1 . Hence,

where we used β > 1 / 5 in the second inequality and the fact that 1 + 1 √ β x ≥ √ β +1 2 √ β (1 + x ) for x ≥ 1 in the last inequality.

Remark 5 . If β ≤ 1 / 5 , then in Case I we still have a k +1 A k +1 + a k +1 ≤ 1 √ β a k A k + a k , while in Case II we have a k +1 A k +1 + a k +1 ≤ 2 √ 5+1 a k A k + a k . Thus, in the case where β ≤ 1 / 5 , the derivation below still holds except that the absolute constant C 2 will be different.

Next, as a corollary of Lemma 6, we establish an upper bound on the series ∑ N -1 k =0 a k A k + a k . Moreover, we use this result to establish an upper bound for ∑ N -1 k =0 ∥ ˆ x k +1 -y k ∥ .

Lemma 7. We have

Moreover,

Proof. Given the initial values of A k and a k we have

Note that using the result in Lemma 6

Hence, if we move the last term in the above upper bound to the left hand side and rescale both sides of the resulted inequality we obtain

Now, if we replace the above upper bound into (43) we obtain

Moreover, note that for k / ∈ B , we have A k +1 = A k + a k . Hence,

Now if we replace the above upper bound, i.e., log A N A 1 with ∑ k/ ∈B ,k ≥ 1 a k A k + a k into the expression in the right-hand side of (48) we obtain the result in (41).

Next, note that by Cauchy-Schwarz inequality, we have

where the last inequality follows from (21). Moreover, based on the expression for a k in (2) and the result in (41) that we just proved, we have

Combining the two inequalities above leads to (42).

Now we are ready to present and prove Lemma 8 which characterizes a bound on the path length of the sequence { y k } N k =0

Lemma 8. Consider the iterates generated by Algorithm 1. Then for any N ,

where

Proof. By the triangle inequality, we have

We again distinguish two cases.

Case I: ˆ η k = η k . In this case ˆ x k +1 = x k +1 and y k +1 = A k +1 A k +1 + a k +1 x k +1 + a k +1 A k +1 + a k +1 z k +1 , hence

where we used Lemma 6 and the fact that ∥ z k +1 -x k +1 ∥ ≤ ∥ z k +1 -x ∗ ∥ + ∥ x k +1 -x ∗ ∥ ≤ (1 + √ 2 1 -σ 2 ) ∥ z 0 -x ∗ ∥ in the last inequality. Therefore, using (50) and the above bound we have

Case II: ˆ η k < η k . Since x k +1 = A k A k + γ k a k x k + γ k a k A k + γ k a k ˜ z k +1 and ˆ x k +1 = A k A k + a k x k + a k A k + a k ˜ z k +1 , we get

Thus, given the above equality and the expression y k +1 = A k +1 A k +1 + a k +1 x k +1 + a k +1 A k +1 + a k +1 z k +1 , we have

Moreover, based on the result in (26), we can upper bound ∥ ˜ z k +1 -z k +1 ∥ by σ a k η k ∥ ˆ x k +1 -y k ∥ which implies that

where the equality holds due to the definition of a k , and the last inequality holds as both γ k and σ are in (0 , 1) . On the other hand, note that

where in the second bound we used the result in Lemma 6 and the fact that 2 sqrtβ √ β +1 < 1 . Hence, we get

where the last inequality follows from the fact ∥ z k +1 -x k +1 ∥ ≤ ∥ z k +1 -x ∗ ∥ + ∥ x k +1 -x ∗ ∥ and the bounds in Lemma 5. Now by applying the above upper bound into (50) we obtain that

Considering the upper bounds established for ∥ y k -y k +1 ∥ in case I (equation (51)) and case II (equation (55)), we can conclude that

Finally, Lemma 8 follows from summing (56) over k = 0 to N -1 and the result of Lemma 7.

Subroutine 1 Backtracking line search

1: Input: iterate y ∈ R d , gradient g ∈ R d , Hessian approximation B ∈ S d + , initial trial step size η > 0 2: Parameters: line search parameters β ∈ (0 , 1) , α 1 ≥ 0 and α 2 > 0 such that α 1 + α 2 < 1 3: Set ˆ η ← η 4: Compute s + ← LinearSolver ( I + ˆ η B , -ˆ η g ; α 1 ) and ˆ x + ← y + s + 5: while ∥ ˆ x + -y + ˆ η ∇ f (ˆ x + ) ∥ 2 > ( α 1 + α 2 ) ∥ ˆ x + -y ∥ 2 do 6: Set ˜ x + ← ˆ x + and ˆ η ← β ˆ η 7: Compute s + ← LinearSolver ( I + ˆ η B , -ˆ η g ; α 1 ) and ˆ x + ← y + s + 8: end while 9: if ˆ η = η then 10: Return ˆ η and ˆ x + 11: else 12: Return ˆ η , ˆ x + and ˜ x + 13: end if

B Line Search Subroutine

In this section, we provide further details on our line search subroutine in Section 3.1. For completeness, the pseudocode of our line search scheme is shown in Subroutine 1. In Section B.1, we prove that Subrountine 1 will always terminate in a finite number of steps. In Section B.2, we provide the proof of Lemma 1.

B.1 The Line Search Subroutine Terminates Properly

Recall that in our line search scheme, we keep decreasing the step size ˆ η by a factor of β until we find a pair (ˆ η, ˆ x + ) satisfying (11) (also see Lines 5 and 6 in Subroutine 1). In the following lemma, we show that when the step size ˆ η is smaller than a certain threshold, then the pair (ˆ η, ˆ x + ) satisfies both conditions in (10) and (11), which further implies that Subroutine 1 will stop in a finite number of steps.

Lemma 9. Suppose Assumption 1 holds. If ˆ η < α 2 L 1 + ∥ B ∥ op and ˆ x + is computed according to (12) , then the pair (ˆ η, ˆ x + ) satisfies the conditions in (10) and (11) .

Proof. By Definition 1, the pair (ˆ η, ˆ x + ) always satisfies the condition in (10) when ˆ x + is computed from (12). Hence, in the following we only need to prove that the condition in (11) also holds. Recall that g = ∇ f ( y ) . By Assumption 1, the function f is L 1 -smooth and thus we have

Moreover, by using the triangle inequality, we get

Finally, by using the triangle inequality, we can combine (10) and (57) to show that

which means the condition in (11) is satisfied. The proof is now complete.

B.2 Proof of Lemma 1

We follow a similar proof strategy as Lemma 3 in [35]. In the first case where k / ∈ B , by definition, the line search subroutine accepts the initial step size η k , i.e., ˆ η k = η k . In the second case where k ∈ B , the line search subroutine backtracks and returns the auxiliary iterate ˜ x k +1 , which is computed from

(12) using the step size ˜ η k ≜ ˆ η k /β . Since the step size ˜ η k is rejected in our line search subroutine, it implies that the pair (˜ x k +1 , ˜ η k ) does not satisfy (11), i.e.,

Moreover, since we compute ˜ x k +1 from (12) using step size ˜ η k , the pair (˜ η k , ˜ x k +1 ) also satisfies the condition in (10), which means

Hence, by using the triangle inequality, we can combine (58) and (59) to get

which implies that

This proves the first inequality in (13).

To show the second inequality in (13), first note that ˜ x k +1 and ˆ x k +1 are the inexact solutions of the linear system of equations

respectively. Let ˜ x ∗ k +1 and ˆ x ∗ k +1 be the exact solutions of the above linear systems, that is, ˜ x ∗ k +1 = y k -˜ η k ( I + ˜ η k B k ) -1 g k and ˆ x ∗ k +1 = y k -ˆ η k ( I + ˆ η k B k ) -1 g k . We first establish the following inequality between ∥ ˜ x ∗ k +1 -y k ∥ and ∥ ˆ x ∗ k +1 -y k ∥ :

This follows from

where we used the fact that ( I + ˜ η k B k ) -1 ⪯ ( I + ˆ η k B k ) -1 in the first inequality. Furthermore, we can show that

We will only prove (61) in the following, as (62) can be proved similarly. Note that since (ˆ η k , ˆ x k +1 ) satisfies the condition in (10), we can write

Moreover, since B k ⪰ 0 , we have ∥ ˆ x k +1 -ˆ x ∗ k +1 ∥ ≤ ∥ ( I +ˆ η k B k )(ˆ x k +1 -ˆ x ∗ k +1 ) ∥ ≤ α 1 ∥ ˆ x k +1 -y k ∥ . Thus, by the triangle inequality, we obtain

which proves (61). Finally, by combining (60), (61) and (62), we conclude that

This completes the proof.

C Hessian Approximation Update

In this section, we first prove Lemma 2 in Section C.1 and remark on the computational cost of Euclidean projection in Section C.2. Then we present a general online learning algorithm using an approximate separation oracle in Section C.3 and fully describe our Hessian approximation update in Section C.4.

C.1 Proof of Lemma 2

We decompose the sum ∑ N -1 k =0 1 ˆ η 2 k as

Recall that we have ˆ η k = η k for k / ∈ B . Hence, we can further bound the last term by

Recall that we have η k +1 = ˆ η k if k ∈ B and η k +1 = ˆ η k /β otherwise. Hence, we further have

By moving the last term to the left-hand side and dividing both sides by 1 -β 2 , we obtain

Furthermore, since η 1 ≥ ˆ η 0 , we have 1 η 2 1 ≤ 1 ˆ η 2 0 . Hence, by combining (63) and (64), we get

where in the last inequality we used the fact that ˆ η k = σ 0 if 0 / ∈ B . Finally, (15) follows from Lemma 1 and (65).

C.2 The Computational Cost of Euclidean Projection

Recall that Z ≜ { B ∈ S d + : 0 ⪯ B ⪯ L 1 I } . As described in [35, Section D.1], the Euclidean projection on Z has a closed form solution. Specifically, Given the input A ∈ S d , we first need to perform the eigendecomposition A = VΛV ⊤ , where V is an orthogonal matrix and Λ = diag( λ 1 , . . . , λ d ) is a diagonal matrix. Then the Euclidean projection of A onto Z is given by V ˆ ΛV ⊤ , where ˆ Λ is a diagonal matrix with the diagonals being ˆ λ k = min { L 1 , max { 0 , λ k }} for 1 ≤ k ≤ d . Since the eigendecomposition requires O ( d 3 ) arithmetic operations in general, the cost of computing the Euclidean projection can be prohibitive.

Algorithm 2 Projection-Free Online Learning

C.3 Online Learning with an Approximate Separation Oracle

To set the stage for our Hessian approximation matrix update, we first describe a projection-free online learning algorithm in a general setup. Specifically, the online learning protocol is as follows: For rounds t = 0 , 1 , . . . , T -1 , a learner chooses an action x t ∈ C from a convex set C and then observes a loss function ℓ t : R n → R . We measure the performance of an online learning algorithm by the dynamic regret [49, 52] defined by

where { u t } T t =1 is a sequence of comparators. Moreover, we assume that the convex set C is contained in the Euclidean ball B R (0) for some R > 0 , and we assume 0 ∈ C without loss of generality.

Most existing online learning algorithms are projection-based, that is, they require computing the Euclidean projection on the action set C . However, as we have seen in Section C.2, computing the projection is computationally costly in our setting. Inspired by the work in [50], we will describe an online learning algorithm that relies on an approximate separation oracle defined in Definition 3.

Definition 3. The oracle SEP ( w ; δ ) takes w ∈ B R (0) and δ > 0 as input and returns a scalar γ > 0 and a vector s ∈ R n with one of the following possible outcomes:

Case I: γ ≤ 1 which implies that w ∈ C ;

Case II: γ > 1 which implies that w /γ ∈ C and ⟨ s , w -x ⟩ ≥ γ -1 -δ ∀ x ∈ C .

To sum up, the oracle SEP ( w ; δ ) has two possible outcomes: it either certifies that w is feasible, i.e., w ∈ C , or it produces a scaled version of w that is in C and gives an approximate separating hyperplane between w and the set C .

The full algorithm is shown in Algorithm 2. The key idea here is to introduce surrogate loss functions ˜ ℓ t ( w ) = ⟨ ˜ g t , w ⟩ on the larger set B R (0) for 0 ≤ t ≤ T -1 , where ˜ g t is the surrogate gradient to be defined later. On a high level, we will run online projected gradient descent with ˜ ℓ t ( w ) to update the auxiliary iterates { w t } t ≥ 0 (note that the projection on B R (0) is easy to compute), and then produce the actions { x t } t ≥ 0 for the original problem by calling the SEP ( w t ; δ ) oracle in Definition 3. The follow lemma shows that the immediate regret ˜ ℓ t ( w t ) -˜ ℓ t ( x ) can serve as an upper bound on ℓ t ( x t ) -ℓ t ( x ) for any x ∈ C .

Lemma 10. Let { x t } T -1 t =0 be the iterates generated by Algorithm 2. Then we have x t ∈ C for t = 0 , 1 , . . . , T -1 . Also, for any x ∈ C , we have

and

Subroutine 2 Online Learning Guided Hessian Approximation Update

Proof. By the definition of SEP in Definition 3, we can see that x t ∈ C for all t = 1 , . . . , T . We now show that both (66) and (68) hold. We distinguish two cases depending on the outcomes of SEP ( w t ; δ t ) .

If γ t ≤ 1 , then we have x t = w t and ˜ g t = g t . In this case, (66) and (68) trivially hold.

which leads to (66) after rearranging. Also, by the triangle inequality we obtain

which proves (68).

Finally, from the update rule of w t +1 , for any x ∈ C ⊂ B R (0) we have ⟨ w t -ρ ˜ g t -w t +1 , w t +1 -x ⟩ ≥ 0 , which further implies that

Combining (66) and (71) leads to (67).

C.4 Projection-free Hessian Approximation Update

Now we are ready to describe our Hessian approximation matrix update, which is an specific instantiation of the general projection-free online learning algorithm shown in Algorithm 2. In particular, we only need to specify the convex set C as well as the SEP oracle.

Note that we have Z = { B ∈ S d + : 0 ⪯ B ⪯ L 1 I } in our online learning problem in Section 3.2. Since the projection-free scheme in Subroutine 2 requires the set C to contain the origin, we consider the transform ˆ B ≜ 2 L 1 ( B -L 1 2 I ) and let C = ˆ Z ≜ { ˆ B ∈ S d : -I ⪯ ˆ B ⪯ I } = { ˆ B ∈ S d : ∥ ˆ B ∥ op ≤ 1 } . We note that 0 ∈ ˆ Z and ˆ Z ⊂ B √ d (0) = { W ∈ S d : ∥ W ∥ F ≤ √ d } . Moreover, we can see that the approximate separation oracle SEP ( W ; δ, q ) defined in Definition 2 corresponds to a stochastic version of the oracle in Definition 3. The full algorithm is described in Subroutine 2 and we defer the specific implementation details of SEP ( W ; δ, q ) to Section E.2.

D Proof of Theorem 1

Regarding the choices of the hyper-parameters, we consider Algorithm 1 with the line search scheme in Subroutine 1, where α 1 , α 2 ∈ (0 , 1) with α 1 + α 2 < 1 and β ∈ (0 , 1) , and with the Hessian approximation update in Subroutine 2, where ρ = 1 128 , q t = p / 2 . 5( t +1)log 2 ( t +1) for t ≥ 1 , and δ t = 1 / ( √ t +2ln( t +2)) for t ≥ 0 . In the following, we first provide a proof sketch of Theorem 1. The complete proofs of the lemmas shown below will be provided in the subsequent sections.

Proof Sketch. To begin with, throughout the proof, we assume that every call of the SEP oracle in Definition 2 is successful during the execution of Algorithm 1. Indeed, by using the union bound, we can bound the failure probability by ∑ T -1 t =1 q t ≤ p 2 . 5 ∑ ∞ t =2 1 t log 2 t ≤ p . In particular, we note that Subroutine 2 ensures that 0 ⪯ B k ⪯ L 1 I for any k ≥ 0 .

We first prove Part (a) of Theorem 1, which relies on the following lemma.

We combine Lemma 2 and Lemma 11 to derive

By further using (14) and the elementary inequality that √ a + b ≤ √ a + √ b , we obtain

Next, we divide the proof of Part (b) of Theorem 1 into the following steps.

Step 1: We first use regret analysis to control the cumulative loss ∑ T -1 t =0 ℓ t ( B t ) incurred by our online learning algorithm in Subroutine 2. In particular, we prove a dynamic regret bound, where we compare the cumulative loss of our algorithm against the one achieved by the sequence { H t } T -1 t =0 .

Lemma 12. We have

where H t ≜ ∇ 2 f ( y t ) .

Step 2: In light of Lemma 12, it suffices to upper bound the cumulative loss ∑ T -1 t =0 ℓ t ( H t ) and the path-length ∑ T -1 t =0 ∥ H t +1 -H t ∥ F in the following lemma. To achieve this, we use the stability properties of our algorithm in (21) and Lemma 8, which is most technical part of the proof.

Lemma 13. We have

where C 2 is defined in (49) and C 3 = (1+ α 1 ) 2 β 2 (1 -α 1 ) 2 (1 -σ 2 ) .

Step 3: Thus, we obtain an upper bound on ∑ T -1 t =0 ℓ t ( B t ) by combining Lemma 12 and Lemma 13. Finally, in the following lemma, we prove an upper bound on 1 A N by further using Lemma 2 and Proposition 1.

Lemma 14. We have

where we define log + ( x ) ≜ max { log( x ) , 0 } ,

and C i ( i = 6 , . . . , 10 ) are absolute constants given by

Therefore, Part (b) of Theorem 1 immediately follows from Proposition 1.

In the remaining of this section, we present the proofs for the above lemmas that we used to prove the results in Theorem 1.

D.1 Proof of Lemma 11

Recall that w k ≜ ∇ f (˜ x k +1 ) - ∇ f ( y k ) and s k ≜ ˜ x k +1 -y k for k ∈ B . We can write ∇ f (˜ x k +1 ) - ∇ f ( y k ) = ¯ H k (˜ x k +1 -y k ) by using the fundamental theorem of calculus, where ¯ H k = ∫ 1 0 ∇ 2 f ( t ˜ x k +1 + (1 -t ) y k ) dt . Since we have 0 ⪯ ∇ 2 f ( x ) ⪯ L 1 I for all x ∈ R d by Assumption 1, it implies that 0 ⪯ ¯ H k ⪯ L 1 I . Moreover, since 0 ⪯ B k ⪯ L 1 I , we further have -L 1 I ⪯ ¯ H k -B k ⪯ L 1 I , which yields ∥ ¯ H k -B k ∥ op ≤ L 1 . Thus, we have

which proves that ℓ k ( B k ) ≤ L 2 1 .

D.2 Proof of Lemma 12

To prove Lemma 12, we first present the following lemma showing a smooth property of the loss function ℓ k . The proof is similar to [35, Lemma 15].

Lemma 15. For k ∈ B , we have

Moreover, for any B ∈ S d , it holds that

where ∥ · ∥ F and ∥ · ∥ ∗ denote the Frobenius norm and the nuclear norm, respectively.

Proof. It is straightforward to verify the expression in (73). The first inequality in (74) follows from the fact that ∥ A ∥ F ≤ ∥ A ∥ ∗ for any matrix A ∈ S d . For the second inequality, note that

where in the first inequality we used the triangle inequality, and in the second inequality we used the fact that the rank-one matrix uv ⊤ has only one nonzero singular value ∥ u ∥∥ v ∥ .

We will also need the following helper lemma.

Lemma 16. If the real number x satisfies x ≤ A + B √ x , then we have x ≤ 2 A + B 2 .

Proof. From the assumption, we have

Before proving Lemma 12, we also present the following lemma that bounds the loss in each round. Lemma 17. For any H ∈ Z , we have

Proof. By letting x t = ˆ B t , x = ˆ H ≜ 2 L 1 ( H -L 1 2 I ) , g t = G t ≜ 2 L 1 ∇ ℓ t ( B t ) , ˜ g t = ˜ G t , w t = W t in Lemma 10, we obtain:

(i) ˆ B t ∈ ˆ Z , which means that ∥ ˆ B t ∥ op ≤ 1 .

(ii) It holds that

First, note that ∥ S t ∥ F ≤ 3 by Definition 2 and |⟨ G t , ˆ B t ⟩| ≤ ∥ G t ∥ ∗ ∥ ˆ B t ∥ op ≤ ∥ G t ∥ ∗ . Together with (76), we get

where we used the fact that G t = 2 L 1 ∇ ℓ t ( B t ) and Lemma 15 in the last inequality. Furthermore, since ℓ t is convex, we have

where we used G t = 2 L 1 ∇ ℓ t ( B t ) , ˆ B t ≜ 2 L 1 ( B t -L 1 2 I ) , and ˆ H ≜ 2 L 1 ( H -L 1 2 I ) . Therefore, by combining (75) and (77) we get

Note that ℓ t ( B t ) appears on both sides of (79). By further applying Lemma 16, we obtain

Since ρ = 1 / 128 , by rearranging and simplifying terms in the above inequality, we obtain

Hence, we obtain

Proof of Lemma 12. We let H t = ∇ 2 f ( y t ) for t = 0 , 1 , . . . , T -1 . Thus, we get

Furthermore, note that

where in the last inequality we used the fact that ˆ H t , ˆ H t +1 , W t +1 ∈ B √ d (0) and the triangle inequality. Therefore, we get

By summing the above inequality from t = 0 to T -1 , we get

Finally, we use the fact that W 0 ≜ 2 L 1 ( B 0 -L 1 2 I ) , and ˆ H 0 ≜ 2 L 1 ( H 0 -L 1 2 I ) to obtain Lemma 12.

D.3 Proof of Lemma 13

By Assumption 2, we have ∥ w t -H t s t ∥ = ∥∇ f (˜ x t +1 ) - ∇ f ( y t ) - ∇ f ( y t )(˜ x t +1 -y t ) ∥ ≤ L 2 2 ∥ ˜ x t +1 -y t ∥ 2 . Thus,

where we used Lemma 1 in the last inequality. Also, Since a k ≥ η k for all k ≥ 0 , by (21) we get

Hence, we have

which proves the first inequality in (72).

Furthermore, by Assumption 2, we have

Hence, by using the triangle inequality, we can bound

where we used Lemma 8 in the last inequality.

D.4 Proof of Lemma 14

Before presenting the proof of Lemma 14, we start with a helper lemma that shows a lower bound on A 1 .

Proof. The equality A 1 = ˆ η 0 is shown in the proof of Lemma 3. To show the lower bound on ˆ η 0 , we use Lemma 1 and separate two cases. If 0 / ∈ B , then we have ˆ η 0 = η 0 = σ 0 . Otherwise, if 0 ∈ B , then we have

Moreover, as shown in the proof of Lemma 11, we have ∥∇ f (˜ x 1 ) -∇ f ( y 0 ) -B k (˜ x 1 -y 0 ) ∥ ≤ L 1 ∥ ˜ x 1 -y 0 ∥ , which further implies that ˆ η 0 ≥ α 2 β L 1 . This completes the proof.

We combine Lemma 12 and Lemma 13 to get

Since δ t = 1 / ( √ t +2ln( t +2)) , we have

Hence, it further follows from (14) and Lemma 2 that

To simplify the notation, define

and the inequality in (80) becomes N 5 A 2 N ≤ M + C 10 L 1 L 2 d ∥ z 0 -x ∗ ∥ log A N A 1 . Let A ∗ N be the number that achieves the equality

and we can see that A N ≥ A ∗ N . Thus, we instead try to construct a lower bound on A ∗ N . If A ∗ N ≤ A 1 , then log( A ∗ N /A 1 ) ≤ 0 and furthermore

Otherwise, assume that A ∗ N > A 1 . Then log( A ∗ N /A 1 ) > 0 and we first show an upper bound on A ∗ N :

This in turn leads to a lower bound on A ∗ N :

where we also used the fact that A 1 ≥ min { σ 0 , α 2 β L 1 } (cf. Lemma 18). Thus, we get

Combining both cases in (81) and (82), we conclude the proof of Lemma 14.

E Characterizing the Computational Cost

In this section, we first specify the implementation details of the LinearSolver oracle in Definition 1 and the SEP oracle in Definition 2. Then in Section E.3, we present the proof of Theorem 2.

E.1 Implementation of the LinearSolver Oracle

We implement the LinearSolver oracle by running the conjugate residual (CR) method [48] to solve the linear system As = b . In particular, we initialize the CR method with s 0 = 0 and returns the iterate s k once we achieve ∥ As k -b ∥ ≤ α ∥ s k ∥ . The following lemma provides the convergence guarantee of the CR method, which will be later used in the proof of Theorem 2.

Lemma 19 ([56, Chapter 12.4]) . Let s ∗ be any optimal solution of As ∗ = b and let { s k } be the iterates generated by Subroutine 3. Then we have

E.2 Implementation of SEP Oracle

We implement the SEP oracle in Definition 2 based on the classical Lanczos method with a random start, where the initial vector is chosen randomly and uniformly from the unit sphere (see, e.g., [57, 58]). For completeness, the full algorithm is described in Subroutine 4.

To prove the correctness of our algorithm, we first recall a classical result in [51] on the convergence behavior of the Lanczos method.

Proposition 3 ([51, Theorem 4.2]) . Consider a symmetric matrix W and let λ 1 ( W ) and λ d ( W ) denote its largest and smallest eigenvalues, respectively. Then after k iterations of the Lanczos method with a random start, we find unit vectors u (1) and u ( d ) such that

As a corollary, to ensure that, with probability at least 1 -q ,

the number of iterations can be bounded by ⌈ 1 4 ϵ -1 / 2 log(11 d/q 2 ) + 1 2 ⌉ .

Lemma 20. Let γ and S be the output of SEP ( W ; δ, q ) in Subroutine 4. Then with probability at least 1 -q , they satisfy one of the following properties:

Case I: γ ≤ 1 , then we have ∥ W ∥ op ≤ 1 ;

Case II: γ > 1 , then we have ∥ W /γ ∥ op ≤ 1 , ∥ S ∥ F = 3 and ⟨ S , W -ˆ B ⟩ ≥ γ -1 for any ˆ B such that ∥ ˆ B ∥ op ≤ 1 .

Proof. Note that in Subroutine 4, we first run the Lanczos method for ⌈ ϵ -1 / 2 log 11 d q 2 + 1 2 ⌉ iterations, where ϵ = 1 4 . Thus, by Proposition 3, with probability at least 1 -q/ 2 we have

Combining (83) and (84), we get

By plugging the above inequality back into (83) and (84), we further have

Let ˆ λ max = max { ˆ λ 1 , -ˆ λ d } . By (85) and (86), we can further bound the eigenvalues of W by

Hence, we can see that ∥ W ∥ op = max { λ 1 ( W ) , -λ d ( W ) } ≤ 2 ˆ λ max . Now we distinguish three cases.

(a) If ˆ λ max ≤ 1 2 , then we are in Case I and the ExtEvec oracle outputs γ = 2 ˆ λ max ≤ 1 and S = 0 . In this case, we indeed have ∥ W ∥ op ≤ γ ≤ 1 .

(b) If ˆ λ max ≥ 2 , then we are in Case II . In addition, if ˆ λ 1 ≥ -ˆ λ d , then the ExtEvec oracle returns γ = 2 ˆ λ max and S = 3 u (1) ( u (1) ) ⊤ . Similarly, if -ˆ λ d > ˆ λ 1 , then the ExtEvec oracle returns γ = 2 ˆ λ max and S = -3 u ( d ) ( u ( d ) ) ⊤ . Without loss of generality, consider the case where ˆ λ 1 ≥ -ˆ λ d . Since ∥ W ∥ op ≤ 2 ˆ λ max = γ , we have ∥ W /γ ∥ op ≤ 1 . Also, since u 1 is a unit vector, we have ∥ S ∥ F = 3 ∥ u (1) ∥ 2 = 3 . Finally, for any ˆ B such that ∥ ˆ B ∥ op ≤ 1 , we have

⟨ S , W -ˆ B ⟩ = 3( u (1) ) ⊤ Wu (1) -3( u (1) ) ⊤ ˆ Bu (1) ≥ 3 ˆ λ max -3 ≥ 2 ˆ λ max -1 = γ -1 , where we used the fact that ˆ λ max ≥ 2 in the last inequality.

(c) If 1 2 < ˆ λ max < 2 , we continue to run the Lanczos method for a total number of ⌈ 1 4 ϵ -1 / 2 log 11 d q 2 + 1 2 ⌉ iterations, where ϵ = 1 8 δ . Thus, by Proposition 3, with probability at least 1 -q/ 2 we have

Let ˜ λ max = max { ˜ λ 1 , -˜ λ d } . Since we have λ 1 ( W ) ≤ 2 ˆ λ max ≤ 4 and λ d ( W ) ≥ -2 ˆ λ max ≥ -4 , the above implies that ˜ λ 1 ≥ λ 1 ( W ) -δ and ˜ λ d ≤ λ d ( W ) + δ . Hence, we can see that ∥ W ∥ op = max { λ 1 ( W ) , -λ d ( W ) } ≤ ˆ λ max + δ . We further consider two subcases.

(c1) If ˜ λ max ≤ 1 -δ , then we are in Case I and the ExtEvec oracle outputs γ = ˜ λ max + δ and S = 0 . In this case, we indeed have ∥ W ∥ op ≤ γ ≤ 1 .

(c2) If ˜ λ max > 1 -δ , then we are in Case II . In addition, if ˜ λ 1 ≥ -˜ λ d , then the ExtEvec oracle returns γ = ˜ λ max + δ and S = ˜ u (1) (˜ u (1) ) ⊤ . Similarly, if -˜ λ d > ˜ λ 1 , then the ExtEvec oracle returns γ = ˜ λ max + δ and S = -˜ u ( d ) (˜ u ( d ) ) ⊤ . Without loss of generality, consider the case where ˜ λ 1 ≥ -˜ λ d . Since ∥ W ∥ op ≤ ˜ λ max + δ = γ , we have ∥ W /γ ∥ op ≤ 1 . Also, since ˜ u (1) is a unit vector, we have ∥ S ∥ F = ∥ ˜ u (1) ∥ 2 = 1 . Finally, for any ˆ B such that ∥ ˆ B ∥ op ≤ 1 , we have

This completes the proof.

E.3 Proof of Theorem 2

We divide the proof of Theorem 2 into the following three lemmas.

Lemma 21. If we run Algorithm 1 as specified in Theorem 1 for N iterations, then the total number of line search steps can be bounded by 2 N +log 1 /β ( σ 0 L 1 /α 2 ) . As a corollary, the total number of gradient queries is bounded by 3 N ϵ +log 1 /β ( σ 0 L 1 α 2 ) .

Proof. In our backtracking scheme, the number of steps in each iteration is given by log 1 /β ( η k / ˆ η k ) + 1 . Also note that η k +1 ≤ ˆ η k /β for all k ≥ 0 . Thus, we have

Furthermore, since ˆ η k ≥ α 2 β/L 1 for all k ≥ 0 , we arrive at the conclusion.

Lemma 22. The total number of matrix-vector product evaluations in the LinearSolver oracle is bounded by N ϵ + C 11 √ σ 0 L 1 + C 12 √ L 1 ∥ z 0 -x ∗ ∥ 2 2 ϵ , where C 11 and C 12 are absolute constants.

Proof. Our proof loosely follows the strategy in [32]. We first bound the number of steps required by Subroutine 3 before it terminates.

Lemma 23. Suppose A ⪰ I . Then Subroutine 3 terminates after at most ⌈√ α +1 α λ max ( A ) -1 ⌉ iterations.

Proof. Note that ∥ s k ∥ 2 ≥ ∥ s ∗ ∥ 2 - ∥ s k -s ∗ ∥ 2 . Also, since A ⪰ I , we have ∥ s k -s ∗ ∥ 2 ≤ ∥ A ( s k -s ∗ ) ∥ 2 = ∥ r k ∥ 2 . Therefore, we have

By using Lemma 19, we only need k ≥ √ α +1 α λ max ( A ) -1 to achieve ∥ As k -b ∥ ≤ α ∥ s k ∥ .

Moreover, when the step size is smaller enough, we can show that Subroutine 3 will terminate in one iteration.

Lemma 24. Let A = I + η B . When η ≤ α 2 L 1 , Algorithm 3 terminates in one iteration.

Proof. From the update rule of Subroutine 3, we can compute that s 1 = b ⊤ Ab ∥ Ab ∥ 2 2 b , which implies

On the other hand, we also have

Moreover, when η ≤ α 2 L 1 , we have ηL 1 ≤ α 1+ ηL 1 , which implies that ∥ r 1 ∥ ≤ α ∥ s 1 ∥ .

Now we upper bound the total number of matrix-vector products in Algorithm 1. Note that at the k -th iteration, we use the LinearSolver oracle with A = I + η + B k where η + = η k β i . We can store the vector B k b at the beginning and reuse it to compute s 1 when the step size η + < α 1 2 L 1 . And when β i η k L 1 ≥ α 1 2 , it holds that

Thus, at the k -th iteration, the number of matrix-vector products can be bounded by

Furthermore, we can bound that

Note that ϵ < f ( x N -1 ) -f ( x ∗ ) ≤ ∥ z 0 -x ∗ ∥ 2 2 A N -1 . Hence, we have A N -1 ≤ ∥ z 0 -x ∗ ∥ 2 2 ϵ . Thus, we can bound the total number of matrix-vector product evaluations by

Lemma 25. The total number of matrix-vector product evaluations in the SEP oracle is bounded by O ( N 1 . 25 ϵ (log N ϵ ) 0 . 5 log ( √ dN ϵ p )) .

Proof. Note that we have N t ≤ ⌈ 1 4 √ 2 δ t log 44 d q 2 t + 1 2 ⌉ in Subroutine 4, where δ t = 1 / ( √ t +2log( t + 2)) and q t = p/ (2 . 5( t +1)log 2 ( t +1)) . Thus, we have

F Experiments

In our experiments, we consider the logistic regression problem and minimizing the log-sum-exp function. Below we provide more details about the data generation scheme as well as the implementation of Nesterov's accelerated gradient method, BFGS, and our proposed A-QPNE algorithm.

Dataset generation. In the first experiment of logistic regression, the dataset consists of n data points { ( a i , y i ) } n i =1 , where a i ∈ R d is the i -th feature vector and y i ∈ {-1 , 1 } is its corresponding label. The labels { y i } n i =1 are generated by

where a ∗ i ∈ R d -1 and x ∗ ∈ R d -1 are the underlying true feature vector and the underlying true parameter, respectively. Moreover, each entry of a ∗ i and x ∗ is drawn independently according to the standard normal distribution N (0 , 1) . Note that the true feature vectors { a ∗ i } n i =1 are not given in our dataset; instead, we generate { a i } n i =1 by adding noises and appending an extra dimension to { a ∗ i } n i =1 . Specifically, we let a i = [ a ∗ i + n i + 1 ; 1] ⊤ ∈ R d , where n i ∼ N (0 , σ 2 I ) is the i.i.d. Gaussian noise vector and 1 ∈ R d -1 denotes the all-one vector. In our experiment, we set n = 2 , 000 , d = 150 and σ = 0 . 8 .

In the second experiment of log-sum-exp function, we follow a similar procedure as [16] to generate the dataset { ( a i , b i ) } n i =1 , where a i ∈ R d and b i ∈ R . First, we generate the auxiliary random vectors { ˆ a i } n i =1 by sampling each entry of ˆ a i uniformly and independently from the interval [ -1 , 1] . Moreover, we generate { b i } n i =1 independently from the standard normal distribution N (0 , 1) . Given { (ˆ a i , b i ) } n i =1 , we define an auxiliary function ˆ f ( x ) = log( ∑ n i =1 e ⟨ ˆ a i , x ⟩-b i ) and finally let a i = ˆ a i -∇ ˆ f (0) for i = 1 , . . . , n . As discussed in [16], the purpose of this procedure is to ensure that ∇ f (0) = 0 and thus 0 is the unique minimizer of f . In our experiment, we set n = d = 250 .

NAG. We implemented a monotone variant of the Nesterov accelerated gradient method as described in [54, Section 10.7.4]. Moreover, we determine the step size using a backtracking line search scheme.

BFGS. We implemented the classical BFGS algorithm, where the step size is determined by the Moré-Thuente line search scheme using an implementation by Diane O'Leary 1 .

0

0.5

1

1.5

10 -15

10 -10

10 -5

10 0

10 5

(a) Logistic regression

0

5

10

15

20

10 -15

10 -10

10 -5

10 0

10 5

(b) Log-sum-exp

A-QPNE (our method). We implemented our proposed A-QPNE method following the pseudocode in Algorithm 1, where the line search scheme is given in Subroutine 1 and the Hessian approximation update is given in Subroutine 2. Moreover, the implementations of the LinearSolver oracle and the SEP oracle are given by Subroutines 3 and 4, respectively.

F.1 Additional Plots

In Fig. 3, we compare the performance of our proposed A-QNPE method with NAG and BFGS in terms of the running time. All experiments are conducted using MATLAB R2021b on a MacBook Pro with an Apple M1 chip and 16GB RAM. We observe from Fig. 3(a) that our method requires less running time than NAG due to its faster convergence, especially when we are seeking a solution of high accuracy. On the other hand, there are cases where our method is slower than NAG in terms of the running time, as shown in Fig. 3(b). This is because, in this case, the cost of gradient computation is comparable to the cost of matrix-vector product evaluation, and therefore our method incurs a higher computational cost per iteration than NAG.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a first-order optimization method for convex optimization that requires gradient computations and matrix-vector products. Based on recent advances on quasi-newton methods, this method converges at a rate that matches the optimal rate ($1/k^2$) when $k = \\Omega(d)$ and improves upon the optimal rate when $k >>d$. In order to achieve this rate the authors consider a quasi-Newton method with a backtracking line search and a projection-free online algorithm to approximate the hessian of the objective.\n\nSTRENGTHS:\nThis paper has several strengths:\n- the rate proposed improved upon the best-known rates. \nIn particular, it is faster than the convergence rate of NAG when the number of iterations is larger than the dimension. \n- The authors really try to give the main steps of the results in the main paper. \n- While the paper is quite technical, I found that the contributions are well-presented,\n\nWEAKNESSES:\nI found the paper could be slightly improved:\n- The experiments could support the theory more:\n  - The logistic loss is strictly convex. Thus it seems that the rate in the experiments should be super-linear. It is not clear in the experiments if we are observing these super-linear rates or if we see sublinear convergence. I would instead try the log-sum exp loss (logistic regression for multi-class), which is not strictly convex \n  - I would be curious to see if, actually, one can find a practical loss for which one can observe the sublinear rate $1/k^{2.5}$.\n  - A plot with time as a x-axis could illustrate the fact that in many situations, matrix-vector product computations are not the bottleneck.\n- Some details on the Online method are missing in the main paper:\n   - The subroutine could be described (at least intuitively) \n   - The main result about the regret bound could be stated (basically logarithmic regret) \n   - In theorem 1 the probability p does not appear in the bound. I understand it only appears in the matrix-vector product complexity but it makes the statement quite odd. I would suggest the authors to at least mention it in a remark.\n\nQUESTIONS:\nCan you develop why you cannot use standard projection-free no-regret methods? such as\n\n\nNew Projection-free Algorithms for Online Convex Optimization with Adaptive Regret Guarantees \nDan Garber and Ben Kretzu\n\n\nI assume it would not be straightforward since your notion of regret is different (dynamic regret) but I would be curious to know how your method compares with respect to this related work.\n\nLIMITATIONS:\nThey authors mostly do address the limitation of their work. \n\nThey mention that their algorithm requires many matrix-vector products and, thus, their rate is faster than NAG if the gradient computation is considered the bottleneck. \n\nThey mention that their algorithm is slower than BFGS in practice.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposed a novel quasi-Newton method with faster global convergence rate. The algorithm uses the framework of MS acceleration and updates the Hessian estimator via online learning. The obtained convergence rate is impressive, it firstly show a faster global rate of quasi-Newton method which cannot be achieved by the first order methods.

STRENGTHS:
The proposed algorithm is novel. The global convergence rate of $\tilde{\mathcal{O}}(\sqrt{d}/k^{-2.5})$ is a significant theoretical result.

WEAKNESSES:
1. Although the iteration complexity is impressive and significant, the total computation complexity of the proposed method is not very satisfactory. Compared with the first-order method, it requires additional Hessian-vector products with the complexity of $\min\lbrace d^{0.25}/\epsilon^{0.5},1/\epsilon^{0.625}\rbrace$ which may lead to an even higher computation cost than NAG.

2. The proposed method is complicated and may not practical to use. For the experimental part, the authors do not present the comparison between their method and the baselines in term of running time.

QUESTIONS:
1. Can you compare the detailed computation cost in a table to make the results more clear?
2. Can you conduct some experimental results in terms of the running time?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an accelerated quasi-Newton proximal extragradient method for solving unconstrained smooth convex optimization problems. The algorithm  can achieve a convergence rate of $\\mathcal{O}\\bigl(\\min\\{\\frac{1}{k^2}, \\frac{\\sqrt{d\\log k}}{k^{2.5}}\\}\\bigr)$, where $d$ is the problem dimension and $k$ is the number of iterations. In particular, in the regime where $k = \\mathcal{O}(d)$, our method matches the optimal rate of $\\mathcal{O}(\\frac{1}{k^2})$ by Nesterov's accelerated gradient (NAG). Moreover, in the the regime where $k = \\Omega(d \\log d)$, it outperforms NAG and converges at a faster rate of $\\mathcal{O}\\bigl(\\frac{\\sqrt{d\\log k}}{k^{2.5}}\\bigr)$.\n\nSTRENGTHS:\nThis paper proposes an accelerated quasi-Newton proximal extragradient method for solving unconstrained smooth convex optimization problems. The algorithm  can achieve a convergence rate of $\\mathcal{O}\\bigl(\\min\\{\\frac{1}{k^2}, \\frac{\\sqrt{d\\log k}}{k^{2.5}}\\}\\bigr)$, where $d$ is the problem dimension and $k$ is the number of iterations. In particular, in the regime where $k = \\mathcal{O}(d)$, our method matches the optimal rate of $\\mathcal{O}(\\frac{1}{k^2})$ by Nesterov's accelerated gradient (NAG). Moreover, in the the regime where $k = \\Omega(d \\log d)$, it outperforms NAG and converges at a faster rate of $\\mathcal{O}\\bigl(\\frac{\\sqrt{d\\log k}}{k^{2.5}}\\bigr)$.\n\nWEAKNESSES:\nI have a concern about the total computation cost. \nLet us consider the finite-sum form, that is, $f(x) = \\frac{1}{n} \\sum_{i=1}^nf_i(x)$ with $n = \\mathcal{O}(d)$.\nThen, the algorithm takes $N_\\eps$ gradient queries which implies $\\mathcal{O}(N_\\eps d^2)$ computation cost.\nBy Theorem 2 (c), the cost of computing SEP oracles, the computation cost is $ \\mathcal{O}(N_\\eps^{1.25} d^2) $. \nIn this case, A-QNPE does not have advantages over  Nesterov's accelerated gradient (NAG).\n\nQUESTIONS:\nNo\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses the optimal and adaptive Monteiro Svaiter acceleration framework to create a quasi-Newton method that solves unconstrainted convex problems with Lipschitz gradients and Lipschitz hessians at Nesterov's accelerated rate $O(1/k^2)$ but when the number of iterations is greater enough than the dimension, namely $\\Omega(d\\log d)$ it provides better convergence guarantees in terms of gradient oracle complexity. The number of operations is superlinear $\\widetilde{O}(N_\\epsilon^{1.25})$ in the number of gradient queries $N_\\epsilon$ and memory that is quadratic in the dimension is used.\n\nSTRENGTHS:\nAuthors use a great range of technical and powerful tools like optimal and adaptive MS acceleration, they use projection free online learning with a separation oracle, the conjugate gradients method and Lanczos algorithms.  The results are novel and interesting.\n\nWEAKNESSES:\nThe abstract (and some parts of the paper) says that the method matches the optimal rate O(1/k^2) by NAG, and this rate is known optimal for functions that are convex smooth. Here your setting is convex smooth + Lipchitz Hessians. It would be good to explicitly comment on how that construction has a hessian Lipschitzness L_2 = 0 (since it is a quadratic) and therefore it also applies to your setting.\n\nI guess you can modify your algorithm to get results under the additional assumption of \\mu strong convexity. This could take a lot of work, but on the other hand you can follow the spirit of the usual reductions to show rates for your algorithm under strong convexity by using a sequence of restarts. So I would suggest to add a remark with this. The argument would be like this, you run the algorithm in stages and after each stage you guarantee you halve the distance to x^ast and so you only need to run a logarithmic number of stages. In order to do that, you want to run the algorithm for k iterations such that you guarantee the last equality here:\n$$\n\\mu/2 \\|x_t-x^\\ast\\|^2 \\leq f(x_t) - f(x^\\ast) \\leq O(\\text{your bound}) = \\mu/8 \\|x_0-x^\\ast\\|^2\n$$\nThe first inequality is strong convexity and the second one is your guarantee. In the regime in which \\mu is small enough, the number of iterations necessary in each stage should be \\Omega(d\\log(d)) and so your improved bound kicks in.\nAfter doing this, it would be desirable to have a comparison of these results with the quasi-Newton results mentioned in the introduction that apply to strongly convex functions.\n\n\nSimilarly, regarding \"However, all of the results above only apply under the restrictive assumption that the objective function f is strictly or strongly convex. In the more general setting where f is merely convex...\". Given the reduction that solves convex smooth  problems by regularizing with + \\epsilon/ R^2 \\norm{x_0-x^\\ast}^2, where R is an upper bound on the initial distance to the minimizer, you should discuss what one can get with previous methods under the reduction. That is, the strongly convex setting is not necessarily restrictive and a comparison / discussion should be done.\n\nL839 \"subourtine\" -> \"subroutine\"\n\nQUESTIONS:\n.\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}