SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ruichen Jiang
ECE Department
rjiang@utexas.edu
Aryan Mokhtari
mokhtari@austin.utexas.edu
In this paper, we present an accelerated quasi-Newton proximal extragradient method for solving unconstrained smooth convex optimization problems. With access only to the gradients of the objective function, we prove that our method can achieve a convergence rate of  $\mathcal{O}\left(\min \left\{\frac{1}{k^2}, \frac{\sqrt{d\log k}}{k^{2.5}}\right\}\right)$ , where  $d$  is the problem dimension and  $k$  is the number of iterations. In particular, in the regime where  $k = \mathcal{O}(d)$ , our method matches the optimal rate of  $\mathcal{O}\left(\frac{1}{k^2}\right)$  by Nesterov's accelerated gradient (NAG). Moreover, in the regime where  $k = \Omega(d\log d)$ , it outperforms NAG and converges at a faster rate of  $\mathcal{O}\left(\frac{\sqrt{d\log k}}{k^{2.5}}\right)$ . To the best of our knowledge, this result is the first to demonstrate a provable gain for a quasi-Newton-type method over NAG in the convex setting. To achieve such results, we build our method on a recent variant of the Monteiro-Svaiter acceleration framework and adopt an online learning perspective to update the Hessian approximation matrices, in which we relate the convergence rate of our method to the dynamic regret of a specific online convex optimization problem in the space of matrices.
In this paper, we consider the following unconstrained convex minimization problem
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {d}} f (\mathbf {x}), \tag {1}
$$
where the objective function  $f: \mathbb{R}^d \to \mathbb{R}$  is convex and differentiable. We are particularly interested in quasi-Newton methods, which are among the most popular iterative methods for solving the problem in (1) [1-8]. Like gradient descent and other first-order methods, quasi-Newton methods require only the objective's gradients to update the iterates. On the other hand, they can better exploit the local curvature of  $f$  by constructing a Hessian approximation matrix and using it as a preconditioner, leading to superior convergence performance. In particular, when the objective function in (1) is strictly convex or strongly convex, it has long been proved that quasi-Newton methods achieve an asymptotic superlinear convergence rate [7-15], which significantly improves the linear convergence rate obtained by first-order methods. More recently, there has been progress on establishing a local non-asymptotic superlinear rate of the form  $\mathcal{O}((1 / \sqrt{k})^k)$  for classical quasi-Newton methods and their variants [16-22].
However, all of the results above only apply under the restrictive assumption that the objective function  $f$  is strictly or strongly convex. In the more general setting where  $f$  is merely convex, to the best of our knowledge, there is no result that demonstrates any form of convergence improvement by quasi-Newton methods over first-order methods. More precisely, it is well known that Nesterov's accelerated gradient (NAG) [23] can achieve a convergence rate of  $\mathcal{O}(1 / k^2)$  if  $f$  is convex and has Lipschitz gradients. On the other hand, under the same setting, asymptotic convergence of
classical quasi-Newton methods has been shown in [13, 24] but no explicit rate is given. With certain conditions on the Hessian approximation matrices, the works in [25-27] presented quasi-Newton-type methods with convergence rates of  $\mathcal{O}(1 / k)$  and  $\mathcal{O}(1 / k^2)$ , respectively, which are no better than the rate of NAG and, in fact, can be even worse in terms of constants. This gap raises the following fundamental question:
Can we design a quasi-Newton-type method that achieves a convergence rate faster than  $\mathcal{O}(1 / k^2)$  for the smooth convex minimization problem in (1)?
At first glance, this may seem impossible, as for any first-order method that has access only to a gradient oracle, one can construct a "worst-case" instance and establish a lower bound of  $\Omega(1/k^2)$  on the optimality gap [28, 29]. It is worth noting that while such a lower bound is typically shown under a "linear span" assumption, i.e., the methods only query points in the span of the gradients they observe, this assumption is in fact not necessary and can be removed by the technique of resisting oracle (see, [30, Section 3.3]). In particular, this  $\Omega(1/k^2)$  lower bound applies for any iterative method that only queries gradients of the objective, including quasi-Newton methods. On the other hand, this lower bound is subject to a crucial assumption: it only works in the high-dimensional regime where the problem dimension  $d$  exceeds the number of iterations  $k$ . As such, it does not rule out the possibility of a faster rate than  $\mathcal{O}(1/k^2)$  when the number of iterations  $k$  is larger than  $d$ . Hence, ideally, we are looking for a method that attains the optimal rate of  $\mathcal{O}(1/k^2)$  in the regime that  $k = \mathcal{O}(d)$  and surpasses this rate in the regime that  $k = \Omega(d)$ .
Contributions. In this paper, we achieve the above goal by presenting an accelerated quasi-Newton proximal extragradient (A-QNPE) method. Specifically, under the assumptions that  $f$  in (1) is convex and its gradient and Hessian are Lipschitz, we prove the following guarantees:
- From any initialization, A-QNPE can attain a global convergence rate of  $\mathcal{O}\big(\min \{\frac{1}{k^2},\frac{\sqrt{d\log k}}{k^{2.5}}\} \big)$ . In particular, this implies that our method matches the optimal rate of  $\mathcal{O}\big(\frac{1}{k^2}\big)$  when  $k = \mathcal{O}(d)$ , while it converges at a faster rate of  $\mathcal{O}\big(\frac{\sqrt{d\log k}}{k^{2.5}}\big)$  when  $k = \Omega (d\log d)$ . Alternatively, we can bound the number of iterations required to achieve an  $\epsilon$ -accurate solution by  $N_{\epsilon} = \mathcal{O}\big(\min \{\frac{1}{\epsilon^{0.5}},\frac{d^{0.2}}{\epsilon^{0.4}} (\log \frac{d}{\epsilon^2})^{0.2}\}\big)$ .
- In terms of computational cost, we show that the total number of gradient queries after  $N$  iterations can be bounded by  $3N$ , i.e., on average no more than 3 per iteration. Moreover, the number of matrix-vector products to achieve an  $\epsilon$ -accurate solution can be bounded by  $\tilde{\mathcal{O}}\left(\min \left\{\frac{d^{0.25}}{\epsilon^{0.5}}, \frac{1}{\epsilon^{0.625}}\right\}\right)$ .
Combining the two results above, we conclude that A-QNPE requires  $\mathcal{O}\left(\min \left\{\frac{1}{\epsilon^{0.5}}, \frac{d^{0.2}}{\epsilon^{0.4}} (\log \frac{d}{\epsilon^2})^{0.2}\right\}\right)$  gradient queries to reach an  $\epsilon$ -accurate solution, which is at least as good as NAG and is further superior when  $\epsilon = \mathcal{O}\left(\frac{1}{d^2 \log^2(d)}\right)$ . To the best of our knowledge, this is the first result that demonstrates a provable advantage of a quasi-Newton-type method over NAG in terms of gradient oracle complexity in the smooth convex setting.
To obtain these results, we significantly deviate from the classical quasi-Newton methods such as BFGS and DFP. Specifically, instead of mimicking Newton's method as in the classical updates, our A-QNPE method is built upon the celebrated Monteiro-Svaiter (MS) acceleration framework [31, 32], which can be regarded as an inexact version of the accelerated proximal point method [33, 34]. Another major difference lies in the update rule of the Hessian approximation matrix. Classical quasi-Newton methods typically perform a low-rank update of the Hessian approximation matrix while enforcing the secant condition. On the contrary, our update rule is purely driven by our convergence analysis of the MS acceleration framework. In particular, inspired by [35], we assign certain loss functions to the Hessian approximation matrices and formulate the Hessian approximation matrix update as an online convex optimization problem in the space of matrices. Therefore, we propose to update the Hessian approximation matrices via an online learning algorithm.
Related work. The authors in [32] proposed a refined MS acceleration framework, which simplifies the line search subroutine in the original MS method [31]. By instantiating it with an adaptive second-order oracle, they presented an accelerated second-order method that achieves the optimal rate of  $\mathcal{O}\left(\frac{1}{k^{3.5}}\right)$ . The framework in [32] serves as a basis for our method, but we focus on the setting where we have access only to a gradient oracle and we consider a quasi-Newton-type update. Another closely related work is [35], where the authors proposed a quasi-Newton proximal extragradient method with a global non-asymptotic superlinear rate. In particular, our Hessian approximation update is inspired by the online learning framework in [35]. On the other hand, the major difference is that the authors in [35] focused on the case where  $f$  is strongly convex and presented a global
superlinear rate, while we consider the more general convex setting where  $f$  is only convex (may not be strongly convex). Moreover, we further incorporate the acceleration mechanism into our method, which greatly complicates the convergence analysis; see Remark 2 for more discussions.
Another class of optimization algorithms with better gradient oracle complexities than NAG are cutting plane methods [36-44], which are distinct from quasi-Newton methods we study in this paper. In particular, in the regime where  $\epsilon = \tilde{\mathcal{O}}\left(\frac{1}{d^2}\right)$ , they can achieve the optimal gradient oracle complexity of  $\mathcal{O}(d\log \frac{1}{\epsilon})$  [38]. On the other hand, in the regime where  $\epsilon = \Omega \left(\frac{1}{d^2}\right)$ , the complexity of the cutting plane methods is worse than NAG, while our proposed method matches the complexity of NAG.
Next, we formally state the required assumptions for our main results.
Assumption 1. The function  $f$  is twice differentiable, convex, and  $L_{1}$ -smooth. As a result, we have  $0 \preceq \nabla^{2}f(\mathbf{x}) \preceq L_{1}\mathbf{I}$  for any  $\mathbf{x} \in \mathbb{R}^d$ , where  $\mathbf{I} \in \mathbb{R}^{d\times d}$  is the identity matrix.
Assumption 2. The Hessian of  $f$  is  $L_{2}$ -Lipschitz, i.e., we have  $\| \nabla^{2}f(\mathbf{x}) - \nabla^{2}f(\mathbf{y})\|_{\mathrm{op}}\leq L_{2}\| \mathbf{x} - \mathbf{y}\|_{2}$  for any  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$ , where  $\| \mathbf{A}\|_{\mathrm{op}}\triangleq \sup_{\mathbf{x}:\| \mathbf{x}\| _2 = 1}\| \mathbf{A}\mathbf{x}\| _2$ .
We note that both assumptions are standard in the optimization literature and are satisfied by various loss functions such as the logistic loss and the log-sum-exp function (see, e.g., [16]).
Remark 1. We note that the additional assumption of Lipschitz Hessian does not alter the lower bound of  $\Omega(1/k^2)$  that we discussed in the introduction. Indeed, this lower bound is established by a worst-case quadratic function, whose Hessian is constant (Lipschitz continuous with  $L_2 = 0$ ). Therefore, Assumption 2 does not eliminate this worst-case construction from the considered problem class, and thus the lower bound also applies to our setting.
Monteiro-Svaiter acceleration. As our proposed method uses ideas from the celebrated Monteiro-Svaiter (MS) acceleration algorithm [31], we first briefly recap this method. MS acceleration, also known as accelerated hybrid proximal extragradien (A-HPE), consists of intertwining sequences of iterates  $\{\mathbf{x}_k\}$ ,  $\{\mathbf{y}_k\}$ ,  $\{\mathbf{z}_k\}$ , scalar variables  $\{a_k\}$  and  $\{A_k\}$  as well as step sizes  $\{\eta_k\}$ . The algorithm has three main steps. In the first step, we compute the auxiliary iterate  $\mathbf{y}_k$  according to
$$
\mathbf {y} _ {k} = \frac {A _ {k}}{A _ {k} + a _ {k}} \mathbf {x} _ {k} + \frac {a _ {k}}{A _ {k} + a _ {k}} \mathbf {z} _ {k}, \quad \text {w h e r e} \quad a _ {k} = \frac {\eta_ {k} + \sqrt {\eta_ {k} ^ {2} + 4 \eta_ {k} A _ {k}}}{2}. \tag {2}
$$
In the second step, an inexact proximal point step  $\mathbf{x}_{k + 1}\approx \mathbf{y}_k - \eta_k\nabla f(\mathbf{x}_{k + 1})$  is performed. To be precise, given a parameter  $\sigma \in [0,1)$ , we find  $\mathbf{x}_{k + 1}$  that satisfies
$$
\left\| \mathbf {x} _ {k + 1} - \mathbf {y} _ {k} + \eta_ {k} \nabla f \left(\mathbf {x} _ {k + 1}\right) \right\| \leq \sigma \| \mathbf {x} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {3}
$$
Then in the third step, the iterate  $\mathbf{z}$  is updated by following the update
$$
\mathbf {z} _ {k + 1} = \mathbf {z} _ {k} - a _ {k} \nabla f (\mathbf {x} _ {k + 1}).
$$
Finally, we update the scalar  $A_{k + 1}$  by  $A_{k + 1} = A_k + a_k$ . The above method has two implementation issues. First, to perform the update in (3) directly, one needs to solve the nonlinear system of equations  $\mathbf{x} - \mathbf{y}_k + \eta_k\nabla f(\mathbf{x}) = 0$  to a certain accuracy, which could be costly in general. To address this issue, a principled approach is to replace the gradient operator  $\nabla f(\mathbf{x})$  with a simpler approximation function  $P(\mathbf{x};\mathbf{y}_k)$  and select  $\mathbf{x}_{k + 1}$  as the (approximate) solution of the equation:
$$
\mathbf {x} _ {k + 1} - \mathbf {y} _ {k} + \eta_ {k} P \left(\mathbf {x} _ {k + 1}; \mathbf {y} _ {k}\right) = 0. \tag {4}
$$
For instance, we can use  $P(\mathbf{x}; \mathbf{y}_k) = \nabla f(\mathbf{y}_k)$  and accordingly (4) is equivalent to  $\mathbf{x}_{k+1} = \mathbf{y}_k - \eta_k \nabla f(\mathbf{y}_k)$ , leading to the accelerated first-order method in [31]. If we further have access to the Hessian oracle, we can use  $P(\mathbf{x}; \mathbf{y}_k) = \nabla f(\mathbf{y}_k) + \nabla^2 f(\mathbf{y}_k)(\mathbf{x} - \mathbf{y}_k)$  and (4) becomes  $\mathbf{x}_{k+1} = \mathbf{y}_k - \eta_k (\mathbf{I} + \eta_k \nabla^2 f(\mathbf{y}_k))^{-1} \nabla f(\mathbf{y}_k)$ , leading to the second-order method in [31].
However, approximating  $\nabla f(\mathbf{x})$  by  $P(\mathbf{x};\mathbf{y}_k)$  leads to a second issue related to finding a proper step size  $\eta_{k}$ . More precisely, one needs to first select  $\eta_{k}$ , compute  $\mathbf{y}_k$  from (2), and then solve the system in (4) exactly or approximately to obtain  $\mathbf{x}_{k + 1}$ . However, these three variables, i.e.,  $\mathbf{x}_{k + 1},\mathbf{y}_k$  and  $\eta_{k}$  may not satisfy the condition in (3) due to the gap between  $\nabla f(\mathbf{x})$  and  $P(\mathbf{x};\mathbf{y}_k)$ . If that happens, we need to re-select  $\eta_{k}$  and recalculate both  $\mathbf{y}_k$  and  $\mathbf{x}_{k + 1}$  until the condition in (3) is satisfied. To
address this issue, several bisection subroutines have been proposed in the literature [31, 45-47] and they all incur a computational cost of  $\log (1 / \epsilon)$  per iteration.
Optimal Monteiro-Svaiter acceleration. A recent paper [32] refines the MS acceleration algorithm by separating the update of  $\mathbf{y}_k$  from the line search subroutine. In particular, in the first stage, we use  $\eta_k$  to compute  $a_k$  and then  $\mathbf{y}_k$  from (2), which will stay fixed throughout the line search scheme. In the second stage, we aim to find a pair  $\hat{\mathbf{x}}_{k + 1}$  and  $\hat{\eta}_k$  such that they satisfy
$$
\left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \hat {\eta} _ {k} \nabla f \left(\hat {\mathbf {x}} _ {k + 1}\right) \right\| \leq \sigma \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {5}
$$
To find that pair, we follow a similar line search scheme as above, with the key difference that  $\mathbf{y}_k$  is fixed and  $\hat{\eta}_k$  can be different from  $\eta_k$  that is used to compute  $\mathbf{y}_k$ . More precisely, for a given  $\hat{\eta}_k$ , we find the solution of (4) denoted by  $\hat{\mathbf{x}}_{k + 1}$  and check whether it satisfies (5) or not. If it does not, then we adapt the step size and redo the process until (5) is satisfied. Then given the values of these two parameters  $\eta_k$  and  $\hat{\eta}_k$ , the updates for  $\mathbf{x}$  and  $\mathbf{z}$  would change as we describe next:
- If  $\hat{\eta}_k \geq \eta_k$ , we update  $\mathbf{x}_{k+1} = \hat{\mathbf{x}}_{k+1}$ ,  $A_{k+1} = A_k + a_k$  and  $\mathbf{z}_{k+1} = \mathbf{z}_k - a_k\nabla f(\hat{\mathbf{x}}_{k+1})$ . Moreover, we increase the next tentative step size by choosing  $\eta_{k+1} = \eta_k / \beta$  for some  $\beta \in (0,1)$ .
- Otherwise, if  $\hat{\eta}_k < \eta_k$ , the authors in [32] introduced a momentum damping mechanism. Define  $\gamma_k = \hat{\eta}_k / \eta_k < 1$ . We then choose  $\mathbf{x}_{k+1} = \frac{(1 - \gamma_k)A_k}{A_k + \gamma_ka_k}\mathbf{x}_k + \frac{\gamma_k(A_k + a_k)}{A_k + \gamma_ka_k}\hat{\mathbf{x}}_{k+1}$ , which is a convex combination of  $\mathbf{x}_k$  and  $\hat{\mathbf{x}}_{k+1}$ . Moreover, we update  $A_{k+1} = A_k + \gamma_ka_k$  and  $\mathbf{z}_{k+1} = \mathbf{z}_k - \gamma_ka_k\nabla f(\hat{\mathbf{x}}_{k+1})$ . Finally, we decrease the next tentative step size by choosing  $\eta_{k+1} = \beta \eta_k$ .
This approach not only simplifies the procedure by separating the update of  $\{\mathbf{y}_k\}$  from the line search scheme, but it also shaves a factor of  $\log (1 / \epsilon)$  from the computational cost of the algorithm, leading to optimal first and second-order variants of the MS acceleration method. Therefore, as we will discuss in the next section, we build our method upon this more refined MS acceleration framework.
3 Accelerated Quasi-Newton Proximal Extragradian
In this section, we present our accelerated quasi-Newton proximal extragradient (A-QNPE) method. An informal description of our method is provided in Algorithm 1. On a high level, our method can be viewed as the quasi-Newton counterpart of the adaptive Monteiro-Svaiter-Newton method proposed in [32]. In particular, we only query a gradient oracle and choose the approximation function in (4) as  $P(\mathbf{x};\mathbf{y}_k) = \nabla f(\mathbf{y}_k) + \mathbf{B}_k(\mathbf{x} - \mathbf{y}_k)$ , where  $\mathbf{B}_k$  is a Hessian approximation matrix obtained only using gradient information. Moreover, another central piece of our method is the update scheme of  $\mathbf{B}_k$ . Instead of following the classical quasi-Newton updates such as BFGS or DFP, we use an online learning framework, where we choose a sequence of matrices  $\mathbf{B}_k$  to achieve a small dynamic regret for an online learning problem defined by our analysis; more details will be provided later in Section 3.2. We initialize our method by choosing  $\mathbf{x}_0, \mathbf{z}_0 \in \mathbb{R}^d$  and setting  $A_0 = 0$  and  $\eta_0 = \sigma_0$  where  $\sigma_0$  is a user-specified parameter. Our method can be divided into the following four stages:
- In the first stage, we compute the scalar  $a_k$  and the auxiliary iterate  $\mathbf{y}_k$  according to (2) using the step size  $\eta_k$ . Note that  $\mathbf{y}_k$  is then fixed throughout the  $k$ -th iteration.
- In the second stage, given the Hessian approximation matrix  $\mathbf{B}_k$  and the iterate  $\mathbf{y}_k$ , we use a line search scheme to find the step size  $\hat{\eta}_k$  and the iterate  $\hat{\mathbf{x}}_{k + 1}$  such that
$$
\left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \hat {\eta} _ {k} \left(\nabla f (\mathbf {y} _ {k}) + \mathbf {B} _ {k} \left(\hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}\right)\right) \right\| \leq \alpha_ {1} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|, \tag {6}
$$
$$
\left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \hat {\eta} _ {k} \nabla f \left(\hat {\mathbf {x}} _ {k + 1}\right) \right\| \leq \left(\alpha_ {1} + \alpha_ {2}\right) \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\|, \tag {7}
$$
where  $\alpha_{1} \in [0,1)$  and  $\alpha_{2} \in (0,1)$  are user-specified parameters with  $\alpha_{1} + \alpha_{2} < 1$ . The first condition in (6) requires that  $\hat{\mathbf{x}}_{k+1}$  in exactly solves the linear system of equations  $(\mathbf{I} + \hat{\eta}_{k}\mathbf{B}_{k})(\mathbf{x} - \mathbf{y}_{k}) + \hat{\eta}_{k}\nabla f(\mathbf{y}_{k}) = 0$ , where  $\alpha_{1} \in [0,1)$  controls the accuracy. As a special case, we have  $\hat{\mathbf{x}}_{k+1} = \mathbf{y}_{k} - (\mathbf{I} + \hat{\eta}_{k}\mathbf{B}_{k})^{-1}\nabla f(\mathbf{y}_{k})$  when  $\alpha_{1} = 0$ . The second condition in (7) directly comes from (5) in the optimal MS acceleration framework, which ensures that we approximately follow the proximal point step  $\hat{\mathbf{x}}_{k+1} = \mathbf{y}_{k} - \hat{\eta}_{k}\nabla f(\hat{\mathbf{x}}_{k+1})$ . To find the pair  $(\hat{\eta}_{k},\hat{\mathbf{x}}_{k+1})$  satisfying both (6) and (7), we implement a backtracking line search scheme. Specifically, for some  $\beta \in (0,1)$ , we iteratively try  $\hat{\eta}_{k} = \eta_{k}\beta^{i}$  for  $i \geq 0$  and solve  $\hat{\mathbf{x}}_{k+1}$  from (6) until the condition in (7) is satisfied. The line search scheme will be discussed in more detail in Section 3.1.
Algorithm 1 Accelerated Quasi-Newton Proximal Extragradient Method
1: Input: initial points  $\mathbf{x}_0, \mathbf{z}_0 \in \mathbb{R}^d$ , initial step size  $\sigma_0 > 0$ ,  $\alpha_1, \alpha_2 \in (0,1)$  with  $\alpha_1 + \alpha_2 < 1$ ,  $\beta \in (0,1)$   
2: Initialize: set  $A_0 \gets 0$  and  $\eta_0 \gets \sigma_0$   
3: for iteration  $k = 0, \dots, N-1$  do  
4: Compute  $a_k \leftarrow \frac{\eta_k + \sqrt{\eta_k^2 + 4\eta_k A_k}}{2}$  and  $\mathbf{y}_k \leftarrow \frac{A_k}{A_k + a_k} \mathbf{x}_k + \frac{a_k}{A_k + a_k} \mathbf{z}_k$   
5: Let  $\hat{\eta}_k$  be the largest possible step size in  $\{\eta_k \beta^i : i \geq 0\}$  such that  $\hat{\mathbf{x}}_{k+1} \approx_{\alpha_1} \mathbf{y}_k - \hat{\eta}_k (\mathbf{I} + \hat{\eta}_k \mathbf{B}_k)^{-1} \nabla f(\mathbf{y}_k)$  and  $\| \hat{\mathbf{x}}_{k+1} - \mathbf{y}_k - \hat{\eta}_k \nabla f(\hat{\mathbf{x}}_{k+1}) \| \leq (\alpha_1 + \alpha_2) \| \hat{\mathbf{x}}_{k+1} - \mathbf{y}_k\|$   
6: if  $\hat{\eta}_k = \eta_k$  then  
7: Set  $\mathbf{x}_{k+1} \leftarrow \hat{\mathbf{x}}_{k+1}$ ,  $\mathbf{z}_{k+1} \leftarrow \mathbf{z}_k - a_k \nabla f(\hat{\mathbf{x}}_{k+1})$ ,  $A_{k+1} \leftarrow A_k + a_k$   
8: Set  $\eta_{k+1} \leftarrow \hat{\eta}_k / \beta$   
9: Set  $\mathbf{B}_{k+1} \leftarrow \mathbf{B}_k$   
10: else  
11: Let  $\gamma_k \leftarrow \hat{\eta}_k / \eta_k < 1$   
12: Set  $\mathbf{x}_{k+1} \leftarrow \frac{(1 - \gamma_k) A_k}{A_k + \gamma_k a_k} \mathbf{x}_k + \frac{\gamma_k (A_k + a_k)}{A_k + \gamma_k a_k} \hat{\mathbf{x}}_{k+1}$ ,  $\mathbf{z}_{k+1} \leftarrow \mathbf{z}_k - \gamma_k a_k \nabla f(\hat{\mathbf{x}}_{k+1})$ ,  $A_{k+1} \leftarrow A_k + \gamma_k a_k$   
13: Set  $\eta_{k+1} \leftarrow \hat{\eta}_k$   
14: Set  $\mathbf{w}_k \leftarrow \nabla f(\tilde{\mathbf{x}}_k) - \nabla f(\mathbf{y}_k)$  and  $\mathbf{s}_k \leftarrow \tilde{\mathbf{x}}_k - \mathbf{y}_k$ , where  $\tilde{\mathbf{x}}_k$  is the last rejected iterate in LS  
15: Feed  $\ell_k(\mathbf{B}) \triangleq \| \mathbf{w}_k - \mathbf{B}\mathbf{s}_k\|^2$  to an online learning algorithm and obtain  $\mathbf{B}_{k+1}$   
16: end if  
17: end for
- In the third stage, we update the variables  $\mathbf{x}_{k + 1},\mathbf{z}_{k + 1},A_{k + 1}$  and set the step size  $\eta_{k + 1}$  in the next iteration. Specifically, the update rule we follow depends on the outcome of the line search scheme. In the first case where  $\hat{\eta}_k = \eta_k$  , i.e., the line search scheme accepts the initial trial step size, we let
$$
\mathbf {x} _ {k + 1} = \hat {\mathbf {x}} _ {k + 1}, \quad \mathbf {z} _ {k + 1} = \mathbf {z} _ {k} - a _ {k} \nabla f (\hat {\mathbf {x}} _ {k + 1}), \quad A _ {k + 1} = A _ {k} + a _ {k}, \tag {8}
$$
as in the original MS acceleration framework. Moreover, this also suggests our choice of the step size  $\eta_{k}$  may be too conservative. Therefore, we increase the step size in the next iteration by  $\eta_{k + 1} = \hat{\eta}_k / \beta$ . In the second case where  $\hat{\eta}_k < \eta_k$ , i.e., the line search scheme backtracks, we adopt the momentum damping mechanism in [32]: