SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For easier comparison, we summarize the detailed computational costs of NAG and our method A-QNPE to achieve an  $\epsilon$ -accuracy in Table 1. We observe that A-QNPE outperforms NAG in terms of gradient query complexity: It makes equal or fewer gradient queries especially when  $\epsilon \ll \frac{1}{d^2}$ . On the other hand, A-QNPE requires additional matrix-vector product computations to implement the LinearSolver and SEP oracles. While this is a limitation of our method, in some cases, gradient evaluations are the main bottleneck and can be more expensive than matrix-vector products. As a concrete example, consider the finite-sum minimization problem  $f(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^{n}f_i(\mathbf{x})$ . In this case, one gradient query typically costs  $\mathcal{O}(nd)$ , while one matrix-vector product costs  $\mathcal{O}(d^2)$ . Thus, the total computational cost of NAG and A-QNPE can be bounded by  $\mathcal{O}\left(\frac{nd}{\epsilon^{0.5}}\right)$  and  $\tilde{\mathcal{O}}\left(\frac{nd^{1.2}}{\epsilon^{0.4}} + \frac{d^{2.25}}{\epsilon^{0.5}}\right)$ , respectively. In particular, our method incurs a lower computational cost when  $\epsilon \ll \frac{1}{d^2}$  and  $n \gg d^{1.25}$ .
In this section, we compare the numerical performance of our proposed A-QNPE method with NAG and the classical BFGS quasi-Newton method. For fair comparison, we also use a line search
(a) Convergence by iteration
(b) Convergence by gradient queries
(c) Histogram of gradient queries
Figure 1: Numerical results for logistic regression on a synthetic dataset.
(a) Convergence by iteration
(b) Convergence by gradient queries
Figure 2: Numerical results for log-sum-exp function on a synthetic dataset.
scheme in NAG and BFGS to obtain their best performance [54, 15]. We would like to highlight that our paper mainly focuses on establishing a provable gain for quasi-Newton methods with respect to NAG, and our experimental results are presented to numerically verify our theoretical findings. In the first experiment, we focus on a logistic regression problem with the loss function  $f(\mathbf{x}) = \frac{1}{n}\sum_{i=1}^{n}\log(1 + e^{-y_i\langle\mathbf{a}_i,\mathbf{x}\rangle})$ , where  $\mathbf{a}_1, \ldots, \mathbf{a}_n \in \mathbb{R}^d$  are feature vectors and  $\mathbf{y}_1, \ldots, \mathbf{y}_n \in \{-1,1\}$  are binary labels. We perform our numerical experiments on a synthetic dataset and the data generation process is described in Appendix F. In the second experiment, we consider the log-sum-exp function  $f(\mathbf{x}) = \log(\sum_{i=1}^{n}e^{\langle\mathbf{a}_i,\mathbf{x}\rangle - b_i})$ , where we generate the dataset  $\{(a_i,b_i)\}_{i=1}^n$  following a similar procedure as in [16] (more details in Appendix F). As we observe in Fig. 1(a) and Fig. 2(a), our proposed A-QNPE method converges in much fewer iterations than NAG, while the best performance is achieved by BFGS. Due to the use of line search, we also compare these algorithms in terms of the total number of gradient queries. Moreover, additional plots in terms of the running time are included in Appendix F. As illustrated in Fig. 1(b) and Fig. 2(b), A-QNPE still outperforms NAG but the relative gain becomes less substantial. This is because the line search scheme in NAG only queries the function value at the new point, and thus it only requires one gradient per iteration. On the other hand, we should add that the number of gradient queries per iteration for A-QNPE is still small as guaranteed by our theory. In particular, the histogram of gradient queries in Fig. 1(c) and Fig. 2(c) shows that most of the iterations of A-QNPE require 2-3 gradient queries with an average of less than 3. Finally, although there is no theoretical guarantee showing a convergence gain for BFGS with respect to NAG, we observe that BFGS outperforms all the other considered methods in our experiments. Hence, studying the convergence behavior of BFGS (with line search) in the convex setting is an interesting research direction to explore.
We proposed a quasi-Newton variant of the accelerated proximal extragradient method for solving smooth convex optimization problems. We established two global convergence rates for our A-QNPE method, showing that it requires  $\tilde{\mathcal{O}}\left(\min \left\{\frac{1}{\epsilon^{0.5}}, \frac{d^{0.2}}{\epsilon^{0.4}}\right\}\right)$  gradient queries to find an  $\epsilon$ -accurate solution. In particular, in the regime where  $\epsilon = \Omega \left(\frac{1}{d^2}\right)$ , A-QNPE achieves a gradient oracle complexity of  $\mathcal{O}\left(\frac{1}{\epsilon^{0.5}}\right)$ , matching the complexity of NAG. Moreover, in the regime where  $\epsilon = \tilde{\mathcal{O}}\left(\frac{1}{d^2}\right)$ , it outperforms NAG and improves the complexity to  $\tilde{\mathcal{O}}\left(\frac{d^{0.2}}{\epsilon^{0.4}}\right)$ . To the best of our knowledge, this is the first result showing a provable gain for a quasi-Newton-type method over NAG in the convex setting.
The research of R. Jiang and A. Mokhtari is supported in part by NSF Grants 2007668, 2019844, and 2112471, ARO Grant W911NF2110226, the Machine Learning Lab (MLL) at UT Austin, the Wireless Networking and Communications Group (WNCG) Industrial Affiliates Program, and the NSF AI Institute for Foundations of Machine Learning (IFML). The authors would also like to thank Yair Carmon and the anonymous reviewers for their comments on the first draft of the paper.
[1] W. C. Davidon. Variable metric method for minimization. Technical Report ANL-5990, Argonne National Laboratory, Argonne, IL, 1959.
[2] Roger Fletcher and Michael JD Powell. A rapidly convergent descent method for minimization. The Computer Journal, 6(2):163-168, 1963.
[3] Charles G Broyden. The convergence of single-rank quasi-newton methods. Mathematics of Computation, 24(110):365-382, 1970.
[4] Roger Fletcher. A new approach to variable metric algorithms. The computer journal, 13(3):317-322, 1970.
[5] Donald Goldfarb. A family of variable-metric methods derived by variational means. Mathematics of computation, 24(109):23-26, 1970.
[6] David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathematics of computation, 24(111):647-656, 1970.
[7] Andrew R Conn, Nicholas IM Gould, and Ph L Toint. Convergence of quasi-newton matrices generated by the symmetric rank one update. Mathematical programming, 50(1):177-195, 1991.
[8] H Fayez Khalfan, Richard H Byrd, and Robert B Schnabel. A theoretical and experimental study of the symmetric rank-one update. SIAM Journal on Optimization, 3(1):1-24, 1993.
[9] Charles George Broyden, John E Dennis Jr, and Jorge J Moré. On the local and superlinear convergence of quasi-newton methods. IMA Journal of Applied Mathematics, 12(3):223-245, 1973.
[10] John E Dennis and Jorge J Moré. A characterization of superlinear convergence and its application to quasi-newton methods. Mathematics of computation, 28(126):549-560, 1974.
[11] MJD Powell. On the convergence of the variable metric algorithm. IMA Journal of Applied Mathematics, 7(1):21-36, 1971.
[12] Laurence Charles Ward Dixon. Variable metric algorithms: necessary and sufficient conditions for identical behavior of nonquadratic functions. Journal of Optimization Theory and Applications, 10(1):34-40, 1972.
[13] Richard H Byrd, Jorge Nocedal, and Ya-Xiang Yuan. Global convergence of a class of quasi-newton methods on convex problems. SIAM Journal on Numerical Analysis, 24(5):1171-1190, 1987.
[14] Richard H Byrd, Humaid Fayez Khalfan, and Robert B Schnabel. Analysis of a symmetric rank-one trust region method. SIAM Journal on Optimization, 6(4):1025-1039, 1996.
[15] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer Science+Business Media, LLC, 2006.
[16] Anton Rodomanov and Yurii Nesterov. Greedy quasi-newton methods with explicit superlinear convergence. SIAM Journal on Optimization, 31(1):785-811, 2021.
[17] Anton Rodomanov and Yurii Nesterov. Rates of superlinear convergence for classical quasi-newton methods. Mathematical Programming, 2021.
[56] Arkadi Nemirovski. Information-based complexity of convex programming. Lecture notes, 834, 1995.
[57] Yousef Saad. Numerical methods for large eigenvalue problems: revised edition. SIAM, 2011.
[58] Alp Yurtsever, Joel A Tropp, Olivier Fercoq, Madeleine Udell, and Volkan Cevher. Scalable semidefinite programming. SIAM Journal on Mathematics of Data Science, 3(1):171-200, 2021.
A Optimal Monteiro-Svaiter Acceleration Framework
In this section, we present some general results that hold for the optimal MS Acceleration framework. In particular, in the first part of this section (Section A.1), we present the proof of Proposition 1. In the second part (Section A.2), we further provide some useful additional lemmas.
A.1 Proof of Proposition 1
To begin with, we establish a potential function for Algorithm 1, as shown in Proposition 2. The result is similar to Proposition 1 in [32], but for completeness we present its proof loosely following the strategy in [55, Theorem 5.3]. To simplify the notations, we use  $f^{*}$  to denote the optimal  $f(\mathbf{x}^{*})$ .
Proposition 2. Consider the iterates generated by Algorithm 1. If  $f$  is convex, then
$$
A _ {k + 1} \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) + \frac {1}{2} \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} \leq A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) + \frac {1}{2} \left\| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \right\| ^ {2}. \tag {20}
$$
Moreover, let  $\sigma = \alpha_{1} + \alpha_{2}$  and we have
$$
\sum_ {k = 0} ^ {N - 1} \frac {a _ {k} ^ {2}}{\eta_ {k} ^ {2}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2} \leq \frac {1}{1 - \sigma^ {2}} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2}. \tag {21}
$$
Proof. Since  $f$  is convex, it holds that
$$
f (\mathbf {x} _ {k}) - f (\hat {\mathbf {x}} _ {k + 1}) - \langle \nabla f (\hat {\mathbf {x}} _ {k + 1}), \mathbf {x} _ {k} - \hat {\mathbf {x}} _ {k + 1} \rangle \geq 0,
$$
$$
f \left(\mathbf {x} ^ {*}\right) - f \left(\hat {\mathbf {x}} _ {k + 1}\right) - \left\langle \nabla f \left(\hat {\mathbf {x}} _ {k + 1}\right), \mathbf {x} ^ {*} - \hat {\mathbf {x}} _ {k + 1} \right\rangle \geq 0.
$$
By summing up the two inequalities with weights  $a_{k}$  and  $A_{k}$  respectively, we get
$$
A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) - \left(A _ {k} + a _ {k}\right) \left(f \left(\hat {\mathbf {x}} _ {k + 1}\right) - f ^ {*}\right) - a _ {k} \left\langle \nabla f \left(\hat {\mathbf {x}} _ {k + 1}\right), \mathbf {x} ^ {*} - \hat {\mathbf {x}} _ {k + 1} - \frac {A _ {k}}{a _ {k}} \left(\hat {\mathbf {x}} _ {k + 1} - \mathbf {x} _ {k}\right) \right\rangle \geq 0. \tag {22}
$$
Let  $\tilde{\mathbf{z}}_{k + 1} = \hat{\mathbf{x}}_{k + 1} + \frac{A_k}{a_k}\bigl (\hat{\mathbf{x}}_{k + 1} - \mathbf{x}_k\bigr)$ . By rearranging the terms, (22) can be rewritten as
$$
\left. \left(A _ {k} + a _ {k}\right) \left(f \left(\hat {\mathbf {x}} _ {k + 1}\right) - f ^ {*}\right) - A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) \leq a _ {k} \langle \nabla f \left(\hat {\mathbf {x}} _ {k + 1}\right), \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \rangle . \right. \tag {23}
$$
Moreover, note that the update rule for  $\mathbf{z}_{k + 1}$  in both (8) and (9) can be written as
$$
\mathbf {z} _ {k + 1} - \mathbf {z} _ {k} = - \frac {\hat {\eta} _ {k}}{\eta_ {k}} a _ {k} \nabla f (\hat {\mathbf {x}} _ {k + 1}). \tag {24}
$$
Also, since we also have  $\mathbf{z}_k = \mathbf{y}_k + \frac{A_k}{a_k} (\mathbf{y}_k - \mathbf{x}_k)$  from (2), we can write
$$
\begin{array}{l} \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k} = \left[ \hat {\mathbf {x}} _ {k + 1} + \frac {A _ {k}}{a _ {k}} (\hat {\mathbf {x}} _ {k + 1} - \mathbf {x} _ {k}) \right] - \left[ \mathbf {y} _ {k} + \frac {A _ {k}}{a _ {k}} (\mathbf {y} _ {k} - \mathbf {x} _ {k}) \right] \\ = \frac {A _ {k} + a _ {k}}{a _ {k}} \left(\hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}\right) = \frac {a _ {k}}{\eta_ {k}} \left(\hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}\right), \tag {25} \\ \end{array}
$$
where we used the fact that  $(A_{k} + a_{k})\eta_{k} = a_{k}^{2}$  in the last equality (cf. (2)). Hence, combining (24) and (25) leads to
$$
\left\| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \right\| = \left\| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k} - \left(\mathbf {z} _ {k + 1} - \mathbf {z} _ {k}\right) \right\| = \frac {a _ {k}}{\eta_ {k}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \hat {\eta} _ {k} \nabla f (\hat {\mathbf {x}} _ {k + 1}) \| \leq \sigma \frac {a _ {k}}{\eta_ {k}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {26}
$$
where we used (7) in the last inequality. In the following, we distinguish two cases depending on  $\hat{\eta}_k = \eta_k$  or  $\hat{\eta}_k < \eta_k$ . In both cases, we shall prove that
$$
A _ {k + 1} \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) + \frac {1}{2} \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} \leq A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) + \frac {1}{2} \left\| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \right\| ^ {2} - \frac {\left(1 - \sigma^ {2}\right) a _ {k} ^ {2}}{2 \eta_ {k} ^ {2}} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2}. \tag {27}
$$
If this is true, then Proposition 2 immediately follows. Indeed, since  $\sigma < 1$ , the last term in the right-hand side of (27) is negative, which implies (20). Moreover, (21) follows from summing the inequality in (27) from  $k = 0$  to  $N - 1$ .
Case I:  $\hat{\eta}_k = \eta_k$ . Since by (8) we have  $\mathbf{x}_{k + 1} = \hat{\mathbf{x}}_{k + 1}$  and  $A_{k + 1} = A_{k} + a_{k}$ , (23) becomes
$$
A _ {k + 1} \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) - A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) \leq a _ {k} \langle \nabla f \left(\mathbf {x} _ {k + 1}\right), \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \rangle .
$$
Using  $\mathbf{z}_{k + 1} = \mathbf{z}_k - a_k\nabla f(\mathbf{x}_{k + 1})$  in (8), we have
$$
\begin{array}{l} A _ {k + 1} \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) - A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) \\ \leq \left\langle \mathbf {z} _ {k} - \mathbf {z} _ {k + 1}, \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \right\rangle \\ = \left\langle \mathbf {z} _ {k} - \mathbf {z} _ {k + 1}, \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \right\rangle + \left\langle \mathbf {z} _ {k} - \mathbf {z} _ {k + 1}, \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\rangle \\ = \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {z} _ {k + 1} \| ^ {2} + \frac {1}{2} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \| ^ {2} - \frac {1}{2} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k} \| ^ {2} \tag {28} \\ + \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \| ^ {2} - \frac {1}{2} \| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2} - \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {z} _ {k + 1} \| ^ {2} \\ \leq \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \| ^ {2} - \frac {1}{2} \| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2} - \frac {(1 - \sigma^ {2}) a _ {k} ^ {2}}{2 \eta_ {k} ^ {2}} \| \mathbf {x} _ {k + 1} - \mathbf {y} _ {k} \| ^ {2}, \\ \end{array}
$$
where we used (25) and (26) in the last inequality. This immediately leads to (27) after rearranging the terms.
Case II:  $\hat{\eta}_k < \eta_k$ . Since  $0 < \gamma_k < 1$  and  $\mathbf{x}_{k + 1} = \frac{(1 - \gamma_k)A_k}{A_k + \gamma_ka_k}\mathbf{x}_k + \frac{\gamma_k(A_k + a_k)}{A_k + \gamma_ka_k}\hat{\mathbf{x}}_{k + 1}$  according to (9), by Jensen's inequality we have  $(A_{k} + \gamma_{k}a_{k})f(\mathbf{x}_{k + 1})\leq \gamma_{k}(A_{k} + a_{k})f(\hat{\mathbf{x}}_{k + 1}) + (1 - \gamma_{k})A_{k}f(\mathbf{x}_{k})$ , which further implies that
$$
\left(A _ {k} + \gamma_ {k} a _ {k}\right) \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) - A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) \leq \gamma_ {k} \left(A _ {k} + a _ {k}\right) \left(f \left(\hat {\mathbf {x}} _ {k + 1}\right) - f ^ {*}\right) - \gamma_ {k} A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right).
$$
Moreover, since  $A_{k + 1} = A_k + \gamma_k a_k$  by (9), together with (23) we obtain
$$
A _ {k + 1} \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) - A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) \leq \gamma_ {k} a _ {k} \left\langle \nabla f \left(\tilde {\mathbf {x}} _ {k + 1}\right), \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \right\rangle .
$$
Using  $\mathbf{z}_{k + 1} = \mathbf{z}_k - \gamma_k a_k \nabla f(\hat{\mathbf{x}}_{k + 1})$  in (9), we follow the same reasoning as in (28) to get:
$$
\begin{array}{l} A _ {k + 1} \left(f \left(\mathbf {x} _ {k + 1}\right) - f ^ {*}\right) - A _ {k} \left(f \left(\mathbf {x} _ {k}\right) - f ^ {*}\right) \\ \leq \left\langle \mathbf {z} _ {k} - \mathbf {z} _ {k + 1}, \tilde {\mathbf {z}} _ {k + 1} - \mathbf {x} ^ {*} \right\rangle \\ = \left\langle \mathbf {z} _ {k} - \mathbf {z} _ {k + 1}, \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \right\rangle + \left\langle \mathbf {z} _ {k} - \mathbf {z} _ {k + 1}, \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\rangle \\ = \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {z} _ {k + 1} \| ^ {2} + \frac {1}{2} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k + 1} \| ^ {2} - \frac {1}{2} \| \tilde {\mathbf {z}} _ {k + 1} - \mathbf {z} _ {k} \| ^ {2} \\ + \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \| ^ {2} - \frac {1}{2} \| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \| ^ {2} - \frac {1}{2} \| \mathbf {z} _ {k} - \mathbf {z} _ {k + 1} \| ^ {2} \\ \leq \frac {1}{2} \left\| \mathbf {z} _ {k} - \mathbf {x} ^ {*} \right\| ^ {2} - \frac {1}{2} \left\| \mathbf {z} _ {k + 1} - \mathbf {x} ^ {*} \right\| ^ {2} - \frac {(1 - \sigma^ {2}) a _ {k} ^ {2}}{2 \eta_ {k} ^ {2}} \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| ^ {2}, \\ \end{array}
$$
which also leads to (27).
Next, we prove a lower bound on  $A_N$ . Recall that  $\mathcal{B}$  denotes the set of iteration indices where the line search scheme backtracks, i.e.,  $\mathcal{B} \triangleq \{k : \hat{\eta}_k < \eta_k\}$ .
Lemma 3. For any  $N\geq 0$  , it holds that
$$
A _ {N} \geq \frac {1}{4} \left(\sqrt {\hat {\eta} _ {0}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \sqrt {\hat {\eta} _ {k}}\right) ^ {2}. \tag {29}
$$
Proof. To begin with, according to the update rule of  $A_{k + 1}$  in (8) and (9) and the expression of  $a_{k}$  in (2), the sequence  $\{A_k\}$  follows the dynamic:
$$
A _ {k + 1} = \left\{ \begin{array}{l l} A _ {k} + a _ {k}, & \text {i f} \hat {\eta} _ {k} = \eta_ {k} (k \notin \mathcal {B}); \\ A _ {k} + \gamma_ {k} a _ {k}, & \text {i f} \hat {\eta} _ {k} <   \eta_ {k} (k \in \mathcal {B}), \end{array} \right. \quad \text {w h e r e} \gamma_ {k} = \frac {\hat {\eta} _ {k}}{\eta_ {k}} \text {a n d} a _ {k} = \frac {\eta_ {k} + \sqrt {\eta_ {k} ^ {2} + 4 \eta_ {k} A _ {k}}}{2}.
$$
Since we initialize  $A_0 = 0$ , we have  $a_0 = \eta_0$ . We further have  $A_1 = \hat{\eta}_0$ , since we get  $A_1 = A_0 + a_0 = \hat{\eta}_0$  if  $0 \notin \mathcal{B}$ , while we get  $A_1 = A_0 + \gamma_0 a_0 = \frac{\hat{\eta}_0}{\eta_0} \eta_0 = \hat{\eta}_0$  if  $0 \in \mathcal{B}$ . Moreover:
- In Case I where  $k \notin \mathcal{B}$ , we have
$$
A _ {k + 1} = A _ {k} + a _ {k} = A _ {k} + \frac {\eta_ {k} + \sqrt {\eta_ {k} ^ {2} + 4 \eta_ {k} A _ {k}}}{2} \geq A _ {k} + \frac {\eta_ {k}}{2} + \sqrt {\eta_ {k} A _ {k}} \geq \left(\sqrt {A _ {k}} + \frac {\sqrt {\eta_ {k}}}{2}\right) ^ {2},
$$
which further implies that  $\sqrt{A_{k + 1}}\geq \sqrt{A_k} +\frac{\sqrt{\eta_k}}{2} = \sqrt{A_k} +\frac{\sqrt{\hat{\eta}_k}}{2}$
- In Case II where  $k \in \mathcal{B}$ , we have  $A_{k + 1} = A_k + \gamma_k a_k \geq A_k$ , which implies that  $\sqrt{A_{k + 1}} \geq \sqrt{A_k}$ .
Considering the above, we obtain  $\sqrt{A_N} \geq \sqrt{A_1} + \sum_{1 \leq k \leq N-1, k \notin \mathcal{B}} \frac{\sqrt{\hat{\eta}_k}}{2}$ , which leads to (29).
Lemma 3 provides a lower bound on  $A_{N}$  in terms of the step sizes  $\hat{\eta}_k$  in those iterations where the line search scheme does not backtrack, i.e.,  $k \notin \mathcal{B}$ . The following lemma shows how we can further prove a lower bound in terms of all the step sizes  $\{\hat{\eta}_k\}_{k=0}^{N-1}$ .
Lemma 4. We have
$$
\sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \sqrt {\hat {\eta} _ {k}} \leq \frac {1}{1 - \sqrt {\beta}} \left(\sqrt {\hat {\eta} _ {0}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \sqrt {\hat {\eta} _ {k}}\right). \tag {30}
$$
As a corollary, we have
$$
\sqrt {\hat {\eta} _ {0}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \sqrt {\hat {\eta} _ {k}} \geq \frac {1 - \sqrt {\beta}}{2 - \sqrt {\beta}} \sum_ {k = 0} ^ {N - 1} \sqrt {\hat {\eta} _ {k}}. \tag {31}
$$
Proof. When the line search scheme backtracks, i.e.,  $k \in \mathcal{B}$ , we have  $\hat{\eta}_k \leq \beta \eta_k$ . Therefore,
$$
\sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \sqrt {\hat {\eta} _ {k}} \leq \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \sqrt {\beta \eta_ {k}} \leq \sum_ {k = 1} ^ {N - 1} \sqrt {\beta \eta_ {k}} = \sqrt {\beta \eta_ {1}} + \sum_ {k = 1} ^ {N - 2} \sqrt {\beta \eta_ {k + 1}}. \tag {32}
$$
Moreover, in the update of Algorithm 1, we have  $\eta_{k + 1} = \hat{\eta}_k / \beta$  if  $k\notin \mathcal{B}$  (cf. Line 8) and  $\eta_{k + 1} = \hat{\eta}_k$  otherwise (cf. Line 13). This implies that  $\eta_1\leq \hat{\eta}_0 / \beta$  and we further have
$$
\begin{array}{l} \sqrt {\beta \eta_ {1}} + \sum_ {k = 1} ^ {N - 2} \sqrt {\beta \eta_ {k + 1}} = \sqrt {\beta \eta_ {1}} + \sum_ {1 \leq k \leq N - 2, k \notin \mathcal {B}} \sqrt {\beta \eta_ {k + 1}} + \sum_ {1 \leq k \leq N - 2, k \in \mathcal {B}} \sqrt {\beta \eta_ {k + 1}} \\ \leq \sqrt {\hat {\eta} _ {0}} + \sum_ {1 \leq k \leq N - 2, k \notin \mathcal {B}} \sqrt {\hat {\eta} _ {k}} + \sum_ {1 \leq k \leq N - 2, k \in \mathcal {B}} \sqrt {\beta \hat {\eta} _ {k}} \\ \leq \sqrt {\hat {\eta} _ {0}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \sqrt {\hat {\eta} _ {k}} + \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \sqrt {\beta \hat {\eta} _ {k}}. \tag {33} \\ \end{array}
$$
We combine (32) and (33) to get
$$
\sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \sqrt {\hat {\eta} _ {k}} \leq \sqrt {\hat {\eta} _ {0}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \sqrt {\hat {\eta} _ {k}} + \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \sqrt {\beta \hat {\eta} _ {k}}.
$$
By rearranging the terms and simple algebraic manipulation, we obtain (30) as desired. Finally, (31) follows by adding  $\sqrt{\hat{\eta}_0} +\sum_{1\leq k\leq N - 1,k\notin \mathcal{B}}\sqrt{\hat{\eta}_k}$  to both sides of (30).
Now we are ready to prove Proposition 1.
Proof of Proposition 1. By Proposition 2, the potential function  $\phi_k \triangleq A_k(f(\mathbf{x}_k) - f^*) + \frac{1}{2} \| \mathbf{z}_k - \mathbf{x}^* \|^2$  is non-increasing in each iteration. Hence, via a recursive augment we have  $A_N(f(\mathbf{x}_N) - f^*) \leq \phi_N \leq \dots \leq \phi_0 = \frac{1}{2} \| \mathbf{z}_0 - \mathbf{x}^* \|^2$ , which yields  $f(\mathbf{x}_N) - f^* \leq \frac{\|\mathbf{z}_0 - \mathbf{x}^*\|^2}{2A_N}$ . Moreover, combining Lemma 3 and (31) in Lemma 4 leads to the second inequality in Proposition 1.
A.2 Additional Supporting Lemmas
A crucial part of our analysis is to bound the path length of the sequence  $\{\mathbf{y}_k\}_{k=0}^N$ . This is done in Lemma 8. To achieve this goal we first present the results in Lemmas 5-7, which provide the required ingredients for proving the claim in Lemma 8. In our first intermediate result, we establish uniform upper bounds for the error terms  $\|\mathbf{z}_k - \mathbf{x}^*\|$  and  $\|\mathbf{x}_k - \mathbf{x}^*\|$ .
Lemma 5. Recall that  $\sigma = \alpha_{1} + \alpha_{2}$ . For all  $k\geq 0$ , we have  $\| \mathbf{z}_k - \mathbf{x}^*\| \leq \| \mathbf{z}_0 - \mathbf{x}^*\|$  and  $\| \mathbf{x}_k - \mathbf{x}^*\| \leq \sqrt{\frac{2}{1 - \sigma^2}}\| \mathbf{z}_0 - \mathbf{x}^*\|$ .
Proof. To begin with, it follows from (20) in Proposition 2 that