SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {x} _ {k + 1} = \frac {\left(1 - \gamma_ {k}\right) A _ {k}}{A _ {k} + \gamma_ {k} a _ {k}} \mathbf {x} _ {k} + \frac {\gamma_ {k} \left(A _ {k} + a _ {k}\right)}{A _ {k} + \gamma_ {k} a _ {k}} \hat {\mathbf {x}} _ {k + 1}, \quad \mathbf {z} _ {k + 1} = \mathbf {z} _ {k} - \gamma_ {k} a _ {k} \nabla f (\hat {\mathbf {x}} _ {k + 1}), \quad A _ {k + 1} = A _ {k} + \gamma_ {k} a _ {k}, \tag {9}
$$
where  $\gamma_{k} = \hat{\eta}_{k} / \eta_{k} < 1$ . Accordingly, we decrease the step size in the next iteration by letting  $\eta_{k + 1} = \hat{\eta}_k$  (note that  $\hat{\eta}_k < \eta_k$ ).
- In the fourth stage, we update the Hessian approximation matrix  $\mathbf{B}_{k + 1}$ . Inspired by [35], we depart from the classical quasi-Newton methods and instead let the convergence analysis guide our update scheme. As we will show in Section 3.2, the convergence rate of our method is closely related to the cumulative loss  $\sum_{k\in \mathcal{B}}\ell_k(\mathbf{B}_k)$  incurred by our choices of  $\{\mathbf{B}_k\}$ , where  $\mathcal{B} = \{k:\hat{\eta}_k < \eta_k\}$  denotes the indices where the line search scheme backtracks. Moreover, the loss function has the form  $\ell_k(\mathbf{B}_k)\triangleq \frac{\|\mathbf{w}_k - \mathbf{B}_ks_k\|^2}{\|\mathbf{s}_k\|^2}$ , where  $\mathbf{w}_k\triangleq \nabla f(\tilde{\mathbf{x}}_k) - \nabla f(\mathbf{x}_k)$ ,  $\mathbf{s}_k\triangleq \tilde{\mathbf{x}}_k - \mathbf{x}_k$  and  $\tilde{\mathbf{x}}_k$  is an auxiliary iterate returned by our line search scheme. Thus, this motivates us to employ an online learning algorithm to minimize the cumulative loss. Specifically, in the first case where  $\hat{\eta}_k = \eta_k$  (i.e.,  $k\notin \mathcal{B}$ ), the current Hessian approximation matrix  $\mathbf{B}_k$  does not contribute to the cumulative loss and thus we keep it unchanged (cf. Line 9). Otherwise, we follow an online learning algorithm in the space of matrices. The details will be discussed in Section 3.2.
Finally, we provide a convergence result in the following Proposition for Algorithm 1, which serves as the basis for our convergence analysis. We note that the following results do not require additional conditions on  $\mathbf{B}_k$  other than the ones in (6) and (7). The proof is available in Appendix A.1.
Proposition 1. Let  $\{\mathbf{x}_k\}_{k=0}^N$  be the iterates generated by Algorithm 1. If  $f$  is convex, we have
$$
f (\mathbf {x} _ {N}) - f (\mathbf {x} ^ {*}) \leq \frac {\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2}}{2 A _ {N}} \quad a n d \quad A _ {N} \geq \frac {(1 - \sqrt {\beta}) ^ {2}}{4 (2 - \sqrt {\beta}) ^ {2}} \left(\sum_ {k = 0} ^ {N - 1} \sqrt {\hat {\eta} _ {k}}\right) ^ {2}.
$$
Proposition 1 characterizes the convergence rate of Algorithm 1 by the quantity  $A_N$ , which can be further lower bounded in terms of the step sizes  $\{\hat{\eta}_k\}$ . Moreover, we can observe that larger step sizes will lead to a faster convergence rate. On the other hand, the step size  $\hat{\eta}_k$  is constrained by the condition in (7), which, in turn, depends on our choice of the Hessian approximation matrix  $\mathbf{B}_k$ . Thus, the central goal of our line search scheme and the Hessian approximation update is to make the step size  $\hat{\eta}_k$  as large as possible, which we will describe next.
3.1 Line Search Subroutine
In this section, we specify our line search subroutine to select the step size  $\hat{\eta}_k$  and the iterate  $\hat{\mathbf{x}}_{k + 1}$  in the second stage of A-QNPE. For simplicity, denote  $\nabla f(\mathbf{y}_k)$  by  $\mathbf{g}$  and drop the subscript  $k$  in  $\mathbf{y}_k$  and  $\mathbf{B}_k$ . In light of (6) and (7), our goal in the second stage is to find a pair  $(\hat{\eta},\hat{\mathbf{x}}_{+})$  such that
$$
\left\| \hat {\mathbf {x}} _ {+} - \mathbf {y} + \hat {\eta} (\mathbf {g} + \mathbf {B} (\hat {\mathbf {x}} _ {+} - \mathbf {y})) \right\| \leq \alpha_ {1} \| \hat {\mathbf {x}} _ {+} - \mathbf {y} \|, \tag {10}
$$
$$
\left\| \hat {\mathbf {x}} _ {+} - \mathbf {y} + \hat {\eta} \nabla f \left(\hat {\mathbf {x}} _ {+}\right) \right\| \leq \left(\alpha_ {1} + \alpha_ {2}\right) \left\| \hat {\mathbf {x}} _ {+} - \mathbf {y} \right\|. \tag {11}
$$
As mentioned in the previous section, the condition in (10) can be satisfied by solving the linear system  $(\mathbf{I} + \hat{\eta}\mathbf{B})(\hat{\mathbf{x}}_{+} - \mathbf{y}) = -\hat{\eta}\mathbf{g}$  to a desired accuracy. Specifically, we let
$$
\mathbf {s} _ {+} = \operatorname {L i n e a r S o l v e r} (\mathbf {I} + \hat {\eta} \mathbf {B}, - \hat {\eta} \mathbf {g}; \alpha_ {1}) \quad \text {a n d} \quad \hat {\mathbf {x}} _ {+} = \mathbf {y} + \mathbf {s} _ {+}, \tag {12}
$$
where the oracle LinearSolver is defined as follows.
Definition 1. The oracle LinearSolver(A, b;  $\alpha$ ) takes a matrix  $\mathbf{A} \in \mathbb{S}_{+}^{d}$ , a vector  $\mathbf{b} \in \mathbb{R}^{d}$  and  $\alpha \in (0,1)$  as input, and returns an approximate solution  $\mathbf{s}_{+}$  satisfying  $\| \mathbf{A}\mathbf{s}_{+} - \mathbf{b}\| \leq \alpha \| \mathbf{s}_{+}\|$ .
The most direct way to implement LinearSolver(A, b;  $\alpha$ ) is to compute  $\mathbf{s}_{+} = \mathbf{A}^{-1}\mathbf{b}$ , which however costs  $\mathcal{O}(d^3)$  arithmetic operations. Alternatively, we can implement the oracle more efficiently by using the conjugate residual method [48], which only requires computing matrix-vector products and thus incurs a cost of  $\mathcal{O}(d^2)$ . The details are discussed in Appendix E.1. We characterize the total number of required matrix-vector products for this oracle in Theorem 2.
Now we are ready to describe our line search scheme with the LinearSolver oracle (see also Subroutine 1 in Appendix B). Specifically, we start with the step size  $\eta$  and then reduce it by a factor  $\beta$  until we find a pair  $(\hat{\eta},\hat{\mathbf{x}}_{+})$  that satisfies (11). It can be shown that the line search scheme will terminate in a finite number of steps and return a pair  $(\hat{\eta},\hat{\mathbf{x}}_{+})$  satisfying both conditions in (10) and (11) (see Appendix B.1). Regarding the output, we distinguish two cases: (i) If we pass the test in (11) on our first attempt, we accept the initial step size  $\eta$  and the corresponding iterate  $\hat{\mathbf{x}}_{+}$  (cf. Line 10 in Subroutine 1). (ii) Otherwise, along with the pair  $(\hat{\eta},\hat{\mathbf{x}}_{+})$ , we also return an auxiliary iterate  $\tilde{\mathbf{x}}_{+}$  that we compute from (12) using the rejected step size  $\hat{\eta} /\beta$  (cf. Line 12 in Subroutine 1). As we shall see in Lemma 1, the iterate  $\tilde{\mathbf{x}}_{+}$  is used to derive a lower bound on  $\hat{\eta}$ , which will be the key to our convergence analysis and guide our update of the Hessian approximation matrix. For ease of notation, let  $\mathcal{B}$  be the set of iteration indices where the line search scheme backtracks, i.e.,  $\mathcal{B}\triangleq \{k:\hat{\eta}_k < \eta_k\}$ .
Lemma 1. For  $k \notin \mathcal{B}$  we have  $\hat{\eta}_k = \eta_k$ , while for  $k \in \mathcal{B}$  we have
$$
\hat {\eta} _ {k} > \frac {\alpha_ {2} \beta \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|}{\| \nabla f (\tilde {\mathbf {x}} _ {k + 1}) - \nabla f (\mathbf {y} _ {k}) - \mathbf {B} _ {k} (\tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}) \|} \quad a n d \quad \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \leq \frac {(1 + \alpha_ {1})}{\beta (1 - \alpha_ {1})} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {13}
$$
Lemma 1 provides a lower bound on the step size  $\hat{\eta}_k$  in terms of the approximation error  $\| \nabla f(\tilde{\mathbf{x}}_{k + 1}) - \nabla f(\mathbf{y}_k) - \mathbf{B}_k(\tilde{\mathbf{x}}_{k + 1} - \mathbf{y}_k)\|$ . Hence, a better Hessian approximation matrix  $\mathbf{B}_k$  leads to a larger step size, which in turn implies faster convergence. Also note that the lower bound uses the auxiliary iterate  $\tilde{\mathbf{x}}_{k + 1}$  that is not accepted as the actual iterate. Thus, the second inequality in (13) will be used to relate  $\| \tilde{\mathbf{x}}_{k + 1} - \mathbf{y}_k\|$  with  $\| \hat{\mathbf{x}}_{k + 1} - \mathbf{y}_k\|$ . Finally, we remark that to fully characterize the computational cost, we need to upper bound the total number of line search steps, each of which requires a call to LinearSolver and a call to the gradient oracle. This will be discussed in Theorem 2.
3.2 Hessian Approximation Update via Online Learning with Dynamic Regret
In this section, we discuss how to update the Hessian approximation matrix  $\mathbf{B}_k$  in the fourth stage of A-QNPE. As mentioned earlier, instead of following the classical quasi-Newton updates, we directly motivate our update policy for  $\mathbf{B}_k$  from the convergence analysis. The first step is to connect the
convergence rate of A-QNPE with the Hessian approximation matrices  $\{\mathbf{B}_k\}$ . By Proposition 1, if we define the absolute constant  $C_1 \triangleq \frac{2(2 - \sqrt{\beta})^2}{(1 - \sqrt{\beta})^2}$ , then we can write
$$
f \left(\mathbf {x} _ {N}\right) - f \left(\mathbf {x} ^ {*}\right) \leq \frac {\left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\| ^ {2}}{2 A _ {N}} \leq \frac {C _ {1} \left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\| ^ {2}}{\left(\sum_ {k = 0} ^ {N - 1} \sqrt {\hat {\eta} _ {k}}\right) ^ {2}} \leq \frac {C _ {1} \left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\| ^ {2}}{N ^ {2 . 5}} \sqrt {\sum_ {k = 0} ^ {N - 1} \frac {1}{\hat {\eta} _ {k} ^ {2}}}, \tag {14}
$$
where the last inequality follows from Hölder's inequality. Furthermore, we can establish an upper bound on  $\sum_{k=0}^{N-1} \frac{1}{\hat{\eta}_k^2}$  in terms of the Hessian approximation matrices  $\{\mathbf{B}_k\}$ , as we show next.
Lemma 2. Let  $\{\hat{\eta}_k\}_{k=0}^{N-1}$  be the step sizes in Algorithm 1 using Subroutine 1. Then we have
$$
\sum_ {k = 0} ^ {N - 1} \frac {1}{\hat {\eta} _ {k} ^ {2}} \leq \frac {2 - \beta^ {2}}{(1 - \beta^ {2}) \sigma_ {0} ^ {2}} + \frac {2 - \beta^ {2}}{(1 - \beta^ {2}) \alpha_ {2} ^ {2} \beta^ {2}} \sum_ {0 \leq k \leq N - 1, k \in \mathcal {B}} \frac {\left\| \mathbf {w} _ {k} - \mathbf {B} _ {k} \mathbf {s} _ {k} \right\| ^ {2}}{\left\| \mathbf {s} _ {k} \right\| ^ {2}}, \tag {15}
$$
where  $\mathbf{w}_k\triangleq \nabla f(\tilde{\mathbf{x}}_{k + 1}) - \nabla f(\mathbf{y}_k)$  and  $\mathbf{s}_k\triangleq \tilde{\mathbf{x}}_{k + 1} - \mathbf{y}_k$  for  $k\in \mathcal{B}$ .
The proof of Lemma 2 is given in Appendix C.1. On a high level, for those step sizes  $\hat{\eta}_k$  with  $k\in \mathcal{B}$  we can apply Lemma 1 and directly obtain a lower bound in terms of  $\mathbf{B}_k$ . On the other hand, for  $k\notin \mathcal{B}$ , we have  $\hat{\eta}_k = \eta_k$  and our update rule in Lines 8 and 13 of Algorithm 1 allows us to connect the sequence  $\{\eta_k\}_{k = 0}^{N - 1}$  with the backtracked step sizes  $\{\hat{\eta}_k:k\in \mathcal{B}\}$ . As a result, we note that the sum in (15) only involves the Hessian approximation matrices  $\{\mathbf{B}_k:k\in \mathcal{B}\}$ .
In light of (14) and (15), our update for  $\mathbf{B}_k$  aims to make the right-hand side of (15) as small as possible. To achieve this, we adopt the online learning approach in [35] and view the sum in (15) as the cumulative loss incurred by our choice of  $\{\mathbf{B}_k\}$ . To formalize, define the loss at iteration  $k$  by
$$
\ell_ {k} (\mathbf {B}) \triangleq \left\{ \begin{array}{l l} 0, & \text {i f} k \notin \mathcal {B}, \\ \frac {\| \mathbf {w} _ {k} - \mathbf {B} \mathbf {s} _ {k} \| ^ {2}}{\| \mathbf {s} _ {k} \| ^ {2}}, & \text {o t h e r w i s e}, \end{array} \right. \tag {16}
$$
and consider the following online learning problem: (i) At the  $k$ -th iteration, we choose  $\mathbf{B}_k \in \mathcal{Z}$  where  $\mathcal{Z} \triangleq \{\mathbf{B} \in \mathbb{S}_+^d : 0 \preceq \mathbf{B} \preceq L_1\mathbf{I}\}$ ; (ii) We receive the loss function  $\ell_k(\mathbf{B})$  defined in (16); (iii) We update our Hessian approximation matrix to  $\mathbf{B}_{k + 1}$ . Therefore, we propose to employ an online learning algorithm to update the Hessian approximation matrices  $\{\mathbf{B}_k\}$ , and the task of proving a convergence rate for our A-QNPE algorithm boils down to analyzing the performance of our online learning algorithm. In particular, an upper bound on the cumulative loss  $\sum_{k = 0}^{N - 1} \ell_k(\mathbf{B}_k)$  will directly translate into a convergence rate for A-QNPE by using (14) and (15).
Naturally, the first idea is to update  $\mathbf{B}_k$  by following projected online gradient descent [49]. While this approach would indeed serve our purpose, its implementation could be computationally expensive. Specifically, like other projection-based methods, it requires computing the Euclidean projection onto the set  $\mathcal{Z}$  in each iteration, which in our case amounts to performing a full  $d\times d$  matrix eigendecomposition and would incur a cost of  $\mathcal{O}(d^3)$  (see Appendix C.2). Inspired by the recent work in [50], we circumvent this issue by using a projection-free online learning algorithm, which relies on an approximate separation oracle for  $\mathcal{Z}$  instead of a projection oracle. For simplicity, we first translate and rescale the set  $\mathcal{Z}$  via the transform  $\hat{\mathbf{B}} = \frac{2}{L_1} (\mathbf{B} - \frac{L_1}{2}\mathbf{I})$  to obtain  $\hat{\mathcal{Z}}\triangleq \{\hat{\mathbf{B}}\in \mathbb{S}^d:\| \hat{\mathbf{B}}\|_{\mathrm{op}}\leq 1\}$ . The approximate separation oracle  $\mathrm{SEP}(\mathbf{W};\delta ,q)$  is then defined as follows.
Definition 2. The oracle  $\mathsf{SEP}(\mathbf{W};\delta ,q)$  takes a symmetric matrix  $\mathbf{W}\in \mathbb{S}^d$ $\delta >0$  and  $q\in (0,1)$  as input and returns a scalar  $\gamma >0$  and a matrix  $\mathbf{S}\in \mathbb{S}^d$  with one of the following possible outcomes:
- Case I:  $\gamma \leq 1$ , which implies that, with probability at least  $1 - q$ ,  $\mathbf{W} \in \hat{\mathcal{Z}}$ ;
- Case II:  $\gamma > 1$ , which implies that, with probability at least  $1 - q$ ,  $\mathbf{W} / \gamma \in \hat{\mathcal{Z}}$ ,  $\| \mathbf{S}\|_F \leq 3$  and  $\langle \mathbf{S}, \mathbf{W} - \hat{\mathbf{B}} \rangle \geq \gamma - 1 - \delta$  for any  $\hat{\mathbf{B}}$  such that  $\hat{\mathbf{B}} \in \hat{\mathcal{Z}}$ .
To sum up,  $\mathrm{SEP}(\mathbf{W};\delta ,q)$  has two possible outcomes: with probability  $1 - q$  , either it certifies that  $\mathbf{W}\in \hat{\mathcal{Z}}$  , or it produces a scaled version of  $\mathbf{W}$  that belongs to  $\hat{\mathcal{Z}}$  and an approximate separation hyperplane between  $\mathbf{W}$  and the set  $\hat{\mathcal{Z}}$  . As we show in Appendix E.2, implementing this oracle
requires computing the two extreme eigenvectors and eigenvalues of the matrix  $\mathbf{W}$  in exactly, which can be implemented efficiently by the randomized Lanczos method [51].
Building on the  $\mathrm{SEP}(\mathbf{W};\delta ,q)$  oracle, we design a projection-free online learning algorithm adapted from [35, Subroutine 2]. Since the algorithm is similar to the one proposed in [35], we relegate the details to Appendix C but sketch the main steps in the analysis. To upper bound the cumulative loss  $\sum_{k = 0}^{N - 1}\ell_k(\mathbf{B}_k)$ , we compare the performance of our online learning algorithm against a sequence of reference matrices  $\{\mathbf{H}_k\}_{k = 0}^{N - 1}$ . Specifically, we aim to control the dynamic regret [49, 52, 53] defined by D-Reg $_N$ ( $\{\mathbf{H}_k\}_{k = 0}^{N - 1}$ )  $\triangleq \sum_{k = 0}^{N - 1}(\ell_k(\mathbf{B}_k) - \ell_k(\mathbf{H}_k))$ , as well as the cumulative loss  $\sum_{k = 0}^{N - 1}\ell_k(\mathbf{H}_k)$  by the reference sequence. In particular, in our analysis we show that the choice of  $\mathbf{H}_k \triangleq \nabla^2 f(\mathbf{y}_k)$  for  $k = 0,\dots,N - 1$  allows us to upper bound both quantities.
Remark 2. While our online learning algorithm is similar to the one in [35], our analysis is more challenging due to the lack of strong convexity. Specifically, since  $f$  is assumed to be strongly convex in [35], the iterates converge to  $\mathbf{x}^*$  at least linearly, resulting in less variation in the loss functions  $\{\ell_k\}$ . Hence, the authors in [35] let  $\mathbf{H}_k = \mathbf{H}^* \triangleq \nabla^2 f(\mathbf{x}^*)$  for all  $k$  and proved that  $\sum_{k=0}^{N-1} \ell_k(\mathbf{H}^*)$  remains bounded. In contrast, without linear convergence, we need to use a time-varying sequence  $\{\mathbf{H}_k\}$  to control the cumulative loss. This in turn requires us to bound the variation  $\sum_{k=0}^{N-2} \| \mathbf{H}_{k+1} - \mathbf{H}_k \|_F$ , which involves a careful analysis of the stability property of the sequence  $\{\mathbf{y}_k\}$  in Algorithm 1.
4 Complexity Analysis of A-QNPE
In this section, we present our main theoretical results: we establish the convergence rate of A-QNPE (Theorem 1) and characterize its computational cost in terms of gradient queries and matrix-vector product evaluations (Theorem 2). The proofs are provided in Appendices D and E.3.
Theorem 1. Let  $\{\mathbf{x}_k\}$  be the iterates generated by Algorithm 1 using the line search scheme in Section 3.1, where  $\alpha_{1},\alpha_{2}\in (0,1)$  with  $\alpha_{1} + \alpha_{2} < 1$  and  $\beta \in (0,1)$ , and using the Hessian approximation update in Section 3.2 (the hyperparameters are given in Appendix D). Then with probability at least  $1 - p$ , the following statements hold, where  $C_i$  ( $i = 4,\dots,10$ ) are absolute constants only depending on  $\alpha_{1},\alpha_{2}$  and  $\beta$ .
(a) For any  $k\geq 0$  we have
$$
f \left(\mathbf {x} _ {k}\right) - f \left(\mathbf {x} ^ {*}\right) \leq \frac {C _ {4} L _ {1} \left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\| ^ {2}}{k ^ {2}} + \frac {C _ {5} \left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\| ^ {2}}{\sigma_ {0} k ^ {2 . 5}}. \tag {17}
$$
(b) Furthermore, for any  $k \geq 0$ ,
$$
f \left(\mathbf {x} _ {k}\right) - f \left(\mathbf {x} ^ {*}\right) \leq \frac {\left\| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \right\| ^ {2}}{k ^ {2 . 5}} \left(M + C _ {1 0} L _ {1} L _ {2} d \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| \log^ {+} \left(\frac {\operatorname* {m a x} \left\{\frac {L _ {1}}{\alpha_ {2} \beta} , \frac {1}{\sigma_ {0}} \right\} k ^ {2 . 5}}{\sqrt {M}}\right)\right) ^ {\frac {1}{2}}, \tag {18}
$$
where we define  $\log^{+}(x) \triangleq \max \{\log(x), 0\}$  and the quantity  $M$  is given by
$$
M \triangleq \frac {C _ {6}}{\sigma_ {0} ^ {2}} + C _ {7} L _ {1} ^ {2} + C _ {8} \| \mathbf {B} _ {0} - \nabla^ {2} f (\mathbf {z} _ {0}) \| _ {F} ^ {2} + C _ {9} L _ {2} ^ {2} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2} + C _ {1 0} L _ {1} L _ {2} d \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| \tag {19}
$$
Both results in Theorem 1 are global, as they are valid for any initial points  $\mathbf{x}_0$ ,  $\mathbf{z}_0$  and any initial matrix  $\mathbf{B}_0$ . Specifically, Part (a) of Theorem 1 shows that A-QNPE converges at a rate of  $\mathcal{O}(1 / k^2)$ , matching the rate of NAG [23] that is known to be optimal in the regime where  $k = \mathcal{O}(d)$  [28, 29]. Furthermore, Part (b) of Theorem 1 presents a convergence rate of  $\mathcal{O}(\sqrt{d\log(k)} /k^{2.5})$ . To see this, note that since we have  $0\preceq \mathbf{B}_0\preceq L_1\mathbf{I}$  and  $0\preceq \nabla^{2}f(\mathbf{z}_{0})\preceq L_{1}\mathbf{I}$ , in the worst case  $\| \mathbf{B}_0 - \nabla^2 f(\mathbf{z}_0)\| _F^2$  in the expression of (19) can be upper bounded by  $L_1^2 d$ . Thus, assuming that  $L_{1},L_{2}$  and  $\| \mathbf{z}_0 - \mathbf{x}^*\|$  are on the order of  $\mathcal{O}(1)$ , we have  $M = \mathcal{O}(d)$  and the convergence rate in (18) can be simplified to  $\mathcal{O}(\sqrt{d\log(k)} /k^{2.5})$ . Notably, this rate surpasses the  $\mathcal{O}(1 / k^2)$  rate when  $k = \Omega (d\log d)$ . To the best of our knowledge, this is the first work to show a convergence rate faster than  $\mathcal{O}(1 / k^2)$  for a quasi-Newton-type method in the convex setting, thus establishing a provable advantage over NAG.
Table 1: The comparison of NAG and our proposed method in terms of computational cost.
Methods	Gradient queires	Matrix-vector products
NAG	O(ε-0.5)	N.A.
A-QNPE (ours)	O(min{ε-0.5, d0.2 ε-0.4})	O(min{d0.25 ε-0.5, ε-0.625})
Remark 3 (Iteration complexity). Based on Theorem 1, we can find A-QNPE's iteration complexity. Define  $N_{\epsilon}$  as the number of iterations required by A-QNPE to find an  $\epsilon$ -accurate solution, i.e.,  $f(\mathbf{x}) - f(\mathbf{x}^{*}) \leq \epsilon$ . When  $\sqrt{\epsilon} > \frac{1}{d \log d}$ , the rate in (17) is better and we have  $N_{\epsilon} = \mathcal{O}\left(\frac{1}{\epsilon^{0.5}}\right)$ . Conversely, when  $\sqrt{\epsilon} < \frac{1}{d \log d}$ , the rate in (18) is the better one, resulting in  $N_{\epsilon} = \mathcal{O}\left(\left(\frac{d}{\epsilon^{2}} \log \frac{d}{\epsilon^{2}}\right)^{0.2}\right)$ . Hence, to achieve an  $\epsilon$ -accurate solution A-QNPE requires  $\mathcal{O}\left(\min\left\{\frac{1}{\epsilon^{0.5}}, \frac{d^{0.2}}{\epsilon^{0.4}} (\log \frac{d}{\epsilon^{2}})^{0.2}\right\}\right)$  iterations.
Remark 4 (Special case). If the initial point  $\mathbf{z}_0$  is close to an optimal solution  $\mathbf{x}^*$  and the initial Hessian approximation matrix  $\mathbf{B}_0$  is chosen properly, the dependence on  $d$  in the convergence rate of (18) can be eliminated. Specifically, if  $\| \mathbf{z}_0 - \mathbf{x}^*\| = \mathcal{O}\left(\frac{1}{d}\right)$  and we set  $\mathbf{B}_0 = \nabla^2 f(\mathbf{z}_0)$ , then we have  $M = \mathcal{O}(1)$  and this leads to a local dimension-independent rate of  $\mathcal{O}(\sqrt{\log k} /k^{2.5})$ .
Recall that in each iteration of Algorithm 1, we need to execute a line search subroutine (Section 3.1) and a Hessian approximation update subroutine (Section 3.2). Thus, to fully characterize the computational cost of Algorithm 1, we need to upper bound the total number of gradient queries as well as the total number of matrix-vector product evaluations, which is the goal of Theorem 2.
Theorem 2. Recall that  $N_{\epsilon}$  denotes the minimum number of iterations required by Algorithm 1 to find an  $\epsilon$ -accurate solution according to Theorem 1. Then, with probability at least  $1 - p$ :
(a) The total number of gradient queries is bounded by  $3N_{\epsilon} + \log_{1 / \beta}\left(\frac{\sigma_0L_1}{\alpha_2}\right)$ .
(b) The total number of matrix-vector product evaluations in the LinearSolver oracle is bounded by  $N_{\epsilon} + C_{11}\sqrt{\sigma_0L_1} + C_{12}\sqrt{\frac{L_1\|\mathbf{z}_0 - \mathbf{x}^*\|^2}{2\epsilon}}$ , where  $C_{11}$  and  $C_{12}$  are absolute constants.
(c) The total number of matrix-vector product evaluations in the SEP oracle is bounded by  $\mathcal{O}\big(N_{\epsilon}^{1.25}(\log N_{\epsilon})^{0.5}\log \big(\frac{\sqrt{d}N_{\epsilon}}{p}\big)\big)$ .
If the initial step size is chosen as  $\sigma_0 = \frac{\alpha_2}{L_1}$ , Theorem 2(a) implies that A-QNPE requires no more than 3 gradient queries per iteration on average. Thus, the gradient oracle complexity of A-QNPE is the same as the iteration complexity, i.e.,  $\mathcal{O}(\min\{\frac{1}{\epsilon^{0.5}}, \frac{d^{0.2}}{\epsilon^{0.4}} (\log \frac{d}{\epsilon^2})^{0.2}\})$ . On the other hand, the complexity in terms of matrix-vector products is worse. More precisely, by using the expression of  $N_{\epsilon}$  in Remark 3, Parts (b) and (c) imply that the total number of matrix-vector product evaluations in the LinearSolver and SEP oracles can be bounded by  $\mathcal{O}\left(\frac{1}{\epsilon^{0.5}}\right)$  and  $\tilde{\mathcal{O}}\left(\min\left\{\frac{d^{0.25}}{\epsilon^{0.5}}, \frac{1}{\epsilon^{0.625}}\right\}\right)$ , respectively.