SlowGuess/NeurIPS_2023_OCR huggingface dataset:
E.2 Implementation of SEP Oracle
We implement the SEP oracle in Definition 2 based on the classical Lanczos method with a random start, where the initial vector is chosen randomly and uniformly from the unit sphere (see, e.g., [57, 58]). For completeness, the full algorithm is described in Subroutine 4.
To prove the correctness of our algorithm, we first recall a classical result in [51] on the convergence behavior of the Lanczos method.
Proposition 3 ([51, Theorem 4.2]). Consider a symmetric matrix  $\mathbf{W}$  and let  $\lambda_{1}(\mathbf{W})$  and  $\lambda_{d}(\mathbf{W})$  denote its largest and smallest eigenvalues, respectively. Then after  $k$  iterations of the Lanczos method with a random start, we find unit vectors  $\mathbf{u}^{(1)}$  and  $\mathbf{u}^{(d)}$  such that
$$
\mathbb {P} \left(\left\langle \mathbf {W u} ^ {(1)}, \mathbf {u} ^ {(1)} \right\rangle \leq \lambda_ {1} (\mathbf {W}) - \epsilon \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right)\right) \leq 1. 6 4 8 \sqrt {d e ^ {- \sqrt {\epsilon} (2 k - 1)}},
$$
$$
\mathbb {P} \left(\left\langle \mathbf {W u} ^ {(d)}, \mathbf {u} ^ {(d)} \right\rangle \geq \lambda_ {d} (\mathbf {W}) + \epsilon \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right)\right) \leq 1. 6 4 8 \sqrt {d e ^ {- \sqrt {\epsilon} (2 k - 1)}},
$$
As a corollary, to ensure that, with probability at least  $1 - q$ ,
$\langle \mathbf{W}\mathbf{u}^{(1)},\mathbf{u}^{(1)}\rangle >\lambda_1(\mathbf{W}) - \epsilon (\lambda_1(\mathbf{W}) - \lambda_d(\mathbf{W}))$  and  $\langle \mathbf{W}\mathbf{u}^{(d)},\mathbf{u}^{(d)}\rangle <  \lambda_n(\mathbf{W}) + \epsilon (\lambda_1(\mathbf{W}) - \lambda_d(\mathbf{W}))$  the number of iterations can be bounded by  $\lceil \frac{1}{4}\epsilon^{-1 / 2}\log (11d / q^2) + \frac{1}{2}\rceil$
Lemma 20. Let  $\gamma$  and  $\mathbf{S}$  be the output of  $\mathrm{SEP}(\mathbf{W};\delta ,q)$  in Subroutine 4. Then with probability at least  $1 - q$ , they satisfy one of the following properties:
- Case I:  $\gamma \leq 1$ , then we have  $\| \mathbf{W}\|_{\mathrm{op}}\leq 1$
- Case II:  $\gamma > 1$ , then we have  $\| \mathbf{W} / \gamma \|_{\mathrm{op}} \leq 1$ ,  $\| \mathbf{S} \|_F = 3$  and  $\langle \mathbf{S}, \mathbf{W} - \hat{\mathbf{B}} \rangle \geq \gamma - 1$  for any  $\hat{\mathbf{B}}$  such that  $\| \hat{\mathbf{B}} \|_{\mathrm{op}} \leq 1$ .
Subroutine 4 SEP(W;  $\delta, q$ )
1: Input:  $\mathbf{W} \in \mathbb{S}^d$ ,  $\delta > 0$ ,  $q \in (0,1)$   
2: Set the number of iterations  $N_1 \gets \min \left\{\left[\log \frac{11d}{q^2} + \frac{1}{2}\right], d\right\}$   
3: Run Lanczos method with a random start for  $N_1$  iterations to get  $\mathbf{u}^{(1)}$  and  $\mathbf{u}^{(d)}$  (cf. Proposition 3)  
4: Set  $\hat{\lambda}_1 \gets \langle \mathbf{W}\mathbf{u}^{(1)}, \mathbf{u}^{(1)}\rangle$  and  $\hat{\lambda}_d \gets \langle \mathbf{W}\mathbf{u}^{(d)}, \mathbf{u}^{(d)}\rangle$   
5: Set  $\hat{\lambda}_{\mathrm{max}} \gets \max \{\hat{\lambda}_1, -\hat{\lambda}_d\}$   
6: if  $\hat{\lambda}_{\mathrm{max}} \leq 1/2$  then # Case I:  $\gamma \leq 1$ , which implies  $\|\mathbf{W}\|_{\mathrm{op}} \leq 1$   
7: Return  $\gamma = 2\hat{\lambda}_{\mathrm{max}}$  and  $\mathbf{S} = 0$   
8: else if  $\hat{\lambda}_{\mathrm{max}} \geq 2$  then # Case II:  $\gamma > 1$  and  $\mathbf{S}$  defines a separating hyperplane  
9: if  $\hat{\lambda}_1 > -\hat{\lambda}_d$  then  
10: Return  $\gamma = 2\hat{\lambda}_{\mathrm{max}}$  and  $\mathbf{S} = 3\mathbf{u}^{(1)}(\mathbf{u}^{(1)})^\top$   
11: else  
12: Return  $\gamma = 2\hat{\lambda}_{\mathrm{max}}$  and  $\mathbf{S} = -3\mathbf{u}^{(d)}(\mathbf{u}^{(d)})^\top$   
13: end if  
14: else #  $\frac{1}{2} < \hat{\lambda}_{\mathrm{max}} < 2$   
15: Set the number of iterations  $N_2 \gets \min \left\{\left[\frac{1}{4\sqrt{2}\delta}\log \frac{11d}{q^2} + \frac{1}{2}\right], d\right\}$   
16: Run Lanczos method with a random start for  $N_2$  iterations to get  $\tilde{\mathbf{u}}^{(1)}$  and  $\tilde{\mathbf{u}}^{(d)}$  (cf. Proposition 3)  
17: Set  $\tilde{\lambda}_1 \gets \langle \mathbf{W}\tilde{\mathbf{u}}^{(1)}, \tilde{\mathbf{u}}^{(1)}\rangle$  and  $\tilde{\lambda}_d \gets \langle \mathbf{W}\tilde{\mathbf{u}}^{(d)}, \tilde{\mathbf{u}}^{(d)}\rangle$   
18: Set  $\tilde{\lambda}_{\mathrm{max}} = \max \{\tilde{\lambda}_1, -\tilde{\lambda}_d\}$   
19: if  $\tilde{\lambda}_{\mathrm{max}} \leq 1 - \delta$  then  
20: Return  $\gamma = \tilde{\lambda}_{\mathrm{max}} + \delta$  and  $\mathbf{S} = 0$   
21: else if  $\tilde{\lambda}_1 \geq -\tilde{\lambda}_d$  then  
22: Return  $\gamma = \tilde{\lambda}_{\mathrm{max}} + \delta$  and  $\mathbf{S} = \tilde{\mathbf{u}}^{(1)}(\tilde{\mathbf{u}}^{(1)})^\top$   
23: else  
24: Return  $\gamma = \tilde{\lambda}_{\mathrm{max}} + \delta$  and  $\mathbf{S} = -\tilde{\mathbf{u}}^{(d)}(\tilde{\mathbf{u}}^{(d)})^\top$   
25: end if  
26: end if
Proof. Note that in Subroutine 4, we first run the Lanczos method for  $\left[\epsilon^{-1/2}\log\frac{11d}{q^2}+\frac{1}{2}\right]$  iterations, where  $\epsilon=\frac{1}{4}$ . Thus, by Proposition 3, with probability at least  $1-q/2$  we have
$$
\hat {\lambda} _ {1} \triangleq \left\langle \mathbf {W} \mathbf {u} ^ {(1)}, \mathbf {u} ^ {(1)} \right\rangle \geq \lambda_ {1} (\mathbf {W}) - \frac {1}{4} \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right), \tag {83}
$$
$$
\hat {\lambda} _ {d} \triangleq \langle \mathbf {W} \mathbf {u} ^ {(d)}, \mathbf {u} ^ {(d)} \rangle \leq \lambda_ {d} (\mathbf {W}) + \frac {1}{4} \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right). \tag {84}
$$
Combining (83) and (84), we get
$$
\frac {1}{2} (\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})) \leq \hat {\lambda} _ {1} - \hat {\lambda} _ {d} \Rightarrow \lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W}) \leq 2 (\hat {\lambda} _ {1} - \hat {\lambda} _ {d}).
$$
By plugging the above inequality back into (83) and (84), we further have
$$
\lambda_ {1} (\mathbf {W}) \leq \hat {\lambda} _ {1} + \frac {1}{4} \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right) \leq \hat {\lambda} _ {1} + \frac {1}{2} \left(\hat {\lambda} _ {1} - \hat {\lambda} _ {d}\right), \tag {85}
$$
$$
\lambda_ {d} (\mathbf {W}) \geq \hat {\lambda} _ {d} - \frac {1}{4} \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right) \geq \hat {\lambda} _ {d} - \frac {1}{2} \left(\hat {\lambda} _ {1} - \hat {\lambda} _ {d}\right). \tag {86}
$$
Let  $\hat{\lambda}_{\mathrm{max}} = \max \{\hat{\lambda}_1, - \hat{\lambda}_d\}$ . By (85) and (86), we can further bound the eigenvalues of  $\mathbf{W}$  by
$$
\lambda_ {1} (\mathbf {W}) \leq \hat {\lambda} _ {\max } + \frac {1}{2} \cdot 2 \hat {\lambda} _ {\max } = 2 \hat {\lambda} _ {\max } \quad \text {a n d} \quad \lambda_ {d} (\mathbf {W}) \geq - \hat {\lambda} _ {\max } - \frac {1}{2} \cdot 2 \hat {\lambda} _ {\max } = - 2 \hat {\lambda} _ {\max }.
$$
Hence, we can see that  $\| \mathbf{W}\|_{\mathrm{op}} = \max \{\lambda_1(\mathbf{W}), - \lambda_d(\mathbf{W})\} \leq 2\hat{\lambda}_{\mathrm{max}}$ . Now we distinguish three cases.
(a) If  $\hat{\lambda}_{\mathrm{max}} \leq \frac{1}{2}$ , then we are in Case I and the ExtEvec oracle outputs  $\gamma = 2\hat{\lambda}_{\mathrm{max}} \leq 1$  and  $\mathbf{S} = 0$ . In this case, we indeed have  $\| \mathbf{W} \|_{\mathrm{op}} \leq \gamma \leq 1$ .
(b) If  $\hat{\lambda}_{\mathrm{max}} \geq 2$ , then we are in Case II. In addition, if  $\hat{\lambda}_1 \geq -\hat{\lambda}_d$ , then the ExtEvec oracle returns  $\gamma = 2\hat{\lambda}_{\mathrm{max}}$  and  $\mathbf{S} = 3\mathbf{u}^{(1)}(\mathbf{u}^{(1)})^\top$ . Similarly, if  $-\hat{\lambda}_d > \hat{\lambda}_1$ , then the ExtEvec oracle returns  $\gamma = 2\hat{\lambda}_{\mathrm{max}}$  and  $\mathbf{S} = -3\mathbf{u}^{(d)}(\mathbf{u}^{(d)})^\top$ . Without loss of generality, consider the case where  $\hat{\lambda}_1 \geq -\hat{\lambda}_d$ . Since  $\| \mathbf{W}\|_{\mathrm{op}} \leq 2\hat{\lambda}_{\mathrm{max}} = \gamma$ , we have  $\| \mathbf{W} / \gamma \|_{\mathrm{op}} \leq 1$ . Also, since  $\mathbf{u}_1$  is a unit vector, we have  $\| \mathbf{S}\|_F = 3\| \mathbf{u}^{(1)}\|^2 = 3$ . Finally, for any  $\hat{\mathbf{B}}$  such that  $\| \hat{\mathbf{B}}\|_{\mathrm{op}} \leq 1$ , we have
$$
\langle \mathbf {S}, \mathbf {W} - \hat {\mathbf {B}} \rangle = 3 (\mathbf {u} ^ {(1)}) ^ {\top} \mathbf {W} \mathbf {u} ^ {(1)} - 3 (\mathbf {u} ^ {(1)}) ^ {\top} \hat {\mathbf {B}} \mathbf {u} ^ {(1)} \geq 3 \hat {\lambda} _ {\max } - 3 \geq 2 \hat {\lambda} _ {\max } - 1 = \gamma - 1,
$$
where we used the fact that  $\hat{\lambda}_{\mathrm{max}} \geq 2$  in the last inequality.
(c) If  $\frac{1}{2} < \hat{\lambda}_{\max} < 2$ , we continue to run the Lanczos method for a total number of  $\left[\frac{1}{4}\epsilon^{-1/2}\log \frac{11d}{q^2} + \frac{1}{2}\right]$  iterations, where  $\epsilon = \frac{1}{8}\delta$ . Thus, by Proposition 3, with probability at least  $1 - q/2$  we have
$$
\tilde {\lambda} _ {1} \triangleq \left\langle \mathbf {W} \tilde {\mathbf {u}} ^ {(1)}, \tilde {\mathbf {u}} ^ {(1)} \right\rangle \geq \lambda_ {1} (\mathbf {W}) - \frac {1}{8} \delta \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right), \tag {87}
$$
$$
\tilde {\lambda} _ {d} \triangleq \langle \mathbf {W} \tilde {\mathbf {u}} ^ {(d)}, \tilde {\mathbf {u}} ^ {(d)} \rangle \leq \lambda_ {d} (\mathbf {W}) + \frac {1}{8} \delta \left(\lambda_ {1} (\mathbf {W}) - \lambda_ {d} (\mathbf {W})\right). \tag {88}
$$
Let  $\tilde{\lambda}_{\mathrm{max}} = \max \{\tilde{\lambda}_1, - \tilde{\lambda}_d\}$ . Since we have  $\lambda_1(\mathbf{W})\leq 2\hat{\lambda}_{\mathrm{max}}\leq 4$  and  $\lambda_{d}(\mathbf{W})\geq -2\hat{\lambda}_{\mathrm{max}}\geq -4$ , the above implies that  $\tilde{\lambda}_1\geq \lambda_1(\mathbf{W}) - \delta$  and  $\tilde{\lambda}_d\leq \lambda_d(\mathbf{W}) + \delta$ . Hence, we can see that  $\| \mathbf{W}\|_{\mathrm{op}} = \max \{\lambda_1(\mathbf{W}), - \lambda_d(\mathbf{W})\} \leq \hat{\lambda}_{\mathrm{max}} + \delta$ . We further consider two subcases.
(c1) If  $\tilde{\lambda}_{\mathrm{max}} \leq 1 - \delta$ , then we are in Case I and the ExtEvec oracle outputs  $\gamma = \tilde{\lambda}_{\mathrm{max}} + \delta$  and  $\mathbf{S} = 0$ . In this case, we indeed have  $\| \mathbf{W} \|_{\mathrm{op}} \leq \gamma \leq 1$ .
(c2) If  $\tilde{\lambda}_{\mathrm{max}} > 1 - \delta$ , then we are in Case II. In addition, if  $\tilde{\lambda}_1 \geq -\tilde{\lambda}_d$ , then the ExtEvec oracle returns  $\gamma = \tilde{\lambda}_{\mathrm{max}} + \delta$  and  $\mathbf{S} = \tilde{\mathbf{u}}^{(1)}(\tilde{\mathbf{u}}^{(1)})^\top$ . Similarly, if  $-\tilde{\lambda}_d > \tilde{\lambda}_1$ , then the ExtEvec oracle returns  $\gamma = \tilde{\lambda}_{\mathrm{max}} + \delta$  and  $\mathbf{S} = -\tilde{\mathbf{u}}^{(d)}(\tilde{\mathbf{u}}^{(d)})^\top$ . Without loss of generality, consider the case where  $\tilde{\lambda}_1 \geq -\tilde{\lambda}_d$ . Since  $\| \mathbf{W}\|_{\mathrm{op}} \leq \tilde{\lambda}_{\mathrm{max}} + \delta = \gamma$ , we have  $\| \mathbf{W} / \gamma \|_{\mathrm{op}} \leq 1$ . Also, since  $\tilde{\mathbf{u}}^{(1)}$  is a unit vector, we have  $\| \mathbf{S}\|_F = \| \tilde{\mathbf{u}}^{(1)}\|^2 = 1$ . Finally, for any  $\hat{\mathbf{B}}$  such that  $\| \hat{\mathbf{B}}\|_{\mathrm{op}} \leq 1$ , we have
$$
\langle \mathbf {S}, \mathbf {W} - \hat {\mathbf {B}} \rangle = (\tilde {\mathbf {u}} ^ {(1)}) ^ {\top} \mathbf {W} \tilde {\mathbf {u}} ^ {(1)} - (\tilde {\mathbf {u}} ^ {(1)}) ^ {\top} \hat {\mathbf {B}} \tilde {\mathbf {u}} ^ {(1)} \geq \tilde {\lambda} _ {\max } - 1 = \gamma - 1 - \delta .
$$
E.3 Proof of Theorem 2
We divide the proof of Theorem 2 into the following three lemmas.
Lemma 21. If we run Algorithm 1 as specified in Theorem 1 for  $N$  iterations, then the total number of line search steps can be bounded by  $2N + \log_{1 / \beta}(\sigma_0L_1 / \alpha_2)$ . As a corollary, the total number of gradient queries is bounded by  $3N_{\epsilon} + \log_{1 / \beta}(\frac{\sigma_0L_1}{\alpha_2})$ .
Proof. In our backtracking scheme, the number of steps in each iteration is given by  $\log_{1 / \beta}(\eta_k / \hat{\eta}_k) + 1$ . Also note that  $\eta_{k + 1}\leq \hat{\eta}_k / \beta$  for all  $k\geq 0$ . Thus, we have
$$
\begin{array}{l} \sum_ {k = 0} ^ {N - 1} \left(\log_ {1 / \beta} \frac {\eta_ {k}}{\hat {\eta} _ {k}} + 1\right) = N + \log_ {1 / \beta} \frac {\sigma_ {0}}{\hat {\eta} _ {0}} + \sum_ {k = 0} ^ {N - 2} \log_ {1 / \beta} \frac {\eta_ {k + 1}}{\hat {\eta} _ {k + 1}} \\ \leq N + \log_ {1 / \beta} \frac {\sigma_ {0}}{\hat {\eta} _ {0}} + \sum_ {k = 0} ^ {N - 2} \left(\log_ {1 / \beta} \frac {\hat {\eta} _ {k}}{\hat {\eta} _ {k + 1}} + 1\right) \\ \leq 2 N - 1 + \log_ {1 / \beta} \frac {\sigma_ {0}}{\hat {\eta} _ {N - 1}} \\ \end{array}
$$
Furthermore, since  $\hat{\eta}_k \geq \alpha_2\beta / L_1$  for all  $k \geq 0$ , we arrive at the conclusion.
Lemma 22. The total number of matrix-vector product evaluations in the LinearSolver oracle is bounded by  $N_{\epsilon} + C_{11}\sqrt{\sigma_0L_1} +C_{12}\sqrt{\frac{L_1\|\mathbf{z}_0 - \mathbf{x}^*\|^2}{2\epsilon}}$ , where  $C_{11}$  and  $C_{12}$  are absolute constants.
Proof. Our proof loosely follows the strategy in [32]. We first bound the number of steps required by Subroutine 3 before it terminates.
Lemma 23. Suppose  $\mathbf{A} \succeq \mathbf{I}$ . Then Subroutine 3 terminates after at most  $\left\lceil \sqrt{\frac{\alpha + 1}{\alpha} \lambda_{\max}(\mathbf{A})} - 1 \right\rceil$  iterations.
Proof. Note that  $\| \mathbf{s}_k\| _2\geq \| \mathbf{s}^*\| _2 - \| \mathbf{s}_k - \mathbf{s}^*\| _2$ . Also, since  $\mathbf{A}\succeq \mathbf{I}$ , we have  $\| \mathbf{s}_k - \mathbf{s}^*\| _2\leq$ $\| \mathbf{A}(\mathbf{s}_k - \mathbf{s}^*)\| _2 = \| \mathbf{r}_k\| _2$ . Therefore, we have
$$
\| \mathbf {r} _ {k} \| _ {2} \leq \alpha \| \mathbf {s} _ {k} \| _ {2} \quad \Leftarrow \quad \| \mathbf {r} _ {k} \| _ {2} \leq \alpha \| \mathbf {s} ^ {*} \| _ {2} - \alpha \| \mathbf {r} _ {k} \| _ {2} \quad \Leftarrow \quad \| \mathbf {r} _ {k} \| _ {2} \leq \frac {\alpha}{\alpha + 1} \| \mathbf {s} ^ {*} \| _ {2}.
$$
By using Lemma 19, we only need  $k \geq \sqrt{\frac{\alpha + 1}{\alpha} \lambda_{\max}(\mathbf{A})} - 1$  to achieve  $\| \mathbf{A}\mathbf{s}_k - \mathbf{b}\| \leq \alpha \| \mathbf{s}_k\|$ .
Moreover, when the step size is smaller enough, we can show that Subroutine 3 will terminate in one iteration.
Lemma 24. Let  $\mathbf{A} = \mathbf{I} + \eta \mathbf{B}$ . When  $\eta \leq \frac{\alpha}{2L_1}$ , Algorithm 3 terminates in one iteration.
Proof. From the update rule of Subroutine 3, we can compute that  $\mathbf{s}_1 = \frac{\mathbf{b}^\top\mathbf{Ab}}{\|\mathbf{Ab}\|_2^2}\mathbf{b}$ , which implies
$$
\| \mathbf {s} _ {1} \| = \| \mathbf {b} \| \cdot \frac {\| \mathbf {A} ^ {1 / 2} \mathbf {b} \| ^ {2}}{(\mathbf {A} ^ {1 / 2} \mathbf {b}) ^ {\top} \mathbf {A} (\mathbf {A} ^ {1 / 2} \mathbf {b})} \geq \frac {\| \mathbf {b} \|}{\lambda_ {\max } (\mathbf {A})} \geq \frac {\| \mathbf {b} \|}{1 + \eta L _ {1}}.
$$
On the other hand, we also have
$$
\left\| \mathbf {r} _ {1} \right\| \leq \left\| \mathbf {A b} - \mathbf {b} \right\| = \eta \left\| \mathbf {B b} \right\| \leq \eta L _ {1} \| \mathbf {b} \|.
$$
Moreover, when  $\eta \leq \frac{\alpha}{2L_1}$ , we have  $\eta L_1 \leq \frac{\alpha}{1 + \eta L_1}$ , which implies that  $\| \mathbf{r}_1 \| \leq \alpha \| \mathbf{s}_1 \|$ .
Now we upper bound the total number of matrix-vector products in Algorithm 1. Note that at the  $k$ -th iteration, we use the LinearSolver oracle with  $\mathbf{A} = \mathbf{I} + \eta_{+}\mathbf{B}_{k}$  where  $\eta_{+} = \eta_{k}\beta^{i}$ . We can store the vector  $\mathbf{B}_k\mathbf{b}$  at the beginning and reuse it to compute  $\mathbf{s}_1$  when the step size  $\eta_{+} < \frac{\alpha_{1}}{2L_{1}}$ . And when  $\beta^{i}\eta_{k}L_{1}\geq \frac{\alpha_{1}}{2}$ , it holds that
$$
1 + \beta^ {i} \eta_ {k} L _ {1} \leq \frac {\alpha_ {1} + 2}{\alpha_ {1}} \beta^ {i} \eta_ {k} L _ {1}.
$$
Thus, at the  $k$ -th iteration, the number of matrix-vector products can be bounded by
$$
\begin{array}{l} \mathrm{MV}_{k}\leq 1 + \sum_{i\geq 0,\eta_{k}\beta^{i}\geq \frac{\alpha_{1}}{2L_{1}}}\sqrt{\frac{\alpha_{1} + 1}{\alpha_{1}}(1 + \eta_{k}\beta^{i}L_{1})} \\ \leq 1 + \sum_ {i \geq 0, \eta_ {k} \beta^ {i} \geq \frac {\alpha_ {1}}{2 L _ {1}}} \frac {\alpha_ {1} + 2}{\alpha_ {1}} \sqrt {\beta^ {i} \eta_ {k} L _ {1}} \\ \leq 1 + \frac {\alpha_ {1} + 2}{\alpha_ {1}} \frac {1}{1 - \sqrt {\beta}} \sqrt {\eta_ {k} L _ {1}}. \\ \end{array}
$$
Furthermore, we can bound that
$$
\sum_ {k = 0} ^ {N - 1} \sqrt {\eta_ {k}} \leq \sqrt {\sigma_ {0}} + \sum_ {k = 1} ^ {N - 1} \sqrt {\eta_ {k}} \leq \sqrt {\sigma_ {0}} + \frac {1}{\sqrt {\beta}} \sum_ {k = 0} ^ {N - 2} \sqrt {\hat {\eta} _ {k}} \leq \sqrt {\sigma_ {0}} + \frac {2 (2 - \sqrt {\beta})}{\sqrt {\beta} (1 - \sqrt {\beta})} \sqrt {A _ {N - 1}}
$$
Note that  $\epsilon < f(x_{N - 1}) - f(\mathbf{x}^{*})\leq \frac{\|\mathbf{z}_{0} - \mathbf{x}^{*}\|^{2}}{2A_{N - 1}}$ . Hence, we have  $A_{N - 1}\leq \frac{\|\mathbf{z}_0 - \mathbf{x}^*\|^2}{2\epsilon}$ . Thus, we can bound the total number of matrix-vector product evaluations by
$$
\begin{array}{l} \mathsf {M V} = \sum_ {k = 0} ^ {N _ {\epsilon} - 1} \mathsf {M V} _ {k} \leq N _ {\epsilon} + \frac {\alpha_ {1} + 2}{\alpha_ {1}} \frac {1}{1 - \sqrt {\beta}} \left(\sqrt {\sigma_ {0} L _ {1}} + \frac {2 (2 - \sqrt {\beta})}{\sqrt {\beta} (1 - \sqrt {\beta})} \sqrt {\frac {L _ {1} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2}}{2 \epsilon}}\right), \\ = N _ {\epsilon} + C _ {1 1} \sqrt {\sigma_ {0} L _ {1}} + C _ {1 2} \sqrt {\frac {L _ {1} \| \mathbf {z} _ {0} - \mathbf {x} ^ {*} \| ^ {2}}{2 \epsilon}}, \\ \end{array}
$$
where we define  $C_{11} = \frac{\alpha_1 + 2}{\alpha_1}\frac{1}{1 - \sqrt{\beta}}$  and  $C_{12} = \frac{\alpha_1 + 2}{\alpha_1}\frac{1}{1 - \sqrt{\beta}}\frac{2(2 - \sqrt{\beta})}{\sqrt{\beta}(1 - \sqrt{\beta})}$ .
Lemma 25. The total number of matrix-vector product evaluations in the SEP oracle is bounded by  $\mathcal{O}\big(N_{\epsilon}^{1.25}(\log N_{\epsilon})^{0.5}\log \Big(\frac{\sqrt{d}N_{\epsilon}}{p}\Big)\big)$ .
Proof. Note that we have  $N_{t} \leq \left\lceil \frac{1}{4\sqrt{2\delta_{t}}} \log \frac{44d}{q_{t}^{2}} + \frac{1}{2} \right\rceil$  in Subroutine 4, where  $\delta_{t} = 1 / (\sqrt{t + 2} \log (t + 2))$  and  $q_{t} = p / (2.5(t + 1) \log^{2}(t + 1))$ . Thus, we have
$$
\begin{array}{l} N = \sum_ {t = 0} ^ {T - 1} N _ {t} \leq \sum_ {t = 0} ^ {T - 1} \frac {(t + 2) ^ {0 . 2 5} \log^ {0 . 5} (t + 2)}{2 \sqrt {2}} \log \frac {2 . 5 \sqrt {4 4 d} (t + 1) \log^ {2} (t + 1)}{p} (89) \\ = \mathcal {O} \left(N _ {\epsilon} ^ {1. 2 5} \sqrt {\log N _ {\epsilon}} \log \frac {\sqrt {d} N _ {\epsilon}}{p}\right). (90) \\ \end{array}
$$
F Experiments
In our experiments, we consider the logistic regression problem and minimizing the log-sum-exp function. Below we provide more details about the data generation scheme as well as the implementation of Nesterov's accelerated gradient method, BFGS, and our proposed A-QPNE algorithm.
Dataset generation. In the first experiment of logistic regression, the dataset consists of  $n$  data points  $\{(\mathbf{a}_i, y_i)\}_{i=1}^n$ , where  $\mathbf{a}_i \in \mathbb{R}^d$  is the  $i$ -th feature vector and  $\mathbf{y}_i \in \{-1, 1\}$  is its corresponding label. The labels  $\{y_i\}_{i=1}^n$  are generated by
$$
y _ {i} = \operatorname {s i g n} \left(\left\langle \mathbf {a} _ {i} ^ {*}, \mathbf {x} ^ {*} \right\rangle\right), \quad i = 1, 2, \dots , n,
$$
where  $\mathbf{a}_i^* \in \mathbb{R}^{d-1}$  and  $\mathbf{x}^* \in \mathbb{R}^{d-1}$  are the underlying true feature vector and the underlying true parameter, respectively. Moreover, each entry of  $\mathbf{a}_i^*$  and  $\mathbf{x}^*$  is drawn independently according to the standard normal distribution  $\mathcal{N}(0,1)$ . Note that the true feature vectors  $\{\mathbf{a}_i^*\}_{i=1}^n$  are not given in our dataset; instead, we generate  $\{\mathbf{a}_i\}_{i=1}^n$  by adding noises and appending an extra dimension to  $\{\mathbf{a}_i^*\}_{i=1}^n$ . Specifically, we let  $\mathbf{a}_i = [\mathbf{a}_i^* + \mathbf{n}_i + \mathbf{1}; 1]^\top \in \mathbb{R}^d$ , where  $\mathbf{n}_i \sim \mathcal{N}(0, \sigma^2\mathbf{I})$  is the i.i.d. Gaussian noise vector and  $\mathbf{1} \in \mathbb{R}^{d-1}$  denotes the all-one vector. In our experiment, we set  $n = 2,000$ ,  $d = 150$  and  $\sigma = 0.8$ .
In the second experiment of log-sum-exp function, we follow a similar procedure as [16] to generate the dataset  $\{(\mathbf{a}_i,b_i)\}_{i = 1}^n$ , where  $\mathbf{a}_i\in \mathbb{R}^d$  and  $b_{i}\in \mathbb{R}$ . First, we generate the auxiliary random vectors  $\{\hat{\mathbf{a}}_i\}_{i = 1}^n$  by sampling each entry of  $\hat{\mathbf{a}}_i$  uniformly and independently from the interval  $[-1,1]$ . Moreover, we generate  $\{b_i\}_{i = 1}^n$  independently from the standard normal distribution  $\mathcal{N}(0,1)$ . Given  $\{(\hat{\mathbf{a}}_i,b_i)\}_{i = 1}^n$ , we define an auxiliary function  $\hat{f} (\mathbf{x}) = \log (\sum_{i = 1}^{n}e^{\langle \hat{\mathbf{a}}_i,\mathbf{x}\rangle -b_i})$  and finally let  $\mathbf{a}_i = \hat{\mathbf{a}}_i - \nabla \hat{f} (0)$  for  $i = 1,\dots ,n$ . As discussed in [16], the purpose of this procedure is to ensure that  $\nabla f(0) = 0$  and thus 0 is the unique minimizer of  $f$ . In our experiment, we set  $n = d = 250$ .
NAG. We implemented a monotone variant of the Nesterov accelerated gradient method as described in [54, Section 10.7.4]. Moreover, we determine the step size using a backtracking line search scheme.
BFGS. We implemented the classical BFGS algorithm, where the step size is determined by the Moré-Thuente line search scheme using an implementation by Diane O'Leary<sup>1</sup>.
<sup>1</sup>http://www.cs.umd.edu/users/oleary/software/
Figure 3: Plots of Suboptimality gap in terms of the running time.
A-QPNE (our method). We implemented our proposed A-QPNE method following the pseudocode in Algorithm 1, where the line search scheme is given in Subroutine 1 and the Hessian approximation update is given in Subroutine 2. Moreover, the implementations of the LinearSolver oracle and the SEP oracle are given by Subroutines 3 and 4, respectively.
F.1 Additional Plots
In Fig. 3, we compare the performance of our proposed A-QNPE method with NAG and BFGS in terms of the running time. All experiments are conducted using MATLAB R2021b on a MacBook Pro with an Apple M1 chip and 16GB RAM. We observe from Fig. 3(a) that our method requires less running time than NAG due to its faster convergence, especially when we are seeking a solution of high accuracy. On the other hand, there are cases where our method is slower than NAG in terms of the running time, as shown in Fig. 3(b). This is because, in this case, the cost of gradient computation is comparable to the cost of matrix-vector product evaluation, and therefore our method incurs a higher computational cost per iteration than NAG.