SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \tilde {\eta} _ {k} \nabla f \left(\tilde {\mathbf {x}} _ {k + 1}\right) \right\| > \left(\alpha_ {1} + \alpha_ {2}\right) \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {58}
$$
Moreover, since we compute  $\tilde{\mathbf{x}}_{k + 1}$  from (12) using step size  $\tilde{\eta}_k$ , the pair  $(\tilde{\eta}_k,\tilde{\mathbf{x}}_{k + 1})$  also satisfies the condition in (10), which means
$$
\left\| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \tilde {\eta} _ {k} \left(\nabla f (\mathbf {y} _ {k}) + \mathbf {B} _ {k} \left(\tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}\right)\right) \right\| \leq \alpha_ {1} \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {59}
$$
Hence, by using the triangle inequality, we can combine (58) and (59) to get
$$
\begin{array}{l} \tilde {\eta} _ {k} \left\| \nabla f \left(\tilde {\mathbf {x}} _ {k + 1}\right) - \nabla f \left(\mathbf {y} _ {k}\right) - \mathbf {B} _ {k} \left(\tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}\right) \right\| \\ \geq \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \tilde {\eta} _ {k} \nabla f (\tilde {\mathbf {x}} _ {k + 1}) \| - \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \tilde {\eta} _ {k} (\nabla f (\mathbf {y} _ {k}) + \mathbf {B} _ {k} (\tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k})) \| \\ > (\alpha_ {1} + \alpha_ {2}) \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| - \alpha_ {1} \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \\ = \alpha_ {2} \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|, \\ \end{array}
$$
which implies that
$$
\hat {\eta} _ {k} = \beta \tilde {\eta} _ {k} > \frac {\alpha_ {2} \beta \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|}{\| \nabla f (\tilde {\mathbf {x}} _ {k + 1}) - \nabla f (\mathbf {y} _ {k}) - \mathbf {B} _ {k} (\tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k}) \|}.
$$
This proves the first inequality in (13).
To show the second inequality in (13), first note that  $\tilde{\mathbf{x}}_{k + 1}$  and  $\hat{\mathbf{x}}_{k + 1}$  are the inexact solutions of the linear system of equations
$$
\left(\mathbf {I} + \tilde {\eta} _ {k} \mathbf {B} _ {k}\right) \left(\mathbf {x} - \mathbf {y} _ {k}\right) = - \tilde {\eta} _ {k} \mathbf {g} _ {k} \quad \text {a n d} \quad \left(\mathbf {I} + \hat {\eta} _ {k} \mathbf {B} _ {k}\right) \left(\mathbf {x} - \mathbf {y} _ {k}\right) = - \hat {\eta} _ {k} \mathbf {g} _ {k},
$$
respectively. Let  $\tilde{\mathbf{x}}_{k + 1}^{*}$  and  $\hat{\mathbf{x}}_{k + 1}^{*}$  be the exact solutions of the above linear systems, that is,  $\tilde{\mathbf{x}}_{k + 1}^{*} = \mathbf{y}_k - \hat{\eta}_k(\mathbf{I} + \hat{\eta}_k\mathbf{B}_k)^{-1}\mathbf{g}_k$  and  $\hat{\mathbf{x}}_{k + 1}^{*} = \mathbf{y}_k - \hat{\eta}_k(\mathbf{I} + \hat{\eta}_k\mathbf{B}_k)^{-1}\mathbf{g}_k$ . We first establish the following inequality between  $\| \tilde{\mathbf{x}}_{k + 1}^{*} - \mathbf{y}_{k}\|$  and  $\| \hat{\mathbf{x}}_{k + 1}^{*} - \mathbf{y}_{k}\|$ :
$$
\left\| \tilde {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \right\| \leq \frac {1}{\beta} \left\| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \right\|. \tag {60}
$$
This follows from
$$
\| \tilde {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \| = \| \tilde {\eta} _ {k} (\mathbf {I} + \tilde {\eta} _ {k} \mathbf {B} _ {k}) ^ {- 1} \mathbf {g} _ {k} \| \leq \tilde {\eta} _ {k} \| (\mathbf {I} + \hat {\eta} _ {k} \mathbf {B} _ {k}) ^ {- 1} \mathbf {g} _ {k} \| = \frac {\tilde {\eta} _ {k}}{\hat {\eta} _ {k}} \| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \| = \frac {1}{\beta} \| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \|,
$$
where we used the fact that  $(\mathbf{I} + \tilde{\eta}_k\mathbf{B}_k)^{-1}\preceq (\mathbf{I} + \hat{\eta}_k\mathbf{B}_k)^{-1}$  in the first inequality. Furthermore, we can show that
$$
\left. \left(1 - \alpha_ {1}\right) \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \leq \| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \| \leq \left(1 + \alpha_ {1}\right) \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\|, \tag {61}
$$
$$
\left(1 - \alpha_ {1}\right) \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \leq \| \tilde {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \| \leq \left(1 + \alpha_ {1}\right) \| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|. \tag {62}
$$
We will only prove (61) in the following, as (62) can be proved similarly. Note that since  $(\hat{\eta}_k,\hat{\mathbf{x}}_{k + 1})$  satisfies the condition in (10), we can write
$$
\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} + \hat {\eta} _ {k} (\mathbf {g} _ {k} + \mathbf {B} _ {k} (\hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k})) \| = \| (\mathbf {I} + \hat {\eta} _ {k} \mathbf {B} _ {k}) (\hat {\mathbf {x}} _ {k + 1} - \hat {\mathbf {x}} _ {k + 1} ^ {*}) \| \leq \alpha_ {1} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|.
$$
Moreover, since  $\mathbf{B}_k\succeq 0$ , we have  $\| \hat{\mathbf{x}}_{k + 1} - \hat{\mathbf{x}}_{k + 1}^*\| \leq \| (\mathbf{I} + \hat{\eta}_k\mathbf{B}_k)(\hat{\mathbf{x}}_{k + 1} - \hat{\mathbf{x}}_{k + 1}^*)\| \leq \alpha_1\| \hat{\mathbf{x}}_{k + 1} - \mathbf{y}_k\|$ . Thus, by the triangle inequality, we obtain
$$
\left\| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \right\| \leq \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| + \left\| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \hat {\mathbf {x}} _ {k + 1} \right\| \leq (1 + \alpha_ {1}) \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\|.
$$
$$
\left\| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \right\| \geq \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\| - \left\| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \hat {\mathbf {x}} _ {k + 1} \right\| \geq (1 - \alpha_ {1}) \left\| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \right\|.
$$
which proves (61). Finally, by combining (60), (61) and (62), we conclude that
$$
\| \tilde {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \| \leq \frac {1}{1 - \alpha_ {1}} \| \tilde {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \| \leq \frac {1}{(1 - \alpha_ {1}) \beta} \| \hat {\mathbf {x}} _ {k + 1} ^ {*} - \mathbf {y} _ {k} \| \leq \frac {1 + \alpha_ {1}}{(1 - \alpha_ {1}) \beta} \| \hat {\mathbf {x}} _ {k + 1} - \mathbf {y} _ {k} \|.
$$
C Hessian Approximation Update
In this section, we first prove Lemma 2 in Section C.1 and remark on the computational cost of Euclidean projection in Section C.2. Then we present a general online learning algorithm using an approximate separation oracle in Section C.3 and fully describe our Hessian approximation update in Section C.4.
C.1 Proof of Lemma 2
We decompose the sum  $\sum_{k=0}^{N-1} \frac{1}{\vec{\eta}_k^2}$  as
$$
\sum_ {k = 0} ^ {N - 1} \frac {1}{\hat {\eta} _ {k} ^ {2}} = \frac {1}{\hat {\eta} _ {0} ^ {2}} + \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} \tag {63}
$$
Recall that we have  $\hat{\eta}_k = \eta_k$  for  $k \notin \mathcal{B}$ . Hence, we can further bound the last term by
$$
\begin{array}{l} \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} = \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \frac {1}{\eta_ {k} ^ {2}} \leq \sum_ {k = 1} ^ {N - 1} \frac {1}{\eta_ {k} ^ {2}} \\ = \frac {1}{\eta_ {1} ^ {2}} + \sum_ {1 \leq k \leq N - 2, k \in \mathcal {B}} \frac {1}{\eta_ {k + 1} ^ {2}} + \sum_ {1 \leq k \leq N - 2, k \notin \mathcal {B}} \frac {1}{\eta_ {k + 1} ^ {2}}. \\ \end{array}
$$
Recall that we have  $\eta_{k + 1} = \hat{\eta}_k$  if  $k\in \mathcal{B}$  and  $\eta_{k + 1} = \hat{\eta}_k / \beta$  otherwise. Hence, we further have
$$
\begin{array}{l} \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} \leq \frac {1}{\eta_ {1} ^ {2}} + \sum_ {1 \leq k \leq N - 2, k \in \mathcal {B}} \frac {1}{\eta_ {k + 1} ^ {2}} + \sum_ {1 \leq k \leq N - 2, k \notin \mathcal {B}} \frac {1}{\eta_ {k + 1} ^ {2}} \\ = \frac {1}{\eta_ {1} ^ {2}} + \sum_ {1 \leq k \leq N - 2, k \in \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} + \sum_ {1 \leq k \leq N - 2, k \notin \mathcal {B}} \frac {\beta^ {2}}{\hat {\eta} _ {k} ^ {2}} \\ \leq \frac {1}{\eta_ {1} ^ {2}} + \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} + \sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \frac {\beta^ {2}}{\hat {\eta} _ {k} ^ {2}}. \\ \end{array}
$$
By moving the last term to the left-hand side and dividing both sides by  $1 - \beta^2$ , we obtain
$$
\sum_ {1 \leq k \leq N - 1, k \notin \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}} \leq \frac {1}{1 - \beta^ {2}} \left(\frac {1}{\eta_ {1} ^ {2}} + \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}}\right). \tag {64}
$$
Furthermore, since  $\eta_{1} \geq \hat{\eta}_{0}$ , we have  $\frac{1}{\eta_1^2} \leq \frac{1}{\hat{\eta}_0^2}$ . Hence, by combining (63) and (64), we get
$$
\sum_ {k = 0} ^ {N - 1} \frac {1}{\hat {\eta} _ {k} ^ {2}} \leq \frac {2 - \beta^ {2}}{1 - \beta^ {2}} \left(\frac {1}{\hat {\eta} _ {0} ^ {2}} + \sum_ {1 \leq k \leq N - 1, k \in \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}}\right) \leq \frac {2 - \beta^ {2}}{(1 - \beta^ {2}) \sigma_ {0} ^ {2}} + \frac {2 - \beta^ {2}}{1 - \beta^ {2}} \sum_ {0 \leq k \leq N - 1, k \in \mathcal {B}} \frac {1}{\hat {\eta} _ {k} ^ {2}}, \tag {65}
$$
where in the last inequality we used the fact that  $\hat{\eta}_k = \sigma_0$  if  $0 \notin \mathcal{B}$ . Finally, (15) follows from Lemma 1 and (65).
C.2 The Computational Cost of Euclidean Projection
Recall that  $\mathcal{Z} \triangleq \{\mathbf{B} \in \mathbb{S}_+^d : 0 \preceq \mathbf{B} \preceq L_1\mathbf{I}\}$ . As described in [35, Section D.1], the Euclidean projection on  $\mathcal{Z}$  has a closed form solution. Specifically, Given the input  $\mathbf{A} \in \mathbb{S}^d$ , we first need to perform the eigendecomposition  $\mathbf{A} = \mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^\top$ , where  $\mathbf{V}$  is an orthogonal matrix and  $\boldsymbol{\Lambda} = \mathrm{diag}(\lambda_1, \dots, \lambda_d)$  is a diagonal matrix. Then the Euclidean projection of  $\mathbf{A}$  onto  $\mathcal{Z}$  is given by  $\mathbf{V}\hat{\boldsymbol{\Lambda}}\mathbf{V}^\top$ , where  $\hat{\boldsymbol{\Lambda}}$  is a diagonal matrix with the diagonals being  $\hat{\lambda}_k = \min\{L_1, \max\{0, \lambda_k\}\}$  for  $1 \leq k \leq d$ . Since the eigendecomposition requires  $\mathcal{O}(d^3)$  arithmetic operations in general, the cost of computing the Euclidean projection can be prohibitive.
Algorithm 2 Projection-Free Online Learning
1: Input: Initial point  $\mathbf{w}_0\in \mathcal{B}_R(0)$  , step size  $\rho >0$ $\delta >0$    
2: for  $t = 0,1,\ldots T - 1$  do   
3: Query the oracle  $(\gamma_{t},\mathbf{s}_{t})\gets \mathrm{SEP}(\mathbf{w}_{t};\delta_{t})$    
4: if  $\gamma_t\leq 1$  then # Case I: we have  $\mathbf{w}_t\in \mathcal{C}$    
5: Set  $\mathbf{x}_t\gets \mathbf{w}_t$  and play the action  $\mathbf{x}_t$    
6: Receive the loss  $\ell_t(\mathbf{x}_t)$  and the gradient  $\mathbf{g}_t = \nabla \ell_t(\mathbf{x}_t)$    
7: Set  $\tilde{\mathbf{g}}_t\gets \mathbf{g}_t$    
8: else # Case II: we have  $\mathbf{w}_t / \gamma_t\in \mathcal{C}$    
9: Set  $\mathbf{x}_t\gets \mathbf{w}_t / \gamma_t$  and play the action  $\mathbf{x}_t$    
10: Receive the loss  $\ell_t(\mathbf{x}_t)$  and the gradient  $\mathbf{g}_t = \nabla \ell_t(\mathbf{x}_t)$    
11: Set  $\tilde{\mathbf{g}}_t\gets \mathbf{g}_t + \max \{0, - \langle \mathbf{g}_t,\mathbf{x}_t\rangle \} \mathbf{s}_t$    
12: end if   
13: Update  $\mathbf{w}_{t + 1}\leftarrow \frac{R}{\max\{\|\mathbf{w}_t - \rho\tilde{\mathbf{g}}_t\|_2,R\}} (\mathbf{w}_t - \rho \tilde{\mathbf{g}}_t)\quad \# Euclidean projection onto \mathcal{B}_R(0)$    
14: end for
C.3 Online Learning with an Approximate Separation Oracle
To set the stage for our Hessian approximation matrix update, we first describe a projection-free online learning algorithm in a general setup. Specifically, the online learning protocol is as follows: For rounds  $t = 0,1,\ldots ,T - 1$ , a learner chooses an action  $\mathbf{x}_t\in \mathcal{C}$  from a convex set  $\mathcal{C}$  and then observes a loss function  $\ell_t:\mathbb{R}^n\to \mathbb{R}$ . We measure the performance of an online learning algorithm by the dynamic regret [49, 52] defined by
$$
\mathrm {D} - \operatorname {R e g} _ {T} (\mathbf {u} _ {1}, \dots , \mathbf {u} _ {T - 1}) \triangleq \sum_ {t = 0} ^ {T - 1} \ell_ {t} (\mathbf {x} _ {t}) - \sum_ {t = 0} ^ {T - 1} \ell_ {t} (\mathbf {u} _ {t}),
$$
where  $\{\mathbf{u}_t\}_{t=1}^T$  is a sequence of comparators. Moreover, we assume that the convex set  $\mathcal{C}$  is contained in the Euclidean ball  $\mathcal{B}_R(0)$  for some  $R > 0$ , and we assume  $0 \in \mathcal{C}$  without loss of generality.
Most existing online learning algorithms are projection-based, that is, they require computing the Euclidean projection on the action set  $\mathcal{C}$ . However, as we have seen in Section C.2, computing the projection is computationally costly in our setting. Inspired by the work in [50], we will describe an online learning algorithm that relies on an approximate separation oracle defined in Definition 3.
Definition 3. The oracle  $\mathsf{SEP}(\mathbf{w};\delta)$  takes  $\mathbf{w}\in \mathcal{B}_R(0)$  and  $\delta >0$  as input and returns a scalar  $\gamma >0$  and a vector  $\mathbf{s}\in \mathbb{R}^n$  with one of the following possible outcomes:
- Case I:  $\gamma \leq 1$  which implies that  $\mathbf{w} \in \mathcal{C}$ ;
- Case II:  $\gamma > 1$  which implies that  $\mathbf{w} / \gamma \in \mathcal{C}$  and  $\langle \mathbf{s}, \mathbf{w} - \mathbf{x} \rangle \geq \gamma - 1 - \delta \quad \forall \mathbf{x} \in \mathcal{C}$ .
To sum up, the oracle  $\mathrm{SEP}(\mathbf{w};\delta)$  has two possible outcomes: it either certifies that  $\mathbf{w}$  is feasible, i.e.,  $\mathbf{w}\in \mathcal{C}$ , or it produces a scaled version of  $\mathbf{w}$  that is in  $\mathcal{C}$  and gives an approximate separating hyperplane between  $\mathbf{w}$  and the set  $\mathcal{C}$ .
The full algorithm is shown in Algorithm 2. The key idea here is to introduce surrogate loss functions  $\tilde{\ell}_t(\mathbf{w}) = \langle \tilde{\mathbf{g}}_t,\mathbf{w}\rangle$  on the larger set  $\mathcal{B}_R(0)$  for  $0\leq t\leq T - 1$ , where  $\tilde{\mathbf{g}}_t$  is the surrogate gradient to be defined later. On a high level, we will run online projected gradient descent with  $\tilde{\ell}_t(\mathbf{w})$  to update the auxiliary iterates  $\{\mathbf{w}_t\}_{t\geq 0}$  (note that the projection on  $\mathcal{B}_R(0)$  is easy to compute), and then produce the actions  $\{\mathbf{x}_t\}_{t\geq 0}$  for the original problem by calling the  $\mathrm{SEP}(\mathbf{w}_t;\delta)$  oracle in Definition 3. The following lemma shows that the immediate regret  $\tilde{\ell}_t(\mathbf{w}_t) - \tilde{\ell}_t(\mathbf{x})$  can serve as an upper bound on  $\ell_t(\mathbf{x}_t) - \ell_t(\mathbf{x})$  for any  $\mathbf{x}\in \mathcal{C}$ .
Lemma 10. Let  $\{\mathbf{x}_t\}_{t=0}^{T-1}$  be the iterates generated by Algorithm 2. Then we have  $\mathbf{x}_t \in \mathcal{C}$  for  $t = 0, 1, \ldots, T-1$ . Also, for any  $\mathbf{x} \in \mathcal{C}$ , we have
$$
\begin{array}{l} \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} \rangle \leq \langle \tilde {\mathbf {g}} _ {t}, \mathbf {w} _ {t} - \mathbf {x} \rangle + \max  \{0, - \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle \} \delta_ {t} (66) \\ \leq \frac {1}{2 \rho} \| \mathbf {w} _ {t} - \mathbf {x} \| _ {2} ^ {2} - \frac {1}{2 \rho} \| \mathbf {w} _ {t + 1} - \mathbf {x} \| _ {2} ^ {2} + \frac {\rho}{2} \| \tilde {\mathbf {g}} _ {t} \| _ {2} ^ {2} + \max  \{0, - \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle \} \delta_ {t}, (67) \\ \end{array}
$$
$$
\| \tilde {\mathbf {g}} _ {t} \| \leq \| \mathbf {g} _ {t} \| + | \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle | \| \mathbf {s} _ {t} \|. \tag {68}
$$
Subroutine 2 Online Learning Guided Hessian Approximation Update
1: Input: Initial matrix  $\mathbf{B}_0\in \mathbb{S}^d$  s.t.  $0\preceq \mathbf{B}_0\preceq L_1\mathbf{I}$  , step size  $\rho >0,\delta >0,\{q_t\}_{t = 1}^{T - 1}$    
2: Initialize: set  $\mathbf{W}_0\gets \frac{2}{L_1} (\mathbf{B}_0 - \frac{L_1}{2}\mathbf{I})$ $\mathbf{G}_0\gets \frac{2}{L_1}\nabla \ell_0(\mathbf{B}_0)$  and  $\tilde{\mathbf{G}}_0\gets \mathbf{G}_0$    
3: for  $t = 1,\dots ,T - 1$  do   
4: Query the oracle  $(\gamma_{t},\mathbf{S}_{t})\gets \mathrm{SEP}(\mathbf{W}_{t};\delta_{t},q_{t})$    
5: if  $\gamma_{t}\leq 1$  then # Case I   
6: Set  $\hat{\mathbf{B}}_t\gets \mathbf{W}_t$  and  $\mathbf{B}_t\gets \frac{L_1}{2}\hat{\mathbf{B}}_t + \frac{L_1}{2}\mathbf{I}$    
7: Set  $\mathbf{G}_t\gets \frac{2}{L_1}\nabla \ell_t(\mathbf{B}_t)$  and  $\tilde{\mathbf{G}}_t\gets \mathbf{G}_t$    
8: else # Case II   
9: Set  $\hat{\mathbf{B}}_t\gets \mathbf{W}_t / \gamma_t$  and  $\mathbf{B}_t\gets \frac{L_1}{2}\hat{\mathbf{B}}_t + \frac{L_1}{2}\mathbf{I}$    
10: Set  $\mathbf{G}_t\gets \frac{2}{L_1}\nabla \ell_t(\mathbf{B}_t)$  and  $\tilde{\mathbf{G}}_t\gets \mathbf{G}_t + \max \{0, - \langle \mathbf{G}_t,\mathbf{B}_t\rangle \} \mathbf{S}_t$    
11: end if   
12: Update  $\mathbf{W}_{t + 1}\gets \frac{\sqrt{d}}{\max\{\sqrt{d},\|\mathbf{W}_t - \rho\tilde{\mathbf{G}}_t\|_F\}} (\mathbf{W}_t - \rho \tilde{\mathbf{G}}_t)\quad \# Euclidean projection onto B_{\sqrt{d}}(0)$    
13: end for
Proof. By the definition of SEP in Definition 3, we can see that  $\mathbf{x}_t \in \mathcal{C}$  for all  $t = 1, \ldots, T$ . We now show that both (66) and (68) hold. We distinguish two cases depending on the outcomes of  $\mathrm{SEP}(\mathbf{w}_t; \delta_t)$ .
- If  $\gamma_t \leq 1$ , then we have  $\mathbf{x}_t = \mathbf{w}_t$  and  $\tilde{\mathbf{g}}_t = \mathbf{g}_t$ . In this case, (66) and (68) trivially hold.
- If  $\gamma_t > 1$ , then  $\mathbf{x}_t = \mathbf{w}_t / \gamma_t$  and  $\tilde{\mathbf{g}}_t = \mathbf{g}_t + \max \{0, -\langle \mathbf{g}_t, \mathbf{x}_t \rangle\} \mathbf{s}_t$ . We can then write
$$
\begin{array}{l} \langle \tilde {\mathbf {g}} _ {t}, \mathbf {w} _ {t} - \mathbf {x} \rangle = \langle \mathbf {g} _ {t} + \max  \{0, - \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle \} \mathbf {s} _ {t}, \mathbf {w} _ {t} - \mathbf {x} \rangle \\ = \left\langle \mathbf {g} _ {t}, \gamma_ {t} \mathbf {x} _ {t} - \mathbf {x} \right\rangle + \max  \{0, - \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \right\rangle \} \left\langle \mathbf {s} _ {t}, \mathbf {w} _ {t} - \mathbf {x} \right\rangle \\ \geq \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} \rangle + (\gamma_ {t} - 1) \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle + \max  \{0, - \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle \} (\gamma_ {t} - 1 - \delta_ {t}) \\ = \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} \right\rangle - \max  \{0, - \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \right\rangle \} \delta_ {t} + (\gamma_ {t} - 1) \max  \{0, \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \right\rangle \} \\ \geq \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} - \mathbf {x} \right\rangle - \max  \{0, - \left\langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \right\rangle \} \delta_ {t}, \\ \end{array}
$$
which leads to (66) after rearranging. Also, by the triangle inequality we obtain
$$
\left\| \tilde {\mathbf {g}} _ {t} \right\| \leq \left\| \mathbf {g} _ {t} \right\| + \max  \left\{0, - \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle \right\} \left\| \mathbf {s} _ {t} \right\| \leq \left\| \mathbf {g} _ {t} \right\| + | \langle \mathbf {g} _ {t}, \mathbf {x} _ {t} \rangle | \| \mathbf {s} _ {t} \|,
$$
which proves (68).
Finally, from the update rule of  $\mathbf{w}_{t + 1}$ , for any  $\mathbf{x} \in \mathcal{C} \subset \mathcal{B}_R(0)$  we have  $\langle \mathbf{w}_t - \rho \tilde{\mathbf{g}}_t - \mathbf{w}_{t + 1}, \mathbf{w}_{t + 1} - \mathbf{x} \rangle \geq 0$ , which further implies that
$$
\begin{array}{l} \langle \tilde {\mathbf {g}} _ {t}, \mathbf {w} _ {t} - \mathbf {x} \rangle \leq \langle \tilde {\mathbf {g}} _ {t}, \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \rangle + \frac {1}{\rho} \langle \mathbf {w} _ {t} - \mathbf {w} _ {t + 1}, \mathbf {w} _ {t + 1} - \mathbf {x} \rangle (69) \\ = \left\langle \tilde {\mathbf {g}} _ {t}, \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \right\rangle + \frac {1}{2 \rho} \| \mathbf {w} _ {t} - \mathbf {x} \| _ {2} ^ {2} - \frac {1}{2 \rho} \| \mathbf {w} _ {t + 1} - \mathbf {x} \| _ {2} ^ {2} - \frac {1}{2 \rho} \| \mathbf {w} _ {t} - \mathbf {w} _ {t + 1} \| _ {2} ^ {2} (70) \\ \leq \frac {1}{2 \rho} \| \mathbf {w} _ {t} - \mathbf {x} \| _ {2} ^ {2} - \frac {1}{2 \rho} \| \mathbf {w} _ {t + 1} - \mathbf {x} \| _ {2} ^ {2} + \frac {\rho}{2} \| \tilde {\mathbf {g}} _ {t} \| _ {2} ^ {2}. (71) \\ \end{array}
$$
Combining (66) and (71) leads to (67).
C.4 Projection-free Hessian Approximation Update
Now we are ready to describe our Hessian approximation matrix update, which is an specific instantiation of the general projection-free online learning algorithm shown in Algorithm 2. In particular, we only need to specify the convex set  $\mathcal{C}$  as well as the SEP oracle.
Note that we have  $\mathcal{Z} = \{\mathbf{B}\in \mathbb{S}_+^d:0\preceq \mathbf{B}\preceq L_1\mathbf{I}\}$  in our online learning problem in Section 3.2. Since the projection-free scheme in Subroutine 2 requires the set  $\mathcal{C}$  to contain the origin, we consider the transform  $\hat{\mathbf{B}}\triangleq \frac{2}{L_1}\left(\mathbf{B} - \frac{L_1}{2}\mathbf{I}\right)$  and let  $\mathcal{C} = \hat{\mathcal{Z}}\triangleq \{\hat{\mathbf{B}}\in \mathbb{S}^d: - \mathbf{I}\preceq \hat{\mathbf{B}}\preceq \mathbf{I}\} = \{\hat{\mathbf{B}}\in \mathbb{S}^d:\| \hat{\mathbf{B}}\|_{\mathrm{op}}\leq 1\}$ . We note that  $0\in \hat{\mathcal{Z}}$  and  $\hat{\mathcal{Z}}\subset \mathcal{B}_{\sqrt{d}}(0) = \{\mathbf{W}\in \mathbb{S}^d:\| \mathbf{W}\| _F\leq \sqrt{d}\}$ . Moreover, we can see that the approximate separation oracle  $\mathrm{SEP}(\mathbf{W};\delta ,q)$  defined in Definition 2 corresponds to a stochastic version of the oracle in Definition 3. The full algorithm is described in Subroutine 2 and we defer the specific implementation details of  $\mathrm{SEP}(\mathbf{W};\delta ,q)$  to Section E.2.
D Proof of Theorem 1
Regarding the choices of the hyper-parameters, we consider Algorithm 1 with the line search scheme in Subroutine 1, where  $\alpha_{1},\alpha_{2}\in (0,1)$  with  $\alpha_{1} + \alpha_{2} < 1$  and  $\beta \in (0,1)$ , and with the Hessian approximation update in Subroutine 2, where  $\rho = \frac{1}{128}$ ,  $q_{t} = p / 2.5(t + 1)\log^{2}(t + 1)$  for  $t\geq 1$ , and  $\delta_t = 1 / (\sqrt{t + 2}\ln (t + 2))$  for  $t\geq 0$ . In the following, we first provide a proof sketch of Theorem 1. The complete proofs of the lemmas shown below will be provided in the subsequent sections.
Proof Sketch. To begin with, throughout the proof, we assume that every call of the SEP oracle in Definition 2 is successful during the execution of Algorithm 1. Indeed, by using the union bound, we can bound the failure probability by  $\sum_{t=1}^{T-1} q_t \leq \frac{p}{2.5} \sum_{t=2}^{\infty} \frac{1}{t \log^2 t} \leq p$ . In particular, we note that Subroutine 2 ensures that  $0 \preceq \mathbf{B}_k \preceq L_1 \mathbf{I}$  for any  $k \geq 0$ .
We first prove Part (a) of Theorem 1, which relies on the following lemma.
Lemma 11. For  $k \in \mathcal{B}$ , we have  $\ell_k(\mathbf{B}_k) \triangleq \frac{\|\mathbf{w}_k - \mathbf{B}_k\mathbf{s}_k\|^2}{\|\mathbf{s}_k\|^2} \leq L_1^2$ .
We combine Lemma 2 and Lemma 11 to derive