SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $M_{t}(W_{*}) = \frac{\lambda}{2} I_{Kd} + \sum_{s=1}^{t} \nabla \sigma(W_{*}\mathbf{x}_{s}) \otimes \mathbf{x}_{s}\mathbf{x}_{s}^{\top}$ . Then, we can further bound the last line of the above displayed equations by
$$
\begin{array}{l} (1 + 2 S) \sum_ {t = 1} ^ {T} \operatorname {T r} \left(\left(\nabla \boldsymbol {\sigma} \left(W _ {*} \mathbf {x} _ {t}\right) \otimes \mathbf {x} _ {t} \mathbf {x} _ {t} ^ {\top}\right) H _ {t} ^ {- 1} \left(W _ {*}\right)\right) \\ = (1 + 2 S) \sum_ {t = 1} ^ {T} \operatorname {T r} \left(M _ {t} \left(W _ {*}\right) - M _ {t - 1} \left(W _ {*}\right)\right) H _ {t} ^ {- 1} \left(W _ {*}\right)) \\ \leq (1 + 2 S) \sum_ {t = 1} ^ {T} \operatorname {T r} \left(M _ {t} \left(W _ {*}\right) - M _ {t - 1} \left(W _ {*}\right)\right) M _ {t} ^ {- 1} \left(W _ {*}\right)) \\ \leq (1 + 2 S) \sum_ {t = 1} ^ {T} \log \frac {\left| M _ {t} \left(W _ {*}\right) \right|}{\left| M _ {t - 1} \left(W _ {*}\right) \right|} \\ \leq (1 + 2 S) K d \ln \left(1 + \frac {T}{2 \lambda}\right) \\ \end{array}
$$
The first inequality is by the fact  $H_{t}(W_{*}) \succcurlyeq M_{t}(W_{*})$  under the condition  $\lambda \geq 2$ . The last second inequality is due to Lemma 4.5 of [16].
Lemma 11. Let  $\sigma_k: \mathbf{z} \mapsto [\sigma(\mathbf{x})]_k \in \mathbb{R}$  by the  $k$ -th output of the vector-valued function  $\sigma(\mathbf{z})$ . Then, for any  $\mathbf{z} \in \mathbb{R}^K$ , we have  $\nabla^2 \sigma_k(\mathbf{z}) \leq 3I_K$  for any  $k \in [K]$ .
Proof of Lemma 11. For any  $\mathbf{z} \in \mathbb{R}^K$ , we have
$$
\begin{array}{l} 3 I _ {K} - \nabla^ {2} \sigma_ {k} (\mathbf {z}) \\ = 3 I _ {K} + \sigma_ {k} (\mathbf {z}) \cdot \left(\operatorname {d i a g} (\boldsymbol {\sigma} (\mathbf {z})) + \mathbf {e} _ {k} \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} + \boldsymbol {\sigma} (\mathbf {z}) \mathbf {e} _ {k} ^ {\top}\right) - 2 \sigma_ {k} \cdot (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} - \sigma_ {k} (\mathbf {z}) E _ {k} \\ \succ 3 I _ {K} - 2 \sigma_ {k} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} + \sigma_ {k} (\mathbf {z}) \left(\mathbf {e} _ {k} \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} + \boldsymbol {\sigma} (\mathbf {z}) \mathbf {e} _ {k} ^ {\top} - E _ {k}\right) (28) \\ = 3 I _ {K} - \sigma_ {k} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) \boldsymbol {\sigma} (\mathbf {z}) ^ {\top} - \sigma_ {k} (\mathbf {z}) \left(\boldsymbol {\sigma} (\mathbf {z}) - \mathbf {e} _ {k}\right) \left(\boldsymbol {\sigma} (\mathbf {z}) - \mathbf {e} _ {k}\right) ^ {\top} \succcurlyeq 0 (29) \\ \end{array}
$$
where we denote by  $E_{k}$  as the matrix where the entry at the  $(k,k)$ -th position is 1, and all other entries are 0. In the above, the first inequality is due to the definition of  $\sigma_{k}$ . The second inequality holds since  $\mathrm{diag}(\pmb{\sigma}(\mathbf{z}))$  is semi-positive defined matrix. The last inequality is a consequence of the fact that the maximum eigenvalue of  $\sigma_{k}(\mathbf{z})\pmb{\sigma}(\mathbf{z})\pmb{\sigma}(\mathbf{z})^{\top}$  is bounded by  $\sigma_{k}(\mathbf{z})(\pmb{\sigma}(\mathbf{z}) - \mathbf{e}_{k})(\pmb{\sigma}(\mathbf{z}) - \mathbf{e}_{k})^{\top}$  is less than 1. The maximum eigenvalue of  $\sigma_{k}(\mathbf{z})(\pmb{\sigma}(\mathbf{z}) - \mathbf{e}_{k})(\pmb{\sigma}(\mathbf{z}) - \mathbf{e}_{k})^{\top}$  is bounded by 2.
C Omitted Proofs for Section 3
This section presents the omitted details for Section 3. To simplify the notation, we exclude the superscript OL in all the proofs within this section.
C.1 Proof of Theorem 3
This section provides the proof of Theorem 3. We will first provide the main proof of Theorem 3 and the prove the technical lemma used in the main proof.
C.1.1 Main Proof
Proof of Theorem 3. The proof shares the similarity with the online-to-confidence-set conversion technique [47, 14], where the estimation error of the online estimator  $W_{t}^{\mathrm{OL}}$  is bounded by the regret. However, we have introduced a novel algorithm and analysis ingredient to achieve jointly statistical and computation efficiency.
The following lemma provides the estimation error analysis. Drawing inspiration from the modern analysis of OMD [23], we explicitly extract the negative term  $-\sum_{s=1}^{t}\|\overrightarrow{W}_{s+1}-\overrightarrow{W}_{s}\|_{H_{s}}^{2}$  in the upper bound, which is pivotal for our subsequent theoretical analysis and the algorithm design.
Lemma 12. Under Assumptions 1 and 3, we consider the estimator update rule
$$
\overrightarrow {W} _ {t + 1} = \underset {W \in \mathcal {W}} {\arg \min } \left\{\widetilde {\ell} _ {t} (W) + \frac {1}{2 \eta} \| \overrightarrow {W} - \overrightarrow {W} _ {t} \| _ {H _ {t}} ^ {2} \right\},
$$
where  $\widetilde{\ell}_t(W) = \langle \nabla \ell_t(\overrightarrow{W}_t),\overrightarrow{W}\rangle +\frac{1}{2}\| \overrightarrow{W} -\overrightarrow{W}_t\|_{\nabla^2\ell_t(\overrightarrow{W}_t)}^2$  and  $H_{t} = \lambda I_{Kd} + \sum_{s = 1}^{t - 1}\nabla^{2}\ell_{s}(W_{s + 1})\otimes$
$\mathbf{x}_s\mathbf{x}_s^\top$ . Then, letting  $\alpha = \ln(K + 1) + 2(S + 1)$  and  $\lambda > 0$ , we have
$$
\begin{array}{l} \left\| \overrightarrow {W} _ {t + 1} - \overrightarrow {W} _ {*} \right\| _ {H _ {t + 1}} ^ {2} \leq \alpha \left(\sum_ {s = 1} ^ {t} \ell_ {s} \left(W _ {*}\right) - \sum_ {s = 1} ^ {t} \ell_ {s} \left(W _ {s + 1}\right)\right) + 4 \lambda S ^ {2} \\ + \sum_ {s = 1} ^ {t} \sqrt {6} \alpha S \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {I _ {K} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}} ^ {2} - \sum_ {s = 1} ^ {t} \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {H _ {s}} ^ {2}, \\ \end{array}
$$
when the step size is set as  $\eta = \alpha /2$
Then, we focus on the first term of the right hand side. Inspired by the previous studies on binary logistic bandit [10], we decompose the regret into two parts by inserting an intermediate decision.
$$
\begin{array}{l} \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1}) \\ = \underbrace {\sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s} , y _ {s})} _ {\text {t e r m (A)}} + \underbrace {\sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s} , y _ {s}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1})} _ {\text {t e r m (B)}}, \\ \end{array}
$$
where the  $\widetilde{\mathbf{z}}_s$  is an aggregating forecaster for logistic loss defined by  $\widetilde{\mathbf{z}}_s = \pmb{\sigma}^+(\mathbb{E}_{W \sim P_s}[\pmb{\sigma}(W\mathbf{x}_s)])$  and  $P_s = \mathcal{N}(\overrightarrow{W}_s, (1 + cH_s^{-1}))$  is the Gaussian distribution with mean  $\overrightarrow{W}_s$  and covariance matrix  $cH_s^{-1}$ , where  $c > 0$  is a constant to be specified. In above,  $\pmb{\sigma}^{+}:\Delta \mapsto \mathbb{R}^{K}$  is a pseudo-inverse function of  $\pmb{\sigma}(\cdot)$  whose  $k$ -th output is  $[\pmb{\sigma}^{+}(\mathbf{p})]_k = \log(p_k / (1 - \| \mathbf{p} \|_1))$  for any  $\mathbf{p} \in \{\mathbf{q} \in [0,1]^K | \| \mathbf{q} \|_1 < 1\}$ . We bound the above two terms respectively. First, we show that the term (A) is bounded by  $\mathcal{O}\big(\log^2 t\big)$  with high probability.
Lemma 13. Let  $\delta \in (0,1]$ . Under Assumptions 1 and 3, we have
$$
\operatorname * {P r} \left[ \forall t \geq 1, \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s}, y _ {s}) \leq \gamma_ {t} ^ {\mathtt {A}} (\delta) \right] \geq 1 - \delta ,
$$
where the confidence radius is  $\gamma_t^{\mathsf{A}} = (\log (1 + K) + 2\log (1 + (1 + K)t))\left(\frac{5}{4} +4\log \left(\frac{\sqrt{1 + 2t}}{\delta}\right)\right) + 2 = \mathcal{O}((\log K + \log t)\log t)$ .
Besides, the term (b) can be bounded by the following lemma,
Lemma 14. Under Assumptions 1 and 3, for any  $c > 0$ , we have
$$
\sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s}, y _ {s}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1}) \leq \frac {1}{c} \sum_ {s = 1} ^ {t} \| \overrightarrow {W} _ {s} - \overrightarrow {W} _ {s + 1} \| _ {H _ {s}} ^ {2} + \gamma_ {t} ^ {\tt B} (\delta),
$$
by setting  $\lambda \geq \max \{2, 24Kdc\}$ . In the above,  $\gamma_t^{\mathsf{B}}(\delta) = \sqrt{6} cKd\ln\left(1 + \frac{(t+1)L}{2\lambda}\right)$  and  $L := \max_{W \in \mathbb{R}^{Kd}, t \in [T]} \nabla^2\ell_t(W)$  is the smooth parameter of the logistic loss.
Combining Lemma 12, Lemma 13 and Lemma 14, we arrive at
$$
\begin{array}{l} \left\| \vec {W} _ {t + 1} - \vec {W} _ {*} \right\| _ {H _ {t + 1}} \\ \leq \sqrt {4 \lambda S ^ {2} + \alpha \gamma_ {t} ^ {\mathbf {A}} + \alpha \gamma_ {t} ^ {\mathbf {B}} + (\frac {\alpha}{c} - 1) \sum_ {s = 1} ^ {t} \| \vec {W} _ {s + 1} - \vec {W} _ {s} \| _ {H _ {s}} ^ {2} + \sqrt {6} \alpha S \sum_ {s = 1} ^ {t} \| \vec {W} _ {s + 1} - \vec {W} _ {s} \| _ {I _ {K} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}} ^ {2}} \\ \leq \sqrt {4 \lambda S ^ {2} + \alpha \gamma_ {t} ^ {\mathrm {A}} + \alpha \gamma_ {t} ^ {\mathrm {B}}} \\ = \sqrt {\underbrace {4 \lambda S ^ {2}} _ {=: \mathcal {O} (K d \log K)} + \underbrace {\alpha (3 \log K + 2 \log t) \left(\frac {5}{4} + 4 \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right)\right) + 2} _ {=: \mathcal {O} ((\log K + \log t) \log t)} + \underbrace {7 \alpha^ {2} K d \ln (1 + (t + 1) L) / \sqrt {6}} _ {=: \mathcal {O} (K d \log^ {2} K \log t)}} \\ \triangleq \beta_ {t} ^ {\mathrm {O L}} (\delta). \\ \end{array}
$$
For the last second inequality, we eliminate the additional term by the parameter setting  $\eta = \alpha / 2$ ,  $c = 7\alpha / 6$  and  $\lambda \geq \{7\sqrt{6}\alpha S, 28Kd\alpha\}$  and due to the fact that  $I_K \otimes \mathbf{x}_s \mathbf{x}_s^\top \preccurlyeq I_{Kd}$  under the condition  $\| \mathbf{x}_t \|_2 \leq 1$  for any  $t \in [T]$ . In the above, we have  $\lambda = \mathcal{O}(Kd \log K)$  and  $\alpha = \mathcal{O}(\log K)$ . Then, we can show  $\| \overrightarrow{W}_{t+1} - \overrightarrow{W}_* \|_{H_{t+1}} \leq \beta_t^{0L}(\delta) = \mathcal{O}(\log K \log t\sqrt{Kd})$ , which completes the proof.
C.1.2 Proof of Lemma 12
Proof of Lemma 12. Let  $\widetilde{\ell}_s(W) = \ell_s(W_s) + \langle \overrightarrow{W} -\overrightarrow{W}_s,\nabla \ell_s(\overrightarrow{W}_s)\rangle +\frac{1}{2}\| \overrightarrow{W} -\overrightarrow{W}_s\|_{\nabla^2\ell_s(\overrightarrow{W}_s)}^2$  be a second order approximation of the original function  $\ell_s(W)$  at the point  $\overrightarrow{W}_s$ . The update rule (6) can be equally written as
$$
\overrightarrow {W} _ {s + 1} = \underset {W \in \mathcal {W}} {\arg \min } \widetilde {\ell} _ {s} (W) + \frac {1}{2 \eta} \| \overrightarrow {W} - \overrightarrow {W} _ {s} \| _ {H _ {s}} ^ {2}, \tag {30}
$$
which is an implicit online mirror descent update with the loss function  $\widetilde{\ell}_s$ . Then, according to Lemma 16, the model  $W_{t + 1}$  ensures,
$$
\langle \nabla \widetilde {\ell} _ {s} (\vec {W} _ {s + 1}), \vec {W} _ {s + 1} - \vec {W} _ {*} \rangle \leq \frac {1}{2 \eta} \left(\| \vec {W} _ {s} - \vec {W} _ {*} \| _ {H _ {s}} ^ {2} - \| \vec {W} _ {s + 1} - \vec {W} _ {*} \| _ {H _ {s}} ^ {2} - \| \vec {W} _ {s + 1} - \vec {W} _ {s} \| _ {H _ {s}} ^ {2}\right). \tag {31}
$$
One the other hand, let  $\alpha = \log (1 + K) + 2(1 + S)$ . Since  $\| W_{*}\mathbf{x}_{t}\|_{\infty} = \max_{k\in [K]}\{\|\mathbf{w}_{*}^{(k)}\|_{2}\cdot$ $\| \mathbf{x}_t\| _2\} \leq S$  and  $\ell_s(W) = \ell (W\mathbf{x}_s,y_s)$ , Lemma 1 shows
$$
\ell_ {s} \left(W _ {s + 1}\right) - \ell_ {s} \left(W _ {*}\right) \leq \left\langle \nabla \ell_ {s} \left(W _ {s + 1}\right), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \right\rangle - \frac {1}{\alpha} \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \| _ {\nabla^ {2} \ell_ {s} \left(W _ {s + 1}\right)} ^ {2}. \tag {32}
$$
By setting  $\eta = \alpha /2$ , the combination of (31) and (32) shows
$$
\begin{array}{l} \ell_ {s} \left(W _ {s + 1}\right) - \ell_ {s} \left(W _ {*}\right) \leq \langle \nabla \ell_ {s} \left(W _ {s + 1}\right) - \nabla \widetilde {\ell} _ {s} \left(W _ {s + 1}\right), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \rangle \tag {33} \\ + \frac {1}{\alpha} \left(\| \overrightarrow {W} _ {s} - \overrightarrow {W} _ {*} \| _ {H _ {s}} ^ {2} - \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \| _ {H _ {s + 1}} ^ {2} - \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {H _ {s}} ^ {2}\right). \\ \end{array}
$$
In above, we can further bound inner product term by
$$
\begin{array}{l} \langle \nabla \ell_ {s} (W _ {s + 1}) - \nabla \widetilde {\ell} _ {s} (W _ {s + 1}), \vec {W} _ {s + 1} - \vec {W} _ {*} \rangle \\ = \left\langle \nabla \ell_ {s} \left(W _ {s + 1}\right) - \nabla \ell_ {s} \left(W _ {s}\right) - \nabla^ {2} \ell_ {s} \left(\overrightarrow {W} _ {s}\right) \left(\overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s}\right), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \right\rangle \\ = \left\langle D ^ {3} \ell_ {s} \left(\xi_ {s + 1}\right) \left[ \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \right] \left(\overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s}\right), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \right\rangle \\ = \left\langle D ^ {3} \ell_ {s} \left(\xi_ {s + 1}\right) \left[ \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \right] \left(\overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s}\right), \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \right\rangle \\ \end{array}
$$
$$
\begin{array}{l} \leq \sqrt {6} \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \| _ {2} \cdot \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {\nabla^ {2} \ell_ {s} (\xi_ {s + 1})} ^ {2} \\ \leq \sqrt {6} S \| \vec {W} _ {s + 1} - \vec {W} _ {s} \| _ {I _ {K} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}} ^ {2} \tag {34} \\ \end{array}
$$
In above  $\xi_{s + 1}$  is a certain point on the line connecting  $\overrightarrow{W}_t$  and  $\overrightarrow{W}_{t + 1}$ . The notation is defined as  $D^{3}\ell_{s}(\overrightarrow{W})[\overrightarrow{U}] = \lim_{\alpha \to 0}\alpha^{-1}\left(\nabla^{2}\ell_{s}(\overrightarrow{W} +\alpha \overrightarrow{U}) - \nabla^{2}\ell_{s}(\overrightarrow{W})\right)$ . The first inequality is due to the definition of  $\ell_s$  and the second inequality is by the Taylor series for the vector-valued function. The third inequality is because the third order derivative of the logistic loss is a symmetric tensor as shown in Property 1. The last second inequality is by Definition 1 since the multiclass logistic loss is a  $\sqrt{6}$  self-concordant-like function. The last inequality is by the boundedness of the decision domain  $\mathcal{W}$  and bounded action such that  $\| \mathbf{x}\| _2\leq 1$  for any  $\mathbf{x}\in \mathcal{X}$  under Assumption 1.
Combining (33) and (34), we have
$$
\begin{array}{l} \ell_ {s} \left(W _ {s + 1}\right) - \ell_ {s} \left(W _ {*}\right) \\ \leq \frac {1}{\alpha} \left(\| \overrightarrow {W} _ {s} - \overrightarrow {W} _ {*} \| _ {H _ {s}} ^ {2} - \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {*} \| _ {H _ {s + 1}} ^ {2} - \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {H _ {s}} ^ {2}\right) + \sqrt {6} S \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {I _ {K} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}} ^ {2}, \\ \end{array}
$$
Taking the summation of the above inequality over  $t$  rounds and rearranging the term, we have
$$
\begin{array}{l} \| \overrightarrow {W} _ {t + 1} - \overrightarrow {W} _ {*} \| _ {H _ {t + 1}} ^ {2} \leq \alpha \left(\sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {s + 1})\right) + \| \overrightarrow {W} _ {1} - \overrightarrow {W} _ {*} \| _ {H _ {1}} ^ {2} \\ + \sum_ {s = 1} ^ {t} \left(\sqrt {6} \alpha S \| \vec {W} _ {s + 1} - \vec {W} _ {s} \| _ {I _ {K} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}} ^ {2} - \| \vec {W} _ {s + 1} - \vec {W} _ {s} \| _ {H _ {s}} ^ {2}\right) \\ \leq \alpha \left(\sum_ {s = 1} ^ {t} \ell_ {s} \left(W _ {*}\right) - \sum_ {s = 1} ^ {t} \ell_ {s} \left(W _ {s + 1}\right)\right) + 4 \lambda S ^ {2} \\ + \sum_ {s = 1} ^ {t} \left(\sqrt {6} \alpha S \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {I _ {K} \otimes \mathbf {x} _ {s} \mathbf {x} _ {s} ^ {\top}} ^ {2} - \| \overrightarrow {W} _ {s + 1} - \overrightarrow {W} _ {s} \| _ {H _ {s}} ^ {2}\right), \\ \end{array}
$$
C.1.3 Proof of Lemma 13
Proof of Lemma 13. Since the norm of the intermediate decision  $\widetilde{\mathbf{z}}_s = \sigma^+(\mathbb{E}_{W \sim P_s}[\sigma(W\mathbf{x}_s)])$  is generally unbounded, as suggested by [26], we use the smoothed version  $\widetilde{\mathbf{z}}_s^\mu = \sigma^+(\mathrm{smooth}_\mu(\mathbb{E}_{W \sim P_s}[\sigma(W\mathbf{x}_s)]))$  as an intermediate term in the analysis. In above, the smooth function  $\mathrm{smooth}_\mu:[0,1]^K \mapsto [0,1]^K$  with parameter  $\mu \in [0,1/2]$  is defined by  $\mathrm{smooth}_\mu(\mathbf{p}) = (1 - \mu)\mathbf{p} + \mu\mathbf{1}/(K + 1)$ , where  $\mathbf{1} \in \mathbb{R}^K$  is an all one vector.
Given the construction of the pseudo inverse function  $\sigma^{+}$  such that  $\sigma(\sigma^{+}(\mathbf{p})) = \mathbf{p}$  for any  $\mathbf{p} \in \{\mathbf{q} \in [0,1]^K \mid \| \mathbf{q}\|_1 < 1\}$ , one can check that  $\widetilde{\mathbf{z}}_s^\mu = \sigma^+(\mathrm{smooth}_\mu(\sigma(\widetilde{\mathbf{z}}_s)))$ . Then, Lemma 17 shows that
$$
\sum_ {s = 1} ^ {t} \ell \left(\widetilde {\mathbf {z}} _ {s} ^ {\mu}, y _ {s}\right) - \sum_ {s = 1} ^ {t} \ell \left(\widetilde {\mathbf {z}} _ {s}, y _ {s}\right) \leq 2 \mu t. \tag {35}
$$
Besides, Lemma 17 also shows that  $\| \widetilde{\mathbf{z}}_s^\mu \|_\infty \leq \log (1 + (K + 1) / \mu)$ . Therefore, to prove the lemma, it is sufficient to bound the gap between the loss of  $W_*$  and  $\widetilde{\mathbf{z}}_s^\mu$ . By the definition of the loss function  $\ell_s$ , we have  $\ell_s(W_*) = \ell(\mathbf{z}_s^*, \mathbf{y}_s)$ , where  $\mathbf{z}_s^* = W_*\mathbf{x}_s$ . Then, we have
$$
\begin{array}{l} \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s} ^ {\mu}, y _ {s}) = \sum_ {s = 1} ^ {t} \ell (\mathbf {z} _ {s} ^ {*}, y _ {s}) - \sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s} ^ {\mu}, y _ {s}) \\ \leq \sum_ {s = 1} ^ {t} \left\langle \nabla_ {z} \ell \left(\mathbf {z} _ {s} ^ {*}, y _ {s}\right), \mathbf {z} _ {s} ^ {*} - \widetilde {\mathbf {z}} _ {s} ^ {\mu} \right\rangle - \sum_ {s = 1} ^ {t} \frac {1}{S _ {\mu}} \left\| \mathbf {z} _ {s} ^ {*} - \widetilde {\mathbf {z}} _ {s} ^ {\mu} \right\| _ {\nabla_ {z} ^ {2} \ell \left(\mathbf {z} _ {s} ^ {*}, y _ {s}\right)} ^ {2} \\ \end{array}
$$
$$
= \sum_ {s = 1} ^ {t} \left\langle \boldsymbol {\sigma} \left(\mathbf {z} _ {s} ^ {*}\right) - \mathbf {y} _ {s}, \mathbf {z} _ {s} ^ {*} - \widetilde {\mathbf {z}} _ {s} ^ {\mu} \right\rangle - \sum_ {s = 1} ^ {t} \frac {1}{S _ {\mu}} \| \mathbf {z} _ {s} ^ {*} - \widetilde {\mathbf {z}} _ {s} ^ {\mu} \| _ {\nabla \boldsymbol {\sigma} \left(\mathbf {z} _ {s} ^ {*}\right)} ^ {2} \tag {36}
$$
where  $S_{\mu} = \log (K + 1) + 2\log (1 + (K + 1) / \mu)$ . In above, the first inequality is due to Lemma 1. The second equality is by a direct calculation of the first order and Hessian of the logistic loss as shown in Property 1.
To bound the first term of the right hand side, we require the following lemma, which adapts lemma 6 to a one-dimension case and whose proof is provided C.1.5.
Lemma 15. Let  $\{\mathcal{F}_t\}_{t=1}^{\infty}$  be a filtration. Let  $\{\mathbf{z}_t\}_{t=1}^{\infty}$  be a stochastic process in  $\mathcal{B}_2(K) = \{\mathbf{z} \in \mathbb{R}^K \mid \| \mathbf{z} \|_{\infty} \leq 1\}$  such that  $\mathbf{z}_t$  is  $\mathcal{F}_t$  measurable. Let  $\{\varepsilon_t\}_{t=1}^{\infty}$  be a martingale difference sequence such that  $\varepsilon_t \in \mathbb{R}^K$  is  $\mathcal{F}_{t+1}$  measurable. Furthermore, assume that conditional on  $\mathcal{F}_t$ , we have  $\| \varepsilon_t \|_1 \leq 2$  almost surely, and denote by  $\Sigma_t := \mathbb{E}[\varepsilon_t \varepsilon_t^\top \mid \mathcal{F}_t]$ . Let  $\lambda > 0$  and for any  $t \geq 2$
$$
S _ {t} = \sum_ {s = 1} ^ {t - 1} \langle \pmb {\varepsilon} _ {s}, \mathbf {z} _ {s} \rangle \quad \mathrm {a n d} \quad H _ {t} = \lambda + \sum_ {s = 1} ^ {t - 1} \| \mathbf {z} _ {s} \| _ {\Sigma_ {s}} ^ {2}
$$
$$
\Pr \left[ \exists t > 1, S _ {t} \geq \sqrt {H _ {t}} \left(\frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\sqrt {\frac {H _ {t}}{\lambda}}\right) + \frac {4}{\sqrt {\lambda}} \log (2 / \delta)\right) \right] \leq \delta .
$$
To check the condition of Lemma 15, we let  $\mathbf{d}_s = (\mathbf{z}_s^* -\widetilde{\mathbf{z}}_s^\mu) / (S_\mu +S)$ . Since  $\| \mathbf{z}_s^*\|_{\infty}\leq \max_{k\in [K]}\| \mathbf{w}_*^{(k)}\| _2\| \mathbf{x}_s\| _2\leq S$  and  $\| \widetilde{\mathbf{z}}_s^\mu \|_{\infty}\leq \log (1 + (K + 1) / \mu)$ , one can verify that  $\| \mathbf{d}_s\|_{\infty}\leq 1$ . Besides, since  $\mathbf{z}_s^*$  and  $\widetilde{\mathbf{z}}_s^\mu$  is independent of  $\mathbf{y}_s$ , the variable  $\mathbf{d}_s$  is  $\mathcal{F}_s$  measurable. Furthermore, we can check that  $\mathbb{E}[(\pmb {\sigma}(\mathbf{z}_s^*) - \mathbf{y}_s)(\pmb {\sigma}(\mathbf{z}_s^*) - \mathbf{y}_s)^\top \mid \mathcal{F}_s] = \nabla \pmb {\sigma}(\mathbf{z}_s^*)$  and  $\| \pmb {\sigma}(\mathbf{z}_s^*) - \mathbf{y}_s\| _1\leq 2$ . Thus, a direct application of Lemma 15 shows that, with probability at least  $1 - \delta$ , we have
$$
\begin{array}{l} \sum_ {s = 1} ^ {t} \left\langle \boldsymbol {\sigma} \left(\mathbf {z} _ {s} ^ {*}\right) - \mathbf {y} _ {s}, \mathbf {z} _ {s} ^ {*} - \widetilde {\mathbf {z}} _ {s} ^ {\mu} \right\rangle (37) \\ = \left(S _ {\mu} + S\right) \sum_ {s = 1} ^ {t} \left\langle \boldsymbol {\sigma} \left(\mathbf {z} _ {s} ^ {*}\right) - \mathbf {y} _ {s}, \mathbf {d} _ {s} \right\rangle \\ \leq \left(S _ {\mu} + S\right) \sqrt {\lambda + \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\nabla \sigma (\mathbf {z} _ {s} ^ {*})} ^ {2}} \cdot \sqrt {\frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\frac {\sqrt {1 + \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\nabla \sigma (\mathbf {z} _ {s} ^ {*})} ^ {2}}}{\delta}\right)} \\ \leq \left(S _ {\mu} + S\right) \sqrt {\lambda + \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\nabla \sigma \left(\mathbf {z} _ {s} ^ {*}\right)} ^ {2}} \cdot \sqrt {\frac {\sqrt {\lambda}}{4} + \frac {4}{\sqrt {\lambda}} \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right)}, (38) \\ \end{array}
$$
for any  $t \geq 1$ . In above, the second inequality is a consequence of the fact  $\| \mathbf{d}_s \|_{\Sigma(\mathbf{z}_s^*)}^2 = \mathbf{d}_s^\top \nabla \sigma(\mathbf{z}_s^*) \mathbf{d}_s \leq 2$ . Then, combining (36) and (38) and setting  $\lambda = 1$ , we arrive
$$
\begin{array}{l} \sum_ {s = 1} ^ {t} \ell_ {s} (W _ {*}) - \sum_ {s = 1} ^ {t} \ell (\widetilde {\mathbf {z}} _ {s} ^ {\mu}, \mathbf {y} _ {s}) \\ \leq \left(S _ {\mu} + S\right) \sqrt {1 + \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\Sigma (\mathbf {z} _ {s} ^ {*})} ^ {2}} \cdot \sqrt {\frac {1}{4} + 4 \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right)} - \left(S _ {\mu} + S\right) \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\Sigma (\mathbf {z} _ {s} ^ {*})} ^ {2} \\ \leq \left(S _ {\mu} + S\right) \left(1 + \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\Sigma \left(\mathbf {z} _ {s} ^ {*}\right)}\right) + \left(S _ {\mu} + S\right) \left(\frac {1}{4} + 4 \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right)\right) - \left(S _ {\mu} + S\right) \sum_ {s = 1} ^ {t} \| \mathbf {d} _ {s} \| _ {\Sigma \left(\mathbf {z} _ {s} ^ {*}\right)} ^ {2} \\ \leq \frac {5}{4} \left(S _ {\mu} + S\right) + 4 \left(S _ {\mu} + S\right) \log \left(\frac {\sqrt {1 + 2 t}}{\delta}\right), \tag {39} \\ \end{array}
$$
$\ddagger$ We note that the  $\lambda$  here is irrelevant of the algorithm, we can set it as any value.
where the second inequality is due to AM-GM inequality. Finally, combining (35) and (39), we have