SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Neon2: Finding local minima via first-order oracles. Advances in Neural Information Processing Systems, 31, 2018.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Can sgd learn recurrent neural networks with provable generalization? Advances in Neural Information Processing Systems, 32, 2019.
[3] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. In International Conference on Machine Learning, pages 242-252. PMLR, 2019.
[4] Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
[5] Ainesh Bakshi, Rajesh Jayaram, and David P Woodruff. Learning two layer rectified neural networks in polynomial time. In Conference on Learning Theory, pages 195-268. PMLR, 2019.
[6] Arindam Banerjee, Tiancong Chen, Xinyan Li, and Yingxue Zhou. Stability based generalization bounds for exponential family Langevin dynamics. International Conference on Machine Learning (ICML), 2022.
[7] Alon Brutzkus and Amir Globerson. Globally optimal gradient descent for a convnet with gaussian inputs. In International conference on machine learning, pages 605-614. PMLR, 2017.
[8] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1YfAcfg1.
[9] Rishav Chourasia, Jiayuan Ye, and Reza Shokri. Differential privacy dynamics of langevin diffusion and noisy gradient descent. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=Nfbe1usrgx4.
[10] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Local sgd optimizes overparameterized neural networks in polynomial time. In International Conference on Artificial Intelligence and Statistics, pages 6840-6861. PMLR, 2022.
[11] Aymeric Dieuleveut and Kumar Kshitij Patel. Communication trade-offs for local-sgd with large step size. Advances in Neural Information Processing Systems, 32:13601-13612, 2019.
[12] Simon Du and Jason Lee. On the power of over-parametrization in neural networks with quadratic activation. In International conference on machine learning, pages 1329-1338. PMLR, 2018.
[13] Simon Du, Jason Lee, Yuandong Tian, Aarti Singh, and Barnabas Poczos. Gradient descent learns one-hidden-layer cnn: Don't be afraid of spurious local minima. In International Conference on Machine Learning, pages 1339-1348. PMLR, 2018.
[14] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pages 1675-1685. PMLR, 2019.
1 Introduction 1
2 Related Work 3
3 Notations and Problem Setup 4
4 Theoretical Analysis 4
4.1 Exact Dynamic of Each Layer 5
4.2 Self-correction of Signals in Two Layers 6
4.3 Convergence with Linear Speedup 8
5 Experiments 9
6 Conclusion 10
A Preliminaries for Analysis 16
A.1 Dynamic of the First Layer 16
A.2 Dynamic of the Second Layer 17
A.3 Concentration Inequalities for the Noises 19
A.3.1 Sub-exponential Martingale Difference 19
A.3.2 Scales of Sub-exponential Norm for Noises in Stochastic Gradients 20
A.3.3 Corollaries of Lemma 11. 22
B Bound the Weight's Norm during Training Process 22
B.1 The Scale of the Second Layer's Weight 22
B.2 The Scale of the First Layer's weight 24
C Bound the Discrepancy during Training Process 28
C.1 The Discrepancy in the First Layer 28
C.2 The Discrepancy in the Second Layer 30
D Self-correction of Signals 31
D.1 Self-correction of the Second Layer 32
D.2 Proof of Lemma 3 39
D.3 Self-correction of the First Layer 41
E Convergence with Linear Speedup 45
E.1 Convergence of the First Layer 45
E.2 Convergence of the Second Layer 48
E.3 Conclusion 50
A Preliminaries for Analysis
We write  $\mathbf{0} = (0, \ldots, 0)^{\top}$ ,  $\mathbf{1} = (1, \ldots, 1)^{\top}$  and denote  $\mathbf{I}$  by the identity matrix. The following lemma gives the closed forms of the objective and gradients.
Lemma 6 (Du et al. [13]). If every entry of  $\mathbf{Z}$  is i.i.d. sampled from a Gaussian distribution with mean 0 and variance 1, and  $\| \pmb{w}^{*}\| = 1$ , then the population loss is
$$
\begin{array}{l} L (\boldsymbol {w}, \boldsymbol {a}) = \frac {1}{2} \left[ \frac {\pi - 1}{2 \pi} \| \boldsymbol {a} \| ^ {2} + \frac {\pi - 1}{2 \pi} \| \boldsymbol {a} ^ {*} \| ^ {2} - \frac {g (\phi) - 1}{\pi} \boldsymbol {a} ^ {\top} \boldsymbol {a} ^ {*} \right. \\ \left. + \frac {1}{2 \pi} \left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2} + \frac {1}{2 \pi} \left(\mathbf {1} ^ {\top} \boldsymbol {a}\right) ^ {2} - \frac {1}{\pi} \left(\mathbf {1} ^ {\top} \boldsymbol {a}\right) \left(\mathbf {1} ^ {\top} \boldsymbol {a}\right) \right], \tag {A.1} \\ \end{array}
$$
where  $g(\phi) = (\pi - \phi) \cos \phi + \sin \phi$ . And the expected gradient of  $\mathbf{w}$  and  $\mathbf{a}$  are
$$
\nabla_ {\boldsymbol {w}} L (\boldsymbol {w}, \boldsymbol {a}) = - \frac {\pi - \phi}{2 \pi} \left(\boldsymbol {a} ^ {\top} \boldsymbol {a} ^ {*}\right) \cdot \boldsymbol {w} ^ {*}, \tag {A.2}
$$
$$
\nabla_ {\boldsymbol {a}} L (\boldsymbol {w}, \boldsymbol {a}) = \frac {1}{2 \pi} \left[ \left(\mathbf {1 1} ^ {\top} + (\pi - 1) \mathbf {I}\right) \boldsymbol {a} - \left(\mathbf {1 1} ^ {\top} + (g (\phi) - 1) \mathbf {I}\right) \boldsymbol {a} ^ {*} \right]. \tag {A.3}
$$
Denote the noises of stochastic gradients in each local machine  $i \in [N]$  by
$$
\pmb {\xi} _ {t} ^ {i} = \nabla_ {\pmb {w}} L (\pmb {w} _ {t} ^ {i}, \pmb {a} _ {t} ^ {i}; \pmb {Z} _ {t} ^ {i}) - \nabla_ {\pmb {w}} L (\pmb {w} _ {t} ^ {i}, \pmb {a} _ {t} ^ {i}), \quad \pmb {\epsilon} _ {t} ^ {i} = \nabla_ {\pmb {a}} L (\pmb {w} _ {t} ^ {i}, \pmb {a} _ {t} ^ {i}; \pmb {Z} _ {t} ^ {i}) - \nabla_ {\pmb {a}} L (\pmb {w} _ {t} ^ {i}, \pmb {a} _ {t} ^ {i}).
$$
Then we write the averaged noises as
$$
\boldsymbol {\xi} _ {t} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbf {P} _ {t} ^ {i} \boldsymbol {\xi} _ {t} ^ {i}, \quad \boldsymbol {\epsilon} _ {t} = \frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {\epsilon} _ {t} ^ {i}.
$$
Let  $\mathbf{P}_t = \frac{1}{\|\pmb{v}_t\|} (\mathbf{I} - \pmb {w}_t\pmb {w}_t^\top)$ . We introduce an auxiliary sequence  $\check{\pmb{v}}_t$ , which updates by
$$
\tilde {\boldsymbol {v}} _ {t + 1} = \boldsymbol {v} _ {t} - \eta \left(\mathbf {P} _ {t} \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla_ {\boldsymbol {w}} L \left(\boldsymbol {w} _ {t} ^ {i}; \boldsymbol {a} _ {t} ^ {i}\right) + \boldsymbol {\xi} _ {t}\right). \tag {A.4}
$$
$$
\begin{array}{l} \tilde {\boldsymbol {v}} _ {t + 1} = \boldsymbol {v} _ {t + 1} - \eta \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\mathbf {P} _ {t} ^ {i} - \mathbf {P} _ {t}\right) \nabla_ {\boldsymbol {w}} L \left(\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}\right) \\ =: \boldsymbol {v} _ {t + 1} - \eta \boldsymbol {h} _ {t}. \tag {A.5} \\ \end{array}
$$
We define the  $\sigma$ -filtration in the training process:  $\mathcal{F}_0$  is the trivial filtration and
$$
\mathcal {F} _ {t} = \sigma \left(\left\{\boldsymbol {\xi} _ {s} ^ {i}, \boldsymbol {\epsilon} _ {s} ^ {i}: i \in [ N ] \right\} _ {s = 0} ^ {t - 1}\right), \quad \text {f o r} t \geq 1.
$$
We also write  $\mathbb{E}_t[\cdot ] = \mathbb{E}[\cdot |\mathcal{F}_t]$
A.1 Dynamic of the First Layer
Lemma 1 restated. The first layer in local SGD satisfies that
$$
\| \boldsymbol {v} _ {t + 1} \| ^ {2} \sin^ {2} \phi_ {t + 1} = \left(1 - \eta \lambda_ {t} \cos \phi_ {t}\right) ^ {2} \| \boldsymbol {v} _ {t} \| ^ {2} \sin^ {2} \phi_ {t} - 2 \eta M _ {1, t} + \eta^ {2} M _ {2, t} + H _ {t}, \qquad (A. 6)
$$
where  $\lambda_{t} = \frac{1}{N}\sum_{i = 1}^{N}\frac{\pi - \phi_{t}^{i}}{2\pi}\frac{(\pmb{a}_{t}^{i})^{\top}\pmb{a}^{*}}{\|\pmb{v}_{t}\|^{2}},H_{t} = \| \pmb{v}_{t + 1}\|^{2}\sin^{2}\phi_{t + 1} - \| \check{\pmb{v}}_{t + 1}\|^{2}\sin^{2}\check{\phi}_{t + 1}$  and
$$
M _ {1, t} = \left(\boldsymbol {v} _ {t} - \eta \mathbf {P} _ {t} \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla_ {\boldsymbol {w}} L \left(\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}\right)\right) ^ {\top} \left(\mathbf {I} - \boldsymbol {w} ^ {*} \left(\boldsymbol {w} ^ {*}\right) ^ {\top}\right) \boldsymbol {\xi} _ {t}, M _ {2, t} = \boldsymbol {\xi} _ {t} ^ {\top} \left(\mathbf {I} - \boldsymbol {w} ^ {*} \left(\boldsymbol {w} ^ {*}\right) ^ {\top}\right) \boldsymbol {\xi} _ {t}.
$$
Proof. Let  $\tilde{\pmb{v}}_t = \pmb{v}_t - \eta \mathbf{P}_t\frac{1}{N}\sum_{i=1}^N\nabla_{\pmb{w}}L(\pmb{w}_t^i,\pmb{a}_t^i)$ , then we have  $\check{\pmb{v}}_{t+1} = \tilde{\pmb{v}}_t - \eta \pmb{\xi}_t$ . If follows that
$$
\begin{array}{l} \left\| \check {\boldsymbol {v}} _ {t + 1} \right\| ^ {2} \sin^ {2} \check {\phi} _ {t + 1} = \left\| \check {\boldsymbol {v}} _ {t + 1} \right\| ^ {2} - \left(\check {\boldsymbol {v}} _ {t + 1} ^ {\top} \boldsymbol {w} ^ {*}\right) ^ {2} \\ = \| \tilde {\boldsymbol {v}} _ {t} - \eta \boldsymbol {\xi} _ {t} \| ^ {2} - \left[ (\tilde {\boldsymbol {v}} _ {t} ^ {\top} \boldsymbol {w} ^ {*}) ^ {2} - 2 \eta (\boldsymbol {\xi} _ {t} ^ {\top} \boldsymbol {w} ^ {*}) (\tilde {\boldsymbol {v}} _ {t} ^ {\top} \boldsymbol {w} ^ {*}) + \eta^ {2} (\boldsymbol {\xi} _ {t} ^ {\top} \boldsymbol {w} ^ {*}) ^ {2} \right] \\ = \| \tilde {\boldsymbol {v}} _ {t} \| ^ {2} - \left(\tilde {\boldsymbol {v}} _ {t} ^ {\top} \boldsymbol {w} ^ {*}\right) ^ {2} - 2 \eta \tilde {\boldsymbol {v}} _ {t} ^ {\top} \left(\mathbf {I} - \boldsymbol {w} ^ {*} \left(\boldsymbol {w} ^ {*}\right) ^ {\top}\right) \boldsymbol {\xi} _ {t} + \eta^ {2} \boldsymbol {\xi} _ {t} ^ {\top} \left(\mathbf {I} - \boldsymbol {w} ^ {*} \left(\boldsymbol {w} ^ {*}\right) ^ {\top}\right) \boldsymbol {\xi} _ {t} \\ = \left\| \tilde {\boldsymbol {v}} _ {t} \right\| ^ {2} - \left(\tilde {\boldsymbol {v}} _ {t} ^ {\top} \boldsymbol {w} ^ {*}\right) ^ {2} - 2 \eta M _ {1, t} + \eta^ {2} M _ {2, t}. \tag {A.7} \\ \end{array}
$$
From the definition  $\lambda_{t} = \frac{1}{N}\sum_{i = 1}^{N}\frac{\pi - \phi_{t}^{i}}{2\pi}\frac{\left(\boldsymbol{a}_{t}^{i}\right)^{\top}\boldsymbol{a}^{*}}{\|\boldsymbol{v}_{t}\|^{2}}$  and (A.2), we can write
$$
\begin{array}{l} \tilde {\boldsymbol {v}} _ {t} ^ {\top} \boldsymbol {w} ^ {*} = \boldsymbol {v} _ {t} ^ {\top} \boldsymbol {w} ^ {*} - \eta (\boldsymbol {w} ^ {*}) ^ {\top} \mathbf {P} _ {t} \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla_ {\boldsymbol {w}} L (\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}) \\ = \boldsymbol {v} _ {t} ^ {\top} \boldsymbol {w} ^ {*} + \eta (\boldsymbol {w} ^ {*}) ^ {\top} \left(\mathbf {I} - \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top}\right) \boldsymbol {w} ^ {*} \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\pi - \phi_ {t} ^ {i}}{2 \pi} \frac {\left(\boldsymbol {a} _ {t} ^ {i}\right) ^ {\top} \boldsymbol {a} ^ {*}}{\| \boldsymbol {v} _ {t} \|} \\ = \| \boldsymbol {v} _ {t} \| \cos \phi_ {t} + \eta \| \boldsymbol {v} _ {t} \| (1 - \cos^ {2} \phi_ {t}) \frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\pi - \phi_ {t} ^ {i}}{2 \pi} \frac {\left(\boldsymbol {a} _ {t} ^ {i}\right) ^ {\top} \boldsymbol {a} ^ {*}}{\| \boldsymbol {v} _ {t} \| ^ {2}} \\ = \left\| \boldsymbol {v} _ {t} \right\| \left(\cos \phi_ {t} + \eta \lambda_ {t} \sin^ {2} \phi_ {t}\right), \tag {A.8} \\ \end{array}
$$
$$
\begin{array}{l} \left\| \tilde {\boldsymbol {v}} _ {t} \right\| ^ {2} = \left\| \boldsymbol {v} _ {t} - \eta \mathbf {P} _ {t} \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla_ {\boldsymbol {w}} L \left(\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}\right) \right\| ^ {2} \\ \stackrel {(i)} {=} \| \boldsymbol {v} _ {t} \| ^ {2} + \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\eta}{2 \pi} \frac {\pi - \phi_ {t} ^ {i}}{\| \boldsymbol {v} _ {t} \|} (\boldsymbol {a} _ {t} ^ {i}) ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2} (\boldsymbol {w} ^ {*}) ^ {\top} \left(\mathbf {I} - \boldsymbol {w} _ {t} \boldsymbol {w} _ {t} ^ {\top}\right) ^ {2} \boldsymbol {w} ^ {*} \\ \stackrel {(i i)} {=} \| \boldsymbol {v} _ {t} \| ^ {2} + \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \frac {\eta}{2 \pi} \frac {\pi - \phi_ {t} ^ {i}}{\| \boldsymbol {v} _ {t} \|} (\boldsymbol {a} _ {t} ^ {i}) ^ {\top} \boldsymbol {a} ^ {*}\right) ^ {2} (1 - \cos^ {2} \phi_ {t}) \\ \stackrel {(i i i)} {=} \left\| \boldsymbol {v} _ {t} \right\| ^ {2} \left(1 + \eta^ {2} \lambda_ {t} ^ {2} \sin^ {2} \phi_ {t}\right), \tag {A.9} \\ \end{array}
$$
where  $(i)$  holds since  $\mathbf{P}_t\pmb {v}_t = \mathbf{0}$ ,  $(ii)$  is true due to  $\mathbf{I} - \pmb {w}_t\pmb {w}_t^\top$  is an idempotent matrix, and  $(iii)$  comes from  $\| \pmb {w}^{*}\| = 1$ . Combining (A.8) and (A.9), we can get
$$
\begin{array}{l} \| \tilde {\boldsymbol {v}} _ {t} \| ^ {2} - \left(\tilde {\boldsymbol {v}} _ {t} ^ {\top} \boldsymbol {w} ^ {*}\right) ^ {2} = \left[ 1 + \eta^ {2} \lambda_ {t} ^ {2} \sin^ {2} \phi_ {t} - \left(\cos \phi_ {t} + \eta \lambda_ {t} \sin^ {2} \phi_ {t}\right) ^ {2} \right] \| \boldsymbol {v} _ {t} \| ^ {2} \\ = \left[ \sin^ {2} \phi_ {t} - 2 \eta \lambda_ {t} \cos \phi_ {t} \sin^ {2} \phi_ {t} + \eta^ {2} \lambda_ {t} ^ {2} \left(\sin^ {2} \phi_ {t} - \sin^ {4} \phi_ {t}\right) \right] \| \boldsymbol {v} _ {t} \| ^ {2} \\ = \left(\sin^ {2} \phi_ {t} - 2 \eta \lambda_ {t} \cos \phi_ {t} \sin^ {2} \phi_ {t} + \eta^ {2} \lambda_ {t} ^ {2} \sin^ {2} \phi_ {t} \cos^ {2} \phi_ {t}\right) \| \boldsymbol {v} _ {t} \| ^ {2} \\ = \left(1 - 2 \eta \lambda_ {t} \cos \phi_ {t} + \eta^ {2} \lambda_ {t} ^ {2} \cos^ {2} \phi_ {t}\right) \| \boldsymbol {v} _ {t} \| ^ {2} \sin^ {2} \phi_ {t} \\ = \left(1 - \eta \lambda_ {t} \cos \phi_ {t}\right) ^ {2} \| \boldsymbol {v} _ {t} \| ^ {2} \sin^ {2} \phi_ {t}. \tag {A.10} \\ \end{array}
$$
Plugging (A.10) into (A.7) gives
$$
\left\| \check {\boldsymbol {v}} _ {t + 1} \right\| ^ {2} \sin^ {2} \check {\phi} _ {t + 1} = \left(1 - \eta \lambda_ {t} \cos \phi_ {t}\right) ^ {2} \left\| \boldsymbol {v} _ {t} \right\| ^ {2} \sin^ {2} \phi_ {t} - 2 \eta M _ {1, t} + \eta^ {2} M _ {2, t},
$$
where  $\mathbb{E}_t[M_{1,t}] = 0$  since  $\mathbf{P}_t$ ,  $\tilde{\pmb{v}}_t \in \mathcal{F}_t$  and  $\mathbb{E}_t[\pmb{\xi}_t] = \mathbf{0}$ , and  $M_{2,t} \geq 0$  because  $\mathbf{P}_t$  and  $\mathbf{I} - \pmb{w}^*(\pmb{w}^*)^\top$  are both semi positive definite. The result follows from the definition of  $H_t$ .
A.2 Dynamic of the Second Layer
Lemma 2 restated. Let  $A_0 = (\mathbf{1}^\top \mathbf{a}^*)^2 - (\mathbf{1}^\top \mathbf{a}^*)(\mathbf{1}^\top \mathbf{a}_0)$ . For local SGD algorithm started with the initial point  $(v_0, a_0)$ , the second layer satisfies that
$$
\boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} = \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \boldsymbol {a} _ {0} ^ {\top} \boldsymbol {a} ^ {*} + \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} A _ {0}
$$
$$
+ \frac {\eta \| \boldsymbol {a} ^ {*} \| ^ {2}}{2 \pi} \sum_ {s = 0} ^ {t - 1} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - 1 - s} \frac {1}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} + S \left(\epsilon_ {0: t - 1}\right), \tag {A.11}
$$
$$
\begin{array}{l} S (\pmb {\epsilon} _ {0: t - 1}) = \frac {\eta^ {2}}{2 \pi} \sum_ {s = 0} ^ {t - 1} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - 1 - s} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\pmb {1} ^ {\top} \pmb {a} ^ {*}) (\pmb {1} ^ {\top} \pmb {\epsilon} _ {l}) \\ - \eta \sum_ {s = 0} ^ {t - 1} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - 1 - s} \boldsymbol {\epsilon} _ {s} ^ {\top} \boldsymbol {a} ^ {*}. \\ \end{array}
$$
$$
B _ {s} ^ {i} = (g (\phi_ {s} ^ {i}) - 1) + \frac {\eta}{2 \pi} \frac {(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2}}{\| \mathbf {a} ^ {*} \| ^ {2}} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\pi - g (\phi_ {l} ^ {i})).
$$
Proof. From (A.3), we have
$$
\begin{array}{l} \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla L _ {\boldsymbol {a}} \left(\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}\right) = \frac {1}{2 \pi} \left(\mathbf {1 1} ^ {\top} + (\pi - 1) \mathbf {I}\right) \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \boldsymbol {a} _ {t} ^ {i}\right) - \frac {1}{2 \pi} \left[ \mathbf {1 1} ^ {\top} + \left(\frac {1}{N} \sum_ {i = 1} ^ {N} g \left(\phi_ {t} ^ {i}\right) - 1\right) \mathbf {I} \right] \boldsymbol {a} ^ {*} \\ = \frac {1}{2 \pi} \left(\mathbf {1 1} ^ {\top} + (\pi - 1) \mathbf {I}\right) \boldsymbol {a} _ {t} - \frac {1}{2 \pi} \left[ \mathbf {1 1} ^ {\top} + \left(\frac {1}{N} \sum_ {i = 1} ^ {N} g (\phi_ {t} ^ {i}) - 1\right) \mathbf {I} \right] \boldsymbol {a} ^ {*}. \\ \end{array}
$$
Let  $A_{t} = (\mathbf{1}^{\top}\pmb{a}^{*})^{2} - (\mathbf{1}^{\top}\pmb{a}^{*})(\mathbf{1}^{\top}\pmb{a}_{t})$ . By the definition of  $\pmb{a}_{t + 1}$ , we know
$$
\begin{array}{l} \boldsymbol {a} _ {t + 1} ^ {\top} \boldsymbol {a} ^ {*} = \frac {1}{N} \sum_ {i = 1} ^ {N} \left(\boldsymbol {a} _ {t + 1} ^ {i}\right) ^ {\top} \boldsymbol {a} ^ {*} = \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} - \eta \frac {1}{N} \sum_ {i = 1} ^ {N} \nabla_ {\boldsymbol {a}} L \left(\boldsymbol {w} _ {t} ^ {i}, \boldsymbol {a} _ {t} ^ {i}\right) ^ {\top} \boldsymbol {a} ^ {*} - \eta \epsilon_ {t} ^ {\top} \boldsymbol {a} ^ {*} \\ = \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) \boldsymbol {a} _ {t} ^ {\top} \boldsymbol {a} ^ {*} + \frac {\eta}{2 \pi} \left(\frac {1}{N} \sum_ {i = 1} ^ {N} g \left(\phi_ {t}\right) - 1\right) \| \boldsymbol {a} ^ {*} \| ^ {2} + \frac {\eta}{2 \pi} A _ {t} - \eta \boldsymbol {\epsilon} _ {t} ^ {\top} \boldsymbol {a} ^ {*}, \tag {A.12} \\ \end{array}
$$
For  $t\geq 0$  , we also have
$$
\begin{array}{l} A _ {t + 1} = \left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2} - \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) \left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) \left(\mathbf {1} ^ {\top} \mathbf {a} _ {t}\right) \\ - \eta \frac {k + \frac {1}{N} \sum_ {i = 1} ^ {N} g (\boldsymbol {\phi} _ {t} ^ {i}) - 1}{2 \pi} (\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2} + \eta (\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) (\mathbf {1} ^ {\top} \boldsymbol {\epsilon} _ {t}) \\ = \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) A _ {t} + \eta \frac {\pi - \frac {1}{N} \sum_ {i = 1} ^ {N} g (\phi_ {t} ^ {i})}{2 \pi} (\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) ^ {2} + \eta (\mathbf {1} ^ {\top} \mathbf {a} ^ {*}) (\mathbf {1} ^ {\top} \boldsymbol {\epsilon} _ {t}) \\ = \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {t + 1} A _ {0} + \frac {\eta \left(\mathbf {1} ^ {\top} \mathbf {a} ^ {*}\right) ^ {2}}{2 \pi} \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {t - s} \left(\pi - \frac {1}{N} \sum_ {i = 1} ^ {N} g \left(\phi_ {s} ^ {i}\right)\right) \\ + \eta \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {t - s} \left(\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}\right) \left(\mathbf {1} ^ {\top} \boldsymbol {\epsilon} _ {s}\right). \tag {A.13} \\ \end{array}
$$
Plugging (A.13) into (A.12), then enrolling (A.12) from timestamp 0 to  $t$  gives that
$$
\begin{array}{l} \pmb {a} _ {t + 1} ^ {\top} \pmb {a} ^ {*} = \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t + 1} \pmb {a} _ {0} ^ {\top} \pmb {a} ^ {*} + \frac {\eta \| \pmb {a} ^ {*} \| ^ {2}}{2 \pi} \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - s} \left(\frac {1}{N} \sum_ {i = 1} ^ {N} g (\phi_ {s} ^ {i}) - 1\right) \\ + \left(\frac {\eta | \mathbf {1} ^ {\top} \boldsymbol {a} ^ {*} |}{2 \pi}\right) ^ {2} \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - s} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} \left(\pi - \frac {1}{N} \sum_ {i = 1} ^ {N} g \left(\phi_ {l} ^ {i}\right)\right) \\ + \frac {\eta^ {2}}{2 \pi} \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - s} \sum_ {l = 0} ^ {s - 1} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s - 1 - l} (\mathbf {1} ^ {\top} \boldsymbol {a} ^ {*}) (\mathbf {1} ^ {\top} \boldsymbol {\epsilon} _ {l}) \\ \end{array}
$$
$$
\begin{array}{l} - \eta \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - s} \boldsymbol {\epsilon} _ {s} ^ {\top} \boldsymbol {a} ^ {*} + \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t + 1} A _ {0} \\ = \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \boldsymbol {a} _ {0} ^ {\top} \boldsymbol {a} ^ {*} + \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} A _ {0} \\ + \frac {\eta \| \boldsymbol {a} ^ {*} \| ^ {2}}{2 \pi} \sum_ {s = 0} ^ {t - 1} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - 1 - s} \frac {1}{N} \sum_ {i = 1} ^ {N} B _ {s} ^ {i} + S (\boldsymbol {\epsilon} _ {0: t - 1}). \\ \end{array}
$$
In fact, we also used the following summation result in the first equality
$$
\begin{array}{l} \frac {\eta}{2 \pi} \sum_ {s = 0} ^ {t} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t - s} \left(1 - \eta \frac {\pi + k - 1}{2 \pi}\right) ^ {s} \\ = \frac {\eta}{2 \pi} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \sum_ {s = 0} ^ {t} \left(\frac {1 - \eta^ {\frac {\pi + k - 1}{2 \pi}}}{1 - \eta^ {\frac {\pi - 1}{2 \pi}}}\right) ^ {s} \\ = \frac {\eta}{2 \pi} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t} \frac {2 \pi - \eta (\pi - 1)}{\eta k} \left[ 1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t + 1} \right] \\ = \frac {1 - \left(\frac {2 \pi - \eta (k + \pi - 1)}{2 \pi - \eta (\pi - 1)}\right) ^ {t + 1}}{k} \left(1 - \eta \frac {\pi - 1}{2 \pi}\right) ^ {t + 1}. \\ \end{array}
$$
A.3 Concentration Inequalities for the Noises
A.3.1 Sub-exponential Martingale Difference
Definition A.1 (Sub-exponential and sub-gaussian random variable). The random variable  $\mathbf{x} \in \mathbb{R}$  is called sub-exponential with norm  $\| \mathbf{x} \|_{\psi_1}$  if
$$
\left\| \mathbf {x} \right\| _ {\psi_ {1}} = \inf  \left\{s \geq 0: \mathbb {E} \left[ e ^ {| \mathbf {x} | / s} \right] \leq 2 \right\}.
$$
In addition, the random variable  $\mathbf{x} \in \mathbb{R}$  is called sub-gaussian with norm  $\| \mathbf{x} \|_{\psi_2}$  if
$$
\left\| \mathbf {x} \right\| _ {\psi_ {2}} = \inf  \left\{s \geq 0: \mathbb {E} \left[ e ^ {\mathbf {x} ^ {2} / s} \right] \leq 2 \right\}.
$$
Lemma 7 (Proposition 2.6.1 in [50]). Let  $\mathbf{x}_1, \ldots, \mathbf{x}_n$  be independent mean-zero sub-gaussian random variables. Then  $\sum_{i=1}^{N} \mathbf{x}_i$  is also a sub-gaussian random variable with
$$
\left\| \sum_ {i = 1} ^ {N} \mathbf {x} _ {i} \right\| _ {\psi_ {2}} ^ {2} \lesssim \sum_ {i = 1} ^ {N} \left\| \mathbf {x} _ {i} \right\| _ {\psi_ {2}} ^ {2}.
$$
Lemma 8 (Lemma 2.6.8 in [50]). If  $\mathbf{x}$  is a sub-gaussian random variable, then  $\mathbf{x} - \mathbb{E}[\mathbf{x}]$  is subgaussian too and
$$
\| \mathbf {x} - \mathbb {E} [ \mathbf {x} ] \| _ {\psi_ {2}} \lesssim \| \mathbf {x} \| _ {\psi_ {2}}.
$$
Lemma 9 (Lemma 2.7.7 in [50]). Let  $\mathbf{x}$  and  $\mathbf{y}$  be sub-gaussian random variables. Then  $\mathbf{xy}$  is sub-exponential and
$$
\| \mathbf {x} \mathbf {y} \| _ {\psi_ {1}} \leq \| \mathbf {x} \| _ {\psi_ {2}} \| \mathbf {y} \| _ {\psi_ {2}}.
$$
Lemma 10. Suppose  $\{\mathbf{x}_t\}_{t\geq 1}\subseteq \mathbb{R}$  are sub-exponential martingale difference sequence with norm  $\{\| \mathbf{x}_t\|_{\psi_1}\}_{t\geq 1}$  adapted to the filtration  $\{\mathcal{F}_t\}_{t\geq 1}$ , that is  $\mathbb{E}[\mathbf{x}_t|\mathcal{F}_t] = 0$  and  $\| \mathbf{x}_t\|_{\psi_1} = \inf \{s\geq 0:\mathbb{E}[e^{|\mathbf{x}_t| / s}|\mathcal{F}_t]\leq 2\}$ . Then we have
$$
\mathbb {P} \left(\left| \sum_ {s = 1} ^ {t} \mathbf {x} _ {s} \right| \geq \frac {1}{c} \sqrt {\log (1 / \delta) \sum_ {s = 1} ^ {t} \| \mathbf {x} _ {s} \| _ {\psi_ {1}} ^ {2}} + c \log (1 / \delta) \max _ {1 \leq s \leq t} \| \mathbf {x} _ {s} \| _ {\psi_ {1}}\right) \leq 2 \delta ,
$$