SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Basic Notations
M	Vocabulary size
T	Sequence length
ek	One-hot vector (1 at component k)
X∈RT-1×M	Input sequence (of length T-1)
bT∈RT-1	Vector of self-attention weights to predict the token at time T.
xt∈RM	contextual token (0 ≤ t ≤ T-2) (one-hot)
xt-1∈RM	Last/query token (one-hot)
xt∈RM	Next token (class label) to be predicted (one-hot)
xt[i]∈RM	i-th training sample of token at location t in the sequence
K	Number of possible choices the next token could take.
α(t)	Softmax score of the output layer.
Learnable Parameters
Y∈RM×M	decoder layer parameters
Z∈RM×M	self-attention logits
zm	m-th row of Z (i.e., attention logits for a query/query token m)
Hyperparameters
ηY	Learning rate of the decoder layer
ηZ	Learning rate of the self-attention layer
Token Types and Distribution
ψ(n)	Mapping from next token xt=n to its unique last/query token
ψ-1(m)	The subset of next tokens for last/query token xT-1=m
P(l|m,n)	Conditional probability of contextual token l
GCT	given query token is m and next token to be predicted as n
GDT(n)	Subset of common tokens
Subset of distinct tokens for xT=n
Attention Score
cn∈RM	Unnormalized attention score given next token xt=n
cn∈RM	l1-normalized attention score given next token xt=n
fn∈RM	l2-normalized attention score given next token xt=n
g∈RM	Back-propagated gradient for fn
F	Input matrix of the decoder layer. Each column of F is fn
Self-attention dynamics
rl′/l|n(t)	Relative gain between distinct token l and l′ for next token n
Bn(t)	Growth factor bound of the relative gain
γ(t)	Speed control coefficient
Table 1: Overall notation table of the main symbols in the paper.
A Notation Table
Tbl. 1 gives the notation of the main quantities in the paper.
B Detailed comparison with the concurrent works
B.1 Comparison with [28]
Setting, Assumptions and Conclusions. [28] analyzes the SGD convergence of 1-layer ViT model (1 layer self-attention + 2 layer FFN with ReLU, with the top layer of FFN fixed as random, token embedding fixed). Under a specific binary data model in which the data label is determined by counting the number of tokens that belong to positive/negative pattern, [28] gives a generalization bound when the number of hidden nodes in FFN is large, and at the same time, shows that the self-attention attends to relevant tokens and becomes sparse (if number of relevant tokens are small).
In comparison, our work focuses on language models, assume broader data distribution (e.g., multiple classes, arbitrary conditional probability of token given class label) and incorporate LayerNorm
naturally. We propose more detailed quantitative properties, e.g., attention sparsity even among relevant tokens, two-stage evolution of attention scores, with a much simpler analysis.
Techniques. The techniques used in [28] are based on feature learning techniques applied to MLP (e.g., [87]). It identifies lucky neurons if the number of hidden neurons is large enough. In comparison, our framework and analysis is much simpler by leveraging that certain nonlinear continuous dynamics systems can be integrated out analytically to yield clean solutions (e.g., Theorem 3 (Eqn. 11) and Theorem 4 (Eqn. 128)), avoiding complicated bounds in [28]. This allows us to characterize the converging behavior of self-attention when  $t \to +\infty$ .
B.2 Comparison with [29]
[29] focuses on 1-layer attention-based prompt-tuning, in which some parameters of the models are fixed  $(W_{p}, W_{q})$ . The analysis focuses on the initial (3x one-step) SGD trajectory, and constructs the dataset model containing specific context-relevant/context-irrelevant data, and the context-vector indicates the token relevance. As a result, [29] shows the attention becomes sparse (i.e., attending to context-relevant tokens) over time, which is consistent with ours, and shows that prompt-attention can find the relevant tokens and achieve high accuracy while self-attention/linear-attention can't.
In comparison, our work goes beyond the 2-classes model and further points out that the attention weight will be relevant to the conditional probability of the contextual tokens, which is more detailed than the sparse attention result in [29] that relies on the sparsity assumption of contextual tokens itself. We also focus on the pre-training stage (training from scratch, predicting the next token), characterize the entire trajectory under SGD for the self-attention layer, in particular its converging behavior.
B.3 Comparison with [30]
Compared to [29], [30] also analyzes the dynamics of the query-key matrix and the embedding of a single tunable token (often [CLS] token). It makes connection between the binary classification problem with 1-layer transformer and max-margin SVM formulation, when the tokens are linearly separable. The dynamics is characterized completely, which is nice. Note here is not an attention since its norm can be shown to go to infinity over training.
In comparison, our work does not learn the embedding of an individual token, but focuses on the dynamics of (all-pair) attention scores during training. We also work on multiple-class setup and do not explicitly assume the linear separability among classes.
C Proof of Section 3
$$
\dot {Y} = \eta_ {Y} \ln \left(X ^ {\top} \boldsymbol {b} _ {T}\right) \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top}, \quad \dot {Z} = \eta_ {Z} \boldsymbol {x} _ {T} \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top} Y ^ {\top} \frac {P _ {X ^ {\top} \boldsymbol {b} _ {T}} ^ {\perp}}{\| X ^ {\top} \boldsymbol {b} _ {T} \| _ {2}} X ^ {\top} \operatorname {d i a g} (\boldsymbol {b} _ {T}) X \tag {3}
$$
Proof. With the reparameterization of  $Y$  and  $Z$ , the loss function is the following:
$$
J (Y, Z) = \mathbb {E} _ {\mathcal {D}} \left[ \boldsymbol {x} _ {T + 1} ^ {\top} Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b} _ {T}\right) - \log \left(\boldsymbol {1} ^ {\top} \exp \left(Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b} _ {T}\right)\right)\right) \right] \tag {15}
$$
$$
\alpha_ {m} = \frac {\exp \left(\boldsymbol {e} _ {m} ^ {\top} Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b} _ {T}\right)\right)}{\mathbf {1} ^ {\top} \exp \left(Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b} _ {T}\right)\right)} \tag {16}
$$
Therefore, taking matrix differentials, we have:
$$
\mathrm {d} J = \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top} \mathrm {d} \left(Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b}\right)\right) = \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top} \left(\mathrm {d} Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b}\right) + Y ^ {\top} \frac {P _ {X ^ {\top} \boldsymbol {b}} ^ {\perp}}{\| X ^ {\top} \boldsymbol {b} \|} X ^ {\top} \mathrm {d} \boldsymbol {b}\right) \tag {17}
$$
since in general we have  $\mathrm{d}(\exp (\pmb {a}) / \mathbf{1}^{\top}\exp (\pmb {a})) = L\mathrm{d}\pmb{a}$  with  $L\coloneqq \operatorname {diag}(\pmb {b}) - \pmb {b}\pmb{b}^{\top}$ , let  $\pmb {a}\coloneqq XZ^{\top}\pmb{x}_T$  and we have:
$$
\begin{array}{l} \mathrm {d} J = \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top} \left(\mathrm {d} Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b}\right) + Y ^ {\top} \frac {P _ {X ^ {\top} \boldsymbol {b}} ^ {\perp}}{\| X ^ {\top} \boldsymbol {b} \|} X ^ {\top} L \mathrm {d} \left(X Z ^ {\top} \boldsymbol {x} _ {T}\right)\right) (18) \\ = \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top} \left(\mathrm {d} Y ^ {\top} \operatorname {L N} \left(X ^ {\top} \boldsymbol {b}\right) + Y ^ {\top} \frac {P _ {X ^ {\top} \boldsymbol {b}} ^ {\perp}}{\| X ^ {\top} \boldsymbol {b} \|} X ^ {\top} L X \mathrm {d} Z ^ {\top} \boldsymbol {x} _ {T}\right) (19) \\ \end{array}
$$
Finally notice that  $P_{X^{\top}b}^{\perp} X^{\top} L = P_{X^{\top}b}^{\perp} X^{\top} \operatorname{diag}(b)$  due to the fact that  $P_{\mathbf{v}}^{\perp} \mathbf{v} = 0$  and the conclusion follows.
Proof. Let  $\pmb{p} = [\exp(z_{m1}), \dots, \exp(z_{mM})]^{\top} \in \mathbb{R}^{M}$ ,  $p_{x_t} := \exp(z_{mx_t})$ , and  $\pmb{p}_X := [\exp(z_{mx_1}), \dots, \exp(z_{mx_{T-1}})]^{\top}$ , then for any  $T$  we have
$$
X ^ {\top} \boldsymbol {b} _ {T} = \sum_ {t = 1} ^ {T - 1} b _ {t T} \boldsymbol {x} _ {t} = \sum_ {t = 1} ^ {T - 1} \frac {p _ {x _ {t}} \boldsymbol {x} _ {t}}{\sum_ {t ^ {\prime}} p _ {x _ {t ^ {\prime}}}} = \frac {X ^ {\top} \boldsymbol {p} _ {X}}{\mathbf {1} ^ {\top} X ^ {\top} \boldsymbol {p} _ {X}} \tag {20}
$$
Combining Lemma 18 and the definition of  $c_{l|m,n}$  (Eqn. 5), we have that when  $T \to +\infty$
$$
X ^ {\top} \boldsymbol {b} _ {T} \rightarrow \sum_ {l = 1} ^ {M} \frac {\mathbb {P} (l \mid m , n) \exp \left(z _ {m l}\right) \boldsymbol {e} _ {l}}{\sum_ {l ^ {\prime}} \mathbb {P} \left(l ^ {\prime} \mid m , n\right) \exp \left(z _ {m l ^ {\prime}}\right)} = \boldsymbol {c} _ {m, n} \tag {21}
$$
$$
X ^ {\top} \operatorname {d i a g} \left(\boldsymbol {b} _ {T}\right) X = \frac {X ^ {\top} \operatorname {d i a g} \left(\boldsymbol {p} _ {X}\right) X}{\mathbf {1} ^ {\top} X ^ {\top} \boldsymbol {p} _ {X}} \tag {22}
$$
Let  $T \to +\infty$ , then we also get
$$
X ^ {\top} \operatorname {d i a g} \left(\boldsymbol {b} _ {T}\right) X \rightarrow \operatorname {d i a g} \left(\boldsymbol {c} _ {m, n}\right) \tag {23}
$$
For convenience, we introduce the following notations for this section:
Denote  $E^{\prime}\coloneqq (I + E)^{-1} - I$
- Apply orthogonal diagonalization on  $E$  and obtain  $E = U^{\top}DU$  where  $U \coloneqq [u_1, \dots, u_K] \in O_{K \times K}$ ,  $D = \mathrm{diag}(\lambda_1, \dots, \lambda_K)$  and  $|\lambda_1| \geq \dots \geq |\lambda_K| \geq 0$ .
- Denote  $F' \coloneqq [F, F^{\circ}] \in \mathbb{R}^{M \times M}$  where  $F^{\circ} \in \mathbb{R}^{M \times (M - K)}$  is some matrix such that  $\operatorname{rank}(F') = M$ . This is possible since  $\{f_i\}_{i \in [K]}$  are linear-independent.
Denote  $W^{\prime}\coloneqq (F^{\prime})^{\top}Y = [F,F^{\circ}]^{\top}Y = [W^{\top},Y^{\top}F^{\circ}]^{\top} = [\pmb{w}_{1},\dots,\pmb{w}_{K},\pmb{w}_{K + 1},\dots,$ $\pmb {w}_M]^\top \in \mathbb{R}^{M\times M}$
Denote  $\zeta_{n}:= \frac{M}{M - 1} (e_{n} - \frac{1}{M}\mathbf{1})\in \mathbb{R}^{M}$
- Denote  $q_{1} \coloneqq \zeta_{i}^{\top} \zeta_{i} = 1 + \frac{1}{M - 1}$ ,  $q_{0} \coloneqq \zeta_{j}^{\top} \zeta_{i} = -\frac{M}{(M - 1)^{2}}$  where  $i, j \in [M], i \neq j$ .
- Denote  $h$  to be a continuous function that satisfies  $h(0) = 0$  and  $\dot{h} = \eta_{Y} \cdot (M - 1 + \exp(Mh))^{-1}$ . Details in Lemma 6.
- Denote  $\omega_{1}$  to be the constant defined in Lemma 8 that satisfies  $\omega_{1} = \Theta\left(\frac{\ln\ln(M)}{\ln(M)}\right)$ .
- Denote  $N_{n} \coloneqq \sum_{i=1}^{N} \mathbb{I}[x_{T+1} = n]$  to be the number of times the event  $x_{T+1} = n$  happens.
- Denote  $\bar{N} \coloneqq \lceil N / K \rceil$  to be the average value of  $N_{n}$  when  $\mathbb{P}(n) \equiv 1 / K$  and  $\Delta \coloneqq \lceil \sqrt{N \ln\left(\frac{1}{\delta}\right)} \rceil$  to be the radius of confidence interval centered on  $\bar{N}$  with confidence  $1 - \delta$ . Here  $\Delta / \bar{N} \asymp \frac{K}{\sqrt{N}} \sqrt{\ln\left(\frac{1}{\delta}\right)} \ll 1$  since  $N \gg K^2$ . Details in Lemma 10 and Remark 4.
- Denote  $\bar{W}^{\prime}(N) \coloneqq [\bar{\boldsymbol{w}}_1(N), \dots, \bar{\boldsymbol{w}}_K(N), \mathbf{0}, \dots, \mathbf{0}]^\top \in \mathbb{R}^{M \times M}$ , where  $\bar{\boldsymbol{w}}_n(N) \coloneqq (M - 1)h(\bar{N})\zeta_n$ ,  $\forall n \in [K]$ .
We assume  $\cup_{m\in [M]}\psi^{-1}(m) = [K]$  for convenience, but we claim that our proof can be easily generalized into the case where  $\Omega \neq [K]$  by reordering the subscript of the vectors. First, we prove the dynamics equation of the reparameterized dynamics of  $Y$ .
Proof. We let  $F' \coloneqq [F, F^{\circ}] \in \mathbb{R}^{M \times M}$  where  $\mathrm{rank}(F') = M$ , this is possible since  $\{f_n\}_{n \in [K]}$  are linear-independent. And we further define  $W' \coloneqq (F')^{\top}Y = [F, F^{\circ}]^{\top}Y = [W^{\top}, Y^{\top}F^{\circ}]^{\top} = [\pmb{w}_1, \dots, \pmb{w}_K, \pmb{w}_{K+1}, \dots, \pmb{w}_M]^{\top} \in \mathbb{R}^{M \times M}$ . When given  $x_{T+1} = n$ , the first term of the differential of loss function  $J$  is:
$$
\begin{array}{l} \operatorname {t r} \left(\mathrm {d} Y ^ {\top} \frac {X ^ {\top} \boldsymbol {b} _ {T}}{\| X ^ {\top} \boldsymbol {b} _ {T} \| _ {2}} \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top}\right) = \operatorname {t r} \left(\mathrm {d} Y ^ {\top} F ^ {\prime} \left(F ^ {\prime}\right) ^ {- 1} \boldsymbol {f} _ {n} \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top}\right) \tag {24} \\ = \operatorname {t r} \left(\mathrm {d} \left(W ^ {\prime}\right) ^ {\top} \boldsymbol {e} _ {n} \left(\boldsymbol {x} _ {T + 1} - \boldsymbol {\alpha}\right) ^ {\top}\right) \\ \end{array}
$$
So  $\dot{W}^{\prime} = e_{n}(\pmb{x}_{T + 1} - \pmb{\alpha})^{\top}$ . This nice property will limit  $W$  to independently update its  $n$ -th row for any  $x_{T + 1} = n \in [K]$ , and the last  $M - K$  rows of  $W^{\prime}$  are not updated. Similarly for  $\alpha$  we have
$$
\boldsymbol {\alpha} = \frac {\exp \left(U W _ {V} \tilde {\boldsymbol {u}} _ {T}\right)}{\mathbf {1} ^ {\top} \exp \left(U W _ {V} \tilde {\boldsymbol {u}} _ {T}\right)} = \frac {\exp \left(Y ^ {\top} \boldsymbol {f} _ {n}\right)}{\mathbf {1} ^ {\top} \exp \left(Y ^ {\top} \boldsymbol {f} _ {n}\right)} = \frac {\exp \left(Y ^ {\top} F ^ {\prime} \left(F ^ {\prime}\right) ^ {- 1} \boldsymbol {f} _ {n}\right)}{\mathbf {1} ^ {\top} \exp \left(Y ^ {\top} F ^ {\prime} \left(F ^ {\prime}\right) ^ {- 1} \boldsymbol {f} _ {n}\right)} = \frac {\exp (\boldsymbol {w} _ {n})}{\mathbf {1} ^ {\top} \exp (\boldsymbol {w} _ {n})} \tag {25}
$$
We get Eqn. 8 by combining the above results.
If we don't run gradient update on  $W$  directly, we can run a modified gradient update on  $Y$ :
$$
\dot {Y} = \eta_ {Y} \left(\boldsymbol {f} _ {n} - F E ^ {\prime} \boldsymbol {e} _ {n}\right) \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} \tag {26}
$$
This will lead to (note that  $F$  does not change over time due to Assumption 1 (c)):
$$
\begin{array}{l} \dot {W} = F ^ {\top} \dot {Y} = \eta_ {Y} F ^ {\top} \left(\boldsymbol {f} _ {n} - F E ^ {\prime} \boldsymbol {e} _ {n}\right) \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} (27) \\ = \eta_ {Y} \left[ F ^ {\top} \boldsymbol {f} _ {n} - F ^ {\top} F (I - (I + E) ^ {- 1}) \boldsymbol {e} _ {n} \right] \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} (28) \\ = \eta_ {Y} \left(F ^ {\top} \boldsymbol {f} _ {n} - F ^ {\top} F \boldsymbol {e} _ {n} + \boldsymbol {e} _ {n}\right) \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} (29) \\ = \eta_ {Y} \boldsymbol {e} _ {n} \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} (30) \\ \end{array}
$$
By Lemma 17, we know that if  $\lambda_{1}$  is small, so does  $\max_{i\in [K]}\left|\lambda_i(E')\right|$  and thus the modification is small as well. In Lemma 5 Remark 1, we will show that the additional term  $-FE'e_n$  effectively reduces the learning rate, if all off-diagonal elements of  $E$  are the same.
Lemma 3 shows that we can transfer the problem into solving  $K$  independent and similar non-linear ODE. And we then show that such a problem can be well solved by following Lemma. Recall that  $\zeta_{n} := \frac{M}{M - 1} (e_{n} - \frac{1}{M}\mathbf{1}) \in \mathbb{R}^{M}$ , we have:
Lemma 5. Assume  $Y$  is initialized to be a zero matrix,  $Z$  is fixed, and the learning rate of  $Y$  is  $\eta_{Y}$ . Then if event  $x_{T + 1} = n$  always holds at  $s$  step ( $s \geq 1$ ) we have
$$
\boldsymbol {w} _ {n} (s) = (M - 1) h ^ {*} (s) \boldsymbol {\zeta} _ {n} \tag {31}
$$
$$
\alpha_ {n j} (s) = \left\{ \begin{array}{l l} \frac {\exp \left(M h ^ {*} (s - 1)\right)}{(M - 1) + \exp \left(M h ^ {*} (s - 1)\right)} & , \quad j = n \\ \frac {1}{(M - 1) + \exp \left(M h ^ {*} (s - 1)\right)} & , \quad j \neq n \end{array} \right. \tag {32}
$$
And thus  $e_n - \alpha_n(s) = \frac{M - 1}{M - 1 + \exp(Mh^*(s - 1))} \zeta_n$ . Here  $h^*(s)$  satisfies:
$$
h ^ {*} (s) = \left\{ \begin{array}{l l} h ^ {*} (s - 1) + \frac {\eta_ {Y}}{(M - 1) + \exp \left(M h ^ {*} (s - 1)\right)} & , s \geq 1 \\ 0 & , s = 0 \end{array} \right. \tag {33}
$$
Proof. We prove this Lemma by induction.
Step 1: Note that  $Y$  is initialized to be a zero matrix, then  $w_{i}(0) = 0, \forall i \in [K]$ . So we have
$$
\alpha_ {n} (1) = \frac {1}{M}, \quad \forall j \in [ K ] \tag {34}
$$
$$
\dot {w} _ {n j} (1) = \left\{ \begin{array}{l l} 1 - \frac {1}{M}, & j = n \\ - \frac {1}{M}, & j \neq n \end{array} \right. \tag {35}
$$
$$
w _ {n j} (1) = \left\{ \begin{array}{l l} \eta_ {Y} \left(1 - \frac {1}{M}\right), & j = n \\ - \frac {\eta_ {Y}}{M}, & j \neq n \end{array} \right. \tag {36}
$$
It's easy to check that these equations match that of Lemma 5.
Step  $s$ : Assume the equations of Lemma 5 hold for step  $s - 1$ . Then at the  $s$  step, we have
$$
\alpha_ {n j} (s) = \left\{ \begin{array}{l l} \frac {\exp ((M - 1) h ^ {*} (s - 1))}{\exp ((M - 1) h ^ {*} (s - 1)) + (M - 1) \exp (- h ^ {*} (s - 1))} & = \frac {\exp (M h ^ {*} (s - 1))}{\exp (M h ^ {*} (s - 1)) + (M - 1)}, j = n \\ \frac {\exp (- h ^ {*} (s - 1))}{\exp ((M - 1) h ^ {*} (s - 1)) + (M - 1) \exp (- h ^ {*} (s - 1))} & = \frac {1}{\exp (M h ^ {*} (s - 1)) + (M - 1)}, j \neq n \end{array} \right. \tag {37}
$$
$$
\dot {w} _ {n j} (s) = \left\{ \begin{array}{l l} \frac {M - 1}{\exp \left(M h ^ {*} (s - 1)\right) + (M - 1)}, & j = n \\ - \frac {1}{\exp \left(M h ^ {*} (s - 1)\right) + (M - 1)}, & j \neq n \end{array} \right. \tag {38}
$$
$$
w _ {n j} (s) = \left\{ \begin{array}{l l} (M - 1) \cdot \left(\frac {\eta_ {Y}}{\exp \left(M h ^ {*} (s - 1)\right) + (M - 1)} + h ^ {*} (s - 1)\right) & = (M - 1) h ^ {*} (s), \quad j = n \\ - \left(\frac {\eta_ {Y}}{\exp \left(M h ^ {*} (s - 1)\right) + (M - 1)} + h ^ {*} (s - 1)\right) & = - h ^ {*} (s), \quad j \neq n \end{array} \right. \tag {39}
$$
And the equations of Lemma 5 also hold for step  $s$ . So we finish the proof.
Remark 1. If we following the original dynamics (Eqn. 7), then it corresponds to the  $W$  dynamics as follows:
$$
\dot {W} = \eta_ {Y} \left(\boldsymbol {e} _ {n} + (I + E) E ^ {\prime} \boldsymbol {e} _ {n}\right) \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} = \eta_ {Y} F ^ {\top} \boldsymbol {f} _ {n} \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} \tag {40}
$$
When all off-diagonal elements of  $E$  are identical, i.e.,  $\pmb{f}_n^\top \pmb{f}_{n'} = \rho$  for  $n \neq n'$ , then  $0 \leq \rho \leq 1$  and we have
$$
\dot {w} _ {n} = \eta_ {Y} \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top} \tag {41}
$$
$$
\dot {w} _ {j} = \eta_ {Y} \rho \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) ^ {\top}, \quad j \neq n \tag {42}
$$
So if different sequence classes are sampled uniformly, then by similar induction argument, we will have
$$
\boldsymbol {w} _ {n} (N) = (M - 1) h ^ {*} (N / K) \left[ \boldsymbol {\zeta} _ {n} + \rho \sum_ {n ^ {\prime} \neq n} \boldsymbol {\zeta} _ {n ^ {\prime}} \right] = (1 - \rho) (M - 1) h ^ {*} (N / K) \boldsymbol {\zeta} _ {n} \tag {43}
$$
where the last equation is due to the fact that  $\sum_{n}\zeta_{n} = \frac{M}{M - 1}\sum_{n}\left(e_{n} - \frac{1}{M}\mathbf{1}\right) = \frac{M}{M - 1} (\mathbf{1} - \mathbf{1}) = 0$ . This means that  $\sum_{n^{\prime}\neq n}\zeta_{n^{\prime}} = -\zeta_{n}$ . Therefore, the effective learning rate is  $\eta_Y^\prime \coloneqq (1 - \rho)\eta_Y\leq \eta_Y$
D.3 Property of  $h^*(s)$  and its continuous counterpart.
Before further investigation on  $Y$ , we need to get some basic properties of  $h^*$ , in particular, how fast it grows over time. First, if we consider the continuous version of  $h^*$ , namely  $h$ , then we can directly obtain the equation that  $h$  needs to satisfy by integrating the corresponding differential equation.
Lemma 6. If we consider the continuous version of  $h^*(s)$ , namely  $h$ , as the following ODE:
$$
\frac {\mathrm {d} h}{\mathrm {d} t} = \frac {\eta_ {Y}}{(M - 1) + \exp (M h)} \tag {44}
$$
and assume  $h(0) = 0$ , then we have
$$
\exp (M h (t)) + (M - 1) M h (t) = M \eta_ {Y} t + 1 \tag {45}
$$
Then we will show that the  $h$  is actually almost the same as the original step function  $h^*$ .
Lemma 7. For  $h$  and  $h^*$  we have:
- (a) For any  $s \in \mathbb{N}, 0 \leq h^{*}(s) - h(s) \leq \frac{2\eta_{Y}}{M}$ . Then there exists some constant  $c = \Theta(1)$  such that for any  $s \leq \ln(M) / \eta_{Y}$ ,  $h(s + c) \geq h^{*}(s) \geq h(s)$ .
-  $(b) h^{*}(s) - h(s) \to 0$  when  $s \to +\infty$ .
Proof. (a) First we show that  $h^*(s) \geq h(s)$  for all  $s \in \mathbb{N}$ , and the convex packet function of  $h^*$  can almost control the upper bound of  $h$ . Define  $h^\circ: \mathbb{R}^+ \to \mathbb{R}^+$  as follows:
$$
h ^ {\circ} (t) := (t - \lfloor t \rfloor) \cdot [ h ^ {*} (\lceil t \rceil) - h ^ {*} (\lfloor t \rfloor) ] + h ^ {*} (\lfloor t \rfloor), \forall t \in \mathbb {R} ^ {+} \tag {46}
$$
Here  $\lceil \cdot \rceil$  and  $\lfloor \cdot \rfloor$  mean cell function and floor function, respectively. It's clear that  $h^\circ$  is a strictly monotonically increasing function, and for any  $s\in \mathbb{N}$ ,  $h^\circ (s) = h^* (s)$ , while for any  $t\notin \mathbb{N}$ ,  $(t,h^{\circ}(t))$  lies on the line connecting point  $(\lfloor t\rfloor ,h^{*}(\lfloor t\rfloor))$  and point  $(\lceil t\rceil ,h^{*}(\lceil t\rceil))$ . To prevent ambiguity, we let  $\dot{h}^{\circ}(t)$  to be the left limit of  $h^\circ$ , i.e.,  $\dot{h}^{\circ}(t) = \lim_{t'\to t - }\dot{h}^{\circ}(t')$ .
We claim  $h(t) \leq h^{\circ}(t), \forall t \in \mathbb{R}^{+}$ . We prove it by induction. First when  $t = 0$ , we have  $h^{\circ}(0) = h^{*}(0) = h(0) = 0$ . Then we assume  $h(t') \leq h^{\circ}(t')$  hold for time  $t' \leq t \in \mathbb{N}$  and prove that  $h(t') \leq h^{\circ}(t')$  hold for  $t' \in (t, t + 1]$ . If this is not true, then from the continuity of  $h^{\circ}$  and  $h$ , we know it must exist  $t'' \in (t, t + 1]$  such that  $h(t'') \geq h^{\circ}(t'')$  and  $\dot{h}(t'') > \dot{h}^{\circ}(t'')$ . The later condition results that  $\eta_{Y}[M - 1 + \exp(Mh(t''))]^{-1} > \eta_{Y}[M - 1 + \exp(Mh^{*}(\lfloor t'' \rfloor))]^{-1}$ . So
$$
h \left(t ^ {\prime \prime}\right) <   h ^ {*} (\lfloor t ^ {\prime \prime} \rfloor) = h ^ {\circ} (\lfloor t ^ {\prime \prime} \rfloor) \leq h ^ {\circ} \left(t ^ {\prime \prime}\right) \tag {47}
$$
This contradicts the hypothesis  $h(t^{\prime \prime})\geq h^{\circ}(t^{\prime \prime})$ . So  $h(t^{\prime})\leq h^{\circ}(t^{\prime})$  hold for  $t^\prime \in (t,t + 1]$  and thus for all  $t\in \mathbb{R}^{+}$ . Hence for any  $s\in \mathbb{N}$ , we have  $h(s)\leq h^{\circ}(s) = h^{*}(s)$ . Actually, we can use the similar method to prove that  $h(s) < h^{*}(s)$  for any  $s\in \mathbb{N}^{+}$ .
Then we show  $h^*(s) - h(s) \leq 2\eta_Y / M$  by proving that for any  $s \in \mathbb{N}^+$ ,  $h(s)$  must meet at least one of the following two conditions:
(i)  $h(s)\in [h^{*}(s - 1),h^{*}(s)]$
(ii)  $h^{*}(s) - h(s) <   h^{*}(s - 1) - h(s - 1).$
If (i) doesn't hold, then we have for any  $t \in [s - 1, s)$ ,  $h(t) \leq h(s) < h^{*}(s - 1) = h^{\circ}(s - 1)$ , which results that  $\dot{h}(t) > \dot{h}^{\circ}(t)$  for all  $t \in [s - 1, s)$ . Therefore,  $h^{*}(s) - h^{*}(s - 1) = h^{\circ}(s) - h^{\circ}(s - 1) < h(s) - h(s - 1)$  and thus  $h(s)$  meets condition (ii). It's clear that  $h(0)$  and  $h(1)$  meet (i).
Figure 8: Numerical simulation of  $h^*$  and  $h$  with changing  $\eta_{Y}$ . The stepped folded line represents  $h^*$  and the smooth curve represents  $h$ . The gap between  $h^*$  and  $h$  is bounded and goes to zero when time grows.