SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here  $\pmb{f}_n := \frac{\pmb{X}^\top\pmb{b}_T}{\|\pmb{X}^\top\pmb{b}_T\|_2} \to \frac{\pmb{c}_n}{\|\pmb{c}_n\|_2} \in \mathbb{R}^M$ . Obviously  $\| \pmb{f}_n \|_2 = 1$  and  $\pmb{f}_n \geq 0$ . Define  $F = [f_1, \dots, f_K]$ . Since the vocabulary size  $M$  typically is a huge number, and different sequence classes can cover diverse subsets of vocabulary, we study the weak correlation case:
Assumption 2 (Weak Correlations). We assume  $M \gg K^2$  and  $\{\pmb{f}_n\}_{n=1}^K$  satisfies  $F^\top F = I + E$ , where the eigenvalues of  $E \in \mathbb{R}^{K \times K}$  satisfy  $|\lambda_1| < \frac{1}{K}$  and  $|\lambda_i(E)| \geq \frac{6}{\sqrt{M}}$ ,  $\forall i \in [K]$ .
Assumption 2 means that  $f_{n}$  share some weak correlations and it immediately leads to the fact that  $F^{\top}F$  is invertible and  $F$  is column full-rank. Note that the critical point  $Y^{*}$  of Eqn. 7 should satisfy that for any given  $x_{T + 1} = n$ , we need  $\alpha = e_n$ . But such  $Y^{*}$  must contain infinity entries due to the property of the exponential function in  $\alpha$  and we can not achieve  $Y^{*}$  in finite steps. To analyze Eqn. 7, we leverage a reparameterized version of the dynamics, by setting  $W = [w_{1},\dots ,w_{K}]^{\top}\coloneqq F^{\top}Y\in \mathbb{R}^{K\times M}$  and compute gradient update on top of  $W$  instead of  $Y$ :
Lemma 3. Given  $x_{T + 1} = n$ , the dynamics of  $W$  is (here  $\alpha_{j} = \exp (\pmb{w}_{j}) / \mathbf{1}^{\top}\exp (\pmb{w}_{j}))$ :
$$
\dot {\boldsymbol {w}} _ {j} = \eta_ {Y} \mathbb {I} (j = n) \left(\boldsymbol {e} _ {n} - \boldsymbol {\alpha} _ {n}\right) \tag {8}
$$
While we cannot run gradient update on  $W$  directly, it can be achieved by modifying the gradient of  $Y$  to be  $\dot{Y} = \eta_{Y}(\pmb{f}_{n} - FE^{\prime}\pmb{e}_{n})(\pmb{e}_{n} - \pmb{\alpha}_{n})^{\top}$ . If  $\lambda_{1}$  is small, the modification is small as well.
Please check Appendix D for the proof. Lemma 3 shows that for every fixed  $n$ , only the corresponding row of  $W$  is updated, which makes the analysis much easier. We now can calculate the backpropagated gradient used in Eqn. 3.
Theorem 1. If Assumption 2 holds, the initial condition  $Y(0) = 0$ ,  $M \gg 100$ ,  $\eta_{Y}$  satisfies  $M^{-0.99} \ll \eta_{Y} < 1$ , and each sequence class appears uniformly during training, then after  $t \gg K^2$  steps of batch size 1 update, given event  $x_{T+1}[i] = n$ , the backpropagated gradient  $g[i] := Y(\pmb{x}_{T+1}[i] - \pmb{\alpha}[i])$  takes the following form:
$$
\boldsymbol {g} [ i ] = \gamma \left(\iota_ {n} \boldsymbol {f} _ {n} - \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \boldsymbol {f} _ {n ^ {\prime}}\right) \tag {9}
$$
Here the coefficients  $\iota_n(t),\beta_{nn'}(t)$  and  $\gamma (t)$  are defined in Appendix with the following properties:
-  $(\pmb{a})$ $\xi_{n}(t)\coloneqq \gamma (t)\sum_{n\neq n^{\prime}}\beta_{nn^{\prime}}(t)\pmb{f}_{n}^{\top}(t)\pmb{f}_{n^{\prime}}(t) > 0$  for any  $n\in [K]$  and any  $t$
-  $(\pmb{b})$  The speed control coefficient  $\gamma(t) > 0$  satisfies  $\gamma(t) = O(\eta_Y t / K)$  when  $t \leq \frac{\ln(M) \cdot K}{\eta_Y}$  and  $\gamma(t) = O\left(\frac{K \ln(\eta_Y t / K)}{\eta_Y t}\right)$  when  $t \geq \frac{2(1 + \delta') \ln(M) \cdot K}{\eta_Y}$  with  $\delta' = \Theta\left(\frac{\ln \ln M}{\ln M}\right)$ .
In the remark of Lemma 5 in Appendix, we analyze the original dynamics (Eqn. 7) with identical off-diagonal elements of  $E$ , and Theorem 1 still holds with a smaller effective learning rate.
5 The dynamics of Self-attention
Now we analyze the dynamics of self-attention logits  $Z$ , given the dynamics of upper layer  $Y$ .
Lemma 4 (Self-attention dynamics). With Assumption 1(b) (i.e.,  $T \to +\infty$ ), Eqn. 4 becomes:
$$
\dot {\boldsymbol {z}} _ {m} = \eta_ {Z} \gamma \sum_ {n \in \psi^ {- 1} (m)} \operatorname {d i a g} \left(\boldsymbol {f} _ {n}\right) \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \left(\boldsymbol {f} _ {n} \boldsymbol {f} _ {n} ^ {\top} - I\right) \boldsymbol {f} _ {n ^ {\prime}}, \tag {10}
$$
Please check Appendix E for the proof. Now we study the dynamics of two types of contextual tokens (Sec. 3.2), namely distinct tokens (DT) which appear only for a single next token (i.e.,  $|\Omega(l)| = 1$  with  $\Omega(l) \coloneqq \{n : \mathbb{P}(l|n) > 0\}$ ), and common tokens (CT) that appear across multiple next tokens ( $|\Omega(l)| > 1$ ). We show their fates are very different: over training, distinct tokens gain attention but common ones lose it.
Theorem 2 (Fates of contextual tokens). Let  $G_{CT}$  be the set of common tokens (CT), and  $G_{DT}(n)$  be the set of distinct tokens (DT) that belong to next token  $n$ . Then if Assumption 2 holds, under the self-attention dynamics (Eqn. 10), we have:
-  $(\mathbf{a})$  for any distinct token  $l \in G_{DT}(n)$ ,  $\dot{z}_{ml} > 0$  where  $m = \psi(n)$ ;
-  $(\pmb{b})$  if  $|G_{CT}| = 1$  and at least one next token  $n \in \psi^{-1}(m)$  has at least one distinct token, then for the single common token  $l \in G_{CT}$ ,  $\dot{z}_{ml} < 0$ .
Now we know DTs grow and a single CT will shrink. For multiple CTs to shrink, the condition can be a bit involved (see Corollary 2 in Appendix E). The following theorem further shows that the growth rates of DTs critically depend on their initial conditions:
Theorem 3 (Growth of distinct tokens). For a next token  $n$  and its two distinct tokens  $l$  and  $l'$ , the dynamics of the relative gain  $r_{l / l'|n}(t) \coloneqq f_{nl}^2(t) / f_{nl'}^2(t) - 1 = \tilde{c}_{l|n}^2(t) / \tilde{c}_{l'|n}^2(t) - 1$  has the following analytic form (here the query token  $m = \psi(n)$  and is uniquely determined by distinct token  $l$ ):
$$
r _ {l / l ^ {\prime} | n} (t) = r _ {l / l ^ {\prime} | n} (0) e ^ {2 \left(z _ {m l} (t) - z _ {m l} (0)\right)} =: r _ {l / l ^ {\prime} | n} (0) \chi_ {l} (t) \tag {11}
$$
where  $\chi_l(t) \coloneqq e^{2(z_{ml}(t) - z_{ml}(0))}$  is the growth factor of distinct token  $l$ . If there exist a dominant token  $l_0$  such that the initial condition satisfies  $r_{l_0 / l|n}(0) > 0$  for all its distinct token  $l \neq l_0$ , and all of its common tokens  $l$  satisfy  $\dot{z}_{ml} < 0$ . Then both  $z_{ml_0}(t)$  and  $f_{nl_0}(t)$  are monotonously increasing over  $t$ , and
$$
e ^ {2 f _ {n l _ {0}} ^ {2} (0) B _ {n} (t)} \leq \chi_ {l _ {0}} (t) \leq e ^ {2 B _ {n} (t)} \tag {12}
$$
here  $B_{n}(t)\coloneqq \eta_{Z}\int_{0}^{t}\xi_{n}(t^{\prime})\mathrm{d}t^{\prime}$ . Intuitively, larger  $B_{n}$  gives larger  $r_{l_0 / l|n}$  and sparser attention map.
Self-attention as an algorithm of token scanning. From Eqn. 11, we could see that self-attention performs token scanning. To see that, consider the simplest initialization that  $z(0) = 0$ , which means that  $r_{l_0 / l|n}(0) = \left(\frac{\mathbb{P}(l_0|m,n)}{\mathbb{P}(l|m,n)}\right)^2 - 1$ . Therefore, distinct token  $l$  with low conditional probability  $\mathbb{P}(l|m,n)$  will have  $r_{l_0 / l|n}(0) \gg 0$ , According Eqn. 11, this leads to quickly growing ratio  $r_{l_0 / l|n}(t)$  which means that the corresponding component  $f_{nl}$  will be quickly dwarfed by the dominating component  $f_{nl_0}$ . On the other hand, token with high conditional probability  $\mathbb{P}(l|m,n)$  will have smaller  $r_{l_0 / l|n}(0)$ , and the ratio  $r_{l_0 / l|n}(t)$  grows slower, costing longer time for  $l_0$  to dominate  $l$ .
Initial value as prior information. From the theorems, it is clear that the initial value  $r_{l / l^{\prime}|n}(0) \coloneqq \left(\frac{\mathbb{P}(l|m,n)\exp(z_{ml}(0))}{\mathbb{P}(l^{\prime}|m,n)\exp(z_{ml^{\prime}}(0))}\right)^{2} - 1$  critically determines the fate of the dynamics. Two tokens  $l$  and  $l^{\prime}$  with comparable conditional probability  $\mathbb{P}(l|m,n)$  and  $\mathbb{P}(l^{\prime}|m,n)$  can be suppressed in either way, depending on their initial logits  $z_{ml}(0)$  and  $z_{ml^{\prime}}(0)$ . In the empirical implementation, the initial value of the logits are determined by the inner products of independently initialized high-dimensional vectors, which fluctuate around zero.
The concept of "initial value as prior" can explain empirical design choices such as multi-head self-attention [1]. From this perspective, each head  $h$  has its own  $Z_{h}$  and is initialized independently, which could enable more diverse token combination (e.g., a combination of 1st, 3rd, 5th tokens, rather than a combination of 1st, 2nd, 3rd tokens).
6 The Moment of Snapping: When Token Combination is fixed
Theorem 3 suggests two possible fates of the self-attention weights: if  $\xi_{n}(t)$  decays slowly (e.g.,  $\xi_{n}(t) \geq 1 / t$ ), then  $B_{n}(t) \to +\infty$  and all contextual tokens except for the dominant one will drop (i.e.,  $f_{nl} \to 0$ ) following the ranking order of their conditional probability  $\mathbb{P}(l|m,n)$ . Eventually, winner-takes-all happens. Conversely, if  $\xi_{n}(t)$  drops so fast that  $B_{n}(t)$  grows very slowly, or even has an upper limit, then the self-attention patterns are "snapped" and token combination is learned and fixed.
The conclusion is not obvious, since  $\xi_{n}(t)$  depends on the decay rate of  $\gamma (t)$  and  $\beta_{nn^{\prime}}(t)$ , which in turns depends on the inner product  $\pmb{f}_n^\top (t)\pmb{f}_{n'}(t)$ , which is related to the logits of the common tokens that also decays over time.
Here we perform a qualitative estimation when there is only a single common token  $l$  and every next token shares a single token  $m$  (i.e., for any next token  $n$ ,  $\psi(n) = m$ ). We assume all normalization
Figure 3: Growth factor  $\chi_{l}(t)$  (Theorem 3) over time with fixed  $\eta_Z = 0.5$  and changing  $\eta_Y$ . Each solid line is  $\chi_{l}(t)$  and the dotted line with the same color corresponds to the transition time  $t_0$  for a given  $\eta_Y$ .
terms in  $f_{n}$  are approximately constant, denoted as  $\rho_0$ , which means that  $f_{n}^{\top}f_{n^{\prime}}\approx \exp (2z_{ml}) / \rho_0^2$  and  $\beta_{nn^{\prime}}\approx E_{nn^{\prime}}^{\prime}\approx f_{n}^{\top}f_{n^{\prime}}\approx \exp (2z_{ml}) / \rho_0^2$  as well, and  $1 - f_n^\top f_{n'}\approx 1$  due to the fact that common token components are small, and will continue to shrink during training.
Under these approximations, its dynamics (Eqn. 10) can be written as follows (here  $C_0 \coloneqq \rho_0^4 / K$ ):
$$
\dot {z} _ {m l} = \eta_ {Z} \gamma \sum_ {n \in \psi^ {- 1} (m)} f _ {n l} \sum_ {n ^ {\prime} \neq n} \beta_ {n n ^ {\prime}} \left(f _ {n l} ^ {2} - 1\right) f _ {n l ^ {\prime}} \approx - C _ {0} ^ {- 1} \eta_ {Z} \gamma e ^ {4 z _ {m l}}, \quad \xi_ {n} (t) \approx C _ {0} ^ {- 1} \gamma e ^ {4 z _ {m l}} \tag {13}
$$
Surprisingly, we now find a phase transition by combining the rate change of  $\gamma (t)$  in Theorem 1:
Theorem 4 (Phase Transition in Training). If the dynamics of the single common token  $z_{ml}$  satisfies  $\dot{z}_{ml} = -C_0^{-1}\eta_Z\gamma(t)e^{4z_{ml}}$  and  $\xi_n(t) = C_0^{-1}\gamma(t)e^{4z_{ml}}$ , then we have:
$$
B _ {n} (t) = \left\{ \begin{array}{c c} \frac {1}{4} \ln \left(C _ {0} + \frac {2 (M - 1) ^ {2}}{K M ^ {2}} \eta_ {Y} \eta_ {Z} t ^ {2}\right) & t <   t _ {0} ^ {\prime} := \frac {K \ln M}{\eta_ {Y}} \\ \frac {1}{4} \ln \left(C _ {0} + \frac {2 K (M - 1) ^ {2}}{M ^ {2}} \frac {\eta_ {Z}}{\eta_ {Y}} \ln^ {2} (M \eta_ {Y} t / K)\right) & t \geq t _ {0} := \frac {2 (1 + o (1)) K \ln M}{\eta_ {Y}} \end{array} \right. \tag {14}
$$
As a result, there exists a phase transition during training:
- Attention scanning. At the beginning of the training,  $\gamma(t) = O(\eta_Y t / K)$  and  $B_n(t) \approx \frac{1}{4} \ln K^{-1} (\rho_0^4 + 2\eta_Y \eta_Z t^2) = O(\ln t)$ . This means that the growth factor for dominant token  $l_0$  is (sub-)linear:  $\chi_{l_0}(t) \geq e^{2f_{nl_0}^2(0) B_n(t)} \approx [K^{-1} (\rho_0^4 + 2\eta_Y \eta_Z t^2)]^{0.5 f_{nl_0}^2(0)}$ , and the attention on less co-occurred token drops gradually.
- Attention snapping. When  $t \geq t_0 \coloneqq 2(1 + \delta') K \ln M / \eta_Y$  with  $\delta' = \Theta(\frac{\ln \ln M}{\ln M})$ ,  $\gamma(t) = O\left(\frac{K \ln (\eta_Y t / K)}{\eta_Y t}\right)$  and  $B_n(t) = O(\ln \ln t)$ . Therefore, while  $B_n(t)$  still grows to infinite, the growth factor  $\chi_{l_0}(t) = O(\ln t)$  grows at a much slower logarithmic rate.
See proof in Appendix F. This gives a few insights about the training process: (a) larger learning rate  $\eta_{Y}$  of the decoder  $Y$  leads to shorter phase transition time  $t_0\approx 2K\ln M / \eta_Y$ , (b) scaling up both learning rate  $(\eta_{Y}$  and  $\eta_Z)$  leads to larger  $B_{n}(t)$  when  $t\to +\infty$ , and thus sparser attention maps, and (c) given fixed  $\eta_{Z}$ , small learning rate  $\eta_{Y}$  leads to larger  $B_{n}(t)$  when  $t\geq t_0$ , and thus sparser attention map. Fig. 3 shows numerical simulation results of the growth rate  $\chi_l(t)$ . Here we set  $K = 10$  and  $M = 1000$ , and we find smaller  $\eta_{Y}$  given fixed  $\eta_{Z}$  indeed leads to later transition and larger  $B_{n}(t)$  (and  $\chi_l(t)$ ).
Positional encoding. While our main analysis does not touch positional encoding, it can be added easily following the relative encoding schemes that adds a linear bias when computing self attention (E.g., T5 [17], ALiBi [85], MusicTransformer [86]). More specifically, the added linear bias  $\exp(z_{ml} + z_0) = \exp(z_{ml})\exp(z_0)$  corresponds to a prior of the contextual token to be learned in the self-attention layer.
Figure 4: Visualization of  $c_{n}$  ( $n = 1,2$ ) in the training dynamics of 1-layer Transformer using SGD on Syn-Small setting. Top row for query token  $n = 1$  and bottom row for query token  $n = 2$ . Left: SGD training with  $\eta_{Y} = \eta_{Z} = 1$ . Attention pattern  $c_{n}$  becomes sparse and concentrated on highest  $\mathbb{P}(l|n)$  (rightmost) for each sequence class (Theorem 3). Right: SGD training with  $\eta_{Y} = 10$  and  $\eta_{Z} = 1$ . With larger  $\eta_{Y}$ , convergence becomes faster but the final attention maps are less sparse (Sec. 6).
Residue connection. Residue connection can be added in the formulation, i.e.,  $\bar{u}_T = \mathrm{LN}(\mathrm{LN}(\tilde{\pmb{u}}_T) + \pmb{u}_{x_T})$ , where  $\tilde{\pmb{u}}_T$  is defined in Eqn. 1, and  $\bar{u}_T$  is used instead in the objective (Eqn. 2). In this case, the  $\beta_{nn'}$  in Theorem 1 now is approximately  $\beta_{nn'} \sim f_n^\top f_{n'} + \mathbb{I}(\psi(n) = \psi(n'))$ , which is much larger for sequence classes  $n$  and  $n'$  that share the same query token  $x_T$  than otherwise. In this case, Theorem 1 now gives  $\pmb{g}[i] = \gamma \left( \iota_n \pmb{f}_n - \sum_{n \neq n' \in \psi^{-1}(\psi(n))} \beta_{nn'} \pmb{f}_{n'} \right)$  for  $x_{T+1}[i] = n$ . Due to the additional constraint  $n' \in \psi^{-1}(\psi(n))$  (i.e.,  $n$  and  $n'$  shares the same query token), we can define local distinct and common tokens to be within the sequence class subset  $\psi^{-1}(m)$  and Theorem 2 now applies within each subset. Empirically this makes more sense, since the query token  $x_T = m_1$  or  $m_2$  alone can already separate different subsets  $\psi^{-1}(m_1)$  and  $\psi^{-1}(m_2)$  and there should not be any interactions across the subsets. Here we just present the most straightforward analysis and leave this extension for future work.
Possible future extension to multi-layer cases. For multilayer training, a lasting puzzle is to explain how the input tokens get combined together to form high-level concepts. The analysis above shows that the training leads to sparse attention even among relevant tokens, and demonstrates that there is a priority in token combinations for 1-layer attention based on their co-occurrence: even if there are 10 relevant contextual tokens to the query, the self-attention may only pick 1-2 tokens to combine first due to attention sparsity. This can be regarded as a starting point to study how tokens are composed hierarchically. In comparison, [28, 29, 30] show that attention attends to all relevant tokens, which may not suggest a hierarchical / multi-layer architecture.
We conduct experiments on both synthetic and real-world dataset to verify our theoretical findings.
Syn-Small. Following Sec. 3.2, we construct  $K = 2$  sequence classes with vocabulary size  $M = 30$ . The first 10 tokens (0-9) are shared between classes, while the second and third 10 tokens (10-19 and 20-29) are distinct for class 1 and class 2, respectively. The conditional probability  $\mathbb{P}(l|n)$  for tokens 10-19 is increasing monotonously (the same for 20-29). The 1-layer Transformer is parameterized with  $Y$  and  $Z$  (Sec. 3.1), is trained with initial condition  $Y(0) = Z(0) = 0$  and SGD (with momentum 0.9) using a batchsize of 128 and sequence length  $T = 128$  until convergence.
Fig. 4 shows the simulation results. The attention indeed becomes sparse during training, and increasing  $\eta_{Y}$  with fixed  $\eta_{Z}$  leads to faster convergence but less sparse attention. Both are consistent with our theoretical predictions (Theorem 3 and Sec. 6). Interestingly, if we use Adam optimizer instead, self-attention with different learning rate  $\eta_{Y} = \eta_{Z}$  picks different subsets of distinct tokens to focus on, showing tune-able inductive bias (Fig. 5). We leave analysis on Adam for future work.
Syn-Medium. To further verify our theoretical finding, we now scale up  $K$  to create Syn-Medium and compute how attention sparsity for distinct tokens (in terms of entropy) changes with the learning rates (Fig. 6). We can see indeed the entropy goes down (i.e., attention becomes sparser) with larger  $\eta_Z$ , and goes up (i.e., attention becomes less sparse) by fixing  $\eta_Z$  and increasing  $\eta_Y$  passing the threshold  $\eta_Y / \eta_Z \approx 2$ , consistent with Sec. 6. Note that the threshold is due to the fact that our theory is built on Assumption 1(c), which requires  $\eta_Y$  to be reasonably larger than  $\eta_Z$ .
Figure 5: Visualization of (part of)  $c_n$  for sequence class  $n = 1$  in the training dynamics using Adam [22] on Syn-Small setting. From left to right:  $\eta_V = \eta_Z = 0.1, 0.5, 1$ . With different learning rate Adam seems to steer self-attention towards different subset of distinct tokens, showing tune-able inductive bias.
Figure 6: Average entropy of  $c_{n}$  (Eqn. 5) on distinct tokens versus learning rate ratio  $\eta_{Y} / \eta_{Z}$  with more query tokens  $M$  /next tokens  $K$ . We report mean values over 10 seeds and standard derivation of the mean.
Figure 7: Attention patterns in the lowest self-attention layer for 1-layer (top) and 3-layer (bottom) Transformer trained on WikiText2 using SGD (learning rate is 5). Attention becomes sparse over training.
Real-world Dataset. We also test our finding on WikiText [25] using both 1-layer and multi-layer Transformers with regular parameterization that computes  $Y$  and  $Z$  with embedding  $U$ . In both cases, attentions of the first layer freeze (and become sparse) at some point (Fig. 7), even if the learning rate remains the same throughout training. More results are in Appendix G.
9 Conclusion and Future Work
In this work, we formally characterize SGD training dynamics of 1-layer Transformer, and find that the dynamics corresponds to a scan and snap procedure that progressively pays more attention to key tokens that are distinct and frequently co-occur with the query token in the training set. To our best knowledge, we are the first to analyze the attention dynamics and reveal its inductive bias on data input, and potentially open a new door to understand how Transformer works.
Many future works follow. According to our theory, large dataset suppresses spurious tokens that are perceived as distinct in a small dataset but are actual common ones. Our finding may help suppress such tokens (and spurious correlations) with prior knowledge, without a large amount of data.
[72] Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, pages 4376-4386. PMLR, 2020.
[73] Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022.
[74] Yuandong Tian. Understanding the role of nonlinearity in training dynamics of contrastive learning. arXiv preprint arXiv:2206.01342, 2022.
[75] Yuandong Tian. Understanding the role of nonlinearity in training dynamics of contrastive learning. *ICLR*, 2023.
[76] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33:15383-15393, 2020.
[77] Charlie Snell, Ruiqi Zhong, Dan Klein, and Jacob Steinhardt. Approximating how single head attention learns. arXiv preprint arXiv:2103.07601, 2021.
[78] Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. *ICLR*, 2020.
[79] Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:2110.09456, 2021.
[80] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[81] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[82] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[83] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[84] Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. arXiv preprint arXiv:2305.19466, 2023.
[85] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409, 2021.
[86] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. arXiv preprint arXiv:1809.04281, 2018.
[87] Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. Advances in neural information processing systems, 32, 2019.