SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(R4) Training on rare sequences works best, by a wide margin. By training on a uniform mixture of FFL distributions with  $p_{\mathrm{i}} = \{0.9, 0.98, 0.1\}$ , the baseline architecture reliably converges to solutions with significantly fewer errors on each of these 3 distributions (teal violins in Figure 4). In 6 out of 25 runs, we did not detect a single error.
This should not be surprising, in light of the realizability of flip-flops by self-attention (and, more generally, the existence of shortcuts functionally identical to RNNs (Liu et al., 2023)), and corroborates similar conclusions from (Zhang et al., 2021). We also find that weaker improvements emerge by straightforwardly increasing scale parameters in the model and training pipelines:
(R5) Resource scaling (in-distribution data, training steps, network size) helps. However, the improvements are orders of magnitude smaller than those in (R4), and we observe tradeoffs between sparse- and dense-sequence extrapolation; see the blue violins in Figure 4.
Another class of direct solutions is to externalize the chain of thought (CoT): train (or finetune, or prompt) the model to explicitly output the intermediate reasoning steps (Nye et al., 2021; Wei et al., 2022b). We do not investigate this strategy in this paper, and note that prior work has provided sufficient evidence to affirm its efficacy in inducing the robust learning of recurrences on long synthetic sequences (Anil et al., 2022; Zhou et al., 2022; Liu et al., 2023). Even when applying CoT in practice, we believe attention glitches may still occur, as flip-flops operations may be embedded within a single indivisible reasoning step. Thus, the focus of this work is to isolate and mitigate this intrinsic architectural issue. We provide additional references and discussion in Appendix A.2.
5.2 Indirect algorithmic controls: a bag of regularization tricks
The interventions listed in Section 5.1 are all potentially practical, and may shed light on how closed-domain LLM hallucinations will diminish with data quality, scale, and improved inference strategies. However, it is not always feasible to implement these fixes under resource constraints (especially data). We next investigate an orthogonal design space, of how to robustify the internal memory mechanisms of neural sequence models. Note that the exceptionally strong extrapolative performance of the LSTM provides a "skyline", showing the possibility of far more robust architectures than the Transformer (in the flip-flop setting, with this restricted set of considerations).
Standard regularization heuristics. There is a large array of not-fully-understood algorithmic tricks for "smoothing" the behavior of LLMs. We test the extrapolative behavior of models trained with weight decay and dropout (at the attention, feedforward, and embedding layers), as well as a host of algorithmic choices known to modulate generalization (batch sizes, learning rates, optimizer
Figure 4: A long tail of flip-flop errors for 10,625 Transformer models. Left: Out-of-distribution evaluations for all models; some algorithmic choices help substantially (note the logarithmic axes), but nothing we tried, aside from training on o.o.d. data, could fully eliminate attention glitches. Right: Effects of individual architectural and algorithmic choices on both types of extrapolation (sparse and dense sequences). Some configurations reduce attention glitch rates by orders of magnitude. Horizontal marks denote {min,  $25\%$ , median,  $75\%$ , max} test errors on  $>3 \times 10^5$  predictions, over 25 replicates (500 for the baseline model). Dots at the bottom indicate runs with 0 error.
hyperparameters, position embeddings, activation functions). Due to the extreme variability noted in (R1), we quantify effects on extrapolation by training and evaluating at least 25 replicates for each choice under consideration.
Attention sharpening: a non-standard regularization technique. Inspired by the "diluted hard attention" calculation in Section 4.1, and the fact that the attention heads of trained models do not attend sharply (see Figure 5), we train Transformer models with attention-sharpening regularizers: during training, for attention weights  $\alpha \in \Delta ([T])$ , adding differentiable loss terms which encourage sparsity (e.g. the mixture's entropy  $H(\alpha)$ , or negative  $p$ -norms  $-\| \alpha \|_2, -\| \alpha \|_\infty$ ).
(R6) Many algorithmic choices influence extrapolative behaviors. We find that some architectural variants and regularization tricks have orders-of-magnitude effects on the out-of-distribution performance of Transformers; see the purple, brown, red, and gold violins in Figure 4 (right). Our strongest improvements on sparse sequences are obtained by large (0.5) embedding dropout and attention-sharpening losses; on dense sequences, non-trainable position embeddings are the most helpful.
(R7) Despite many partial mitigations, nothing eliminates attention glitches entirely. The scatter plot in Figure 4 (left) gives an overview of our entire search over architectures and hyperparameters, showing (dense-sequence error, sparse-sequence error) pairs for every model we trained. We found it extremely difficult to find a setting that reliably produces Transformer models with simultaneous improvements over the baseline on sparse and dense sequences. Recall that it is trivial to do so with an LSTM model.
5.3 Preliminary mechanistic study of the trained networks
In this section, we move to a simpler setting to gain finer-grained understanding of how sparsity regularization affects the learned solutions. Specifically, we look at the task of simulating the flip-flip automaton (Definition 1), whose inputs consist of  $\{\sigma_0,\sigma_1,\bot \}$  as two types of write and 1 no-op. This task (elaborated in Appendix A.1) can be solved by a 1-layer Transformer with a single attention head which attends sparsely on the most recent write position. It also serves as a building block for more complex tasks (Liu et al., 2023), hence observations from this simple setup can potentially be useful in broader contexts.
<sup>9</sup>While less popular, such losses have been used to sparsify dependencies in similar contexts (Zhang et al., 2018; Sukhbaatar et al., 2021).
Figure 5: Causal attention patterns for flip-flop simulation (Definition 1); orange dots / blue diamonds mark the positions of write tokens  $\sigma_0 / \sigma_1$ . (a), (b) are subselected respectively from a regular (non-sparse) and a sparse multi-layer model (details in Appendix B.5). (c), (d) are from two 1-layer 1-head models. The attention pattern highlighted by the purple box in (b) coincides with the "ideal" attention pattern in (c). However, sparse models can be wrong, as shown in (d) (error marked in red).
Figure 5 shows examples of attention patterns on the flip-flop simulation task, subselected from 6-layer 8-head models trained with and without attention-sharpening regularization. It is evident that the attention patterns of the sparse model are less complex and easier to interpret compared to those of the un-regularized model. For example, we can identify one head in the sparse model that exactly coincide with the attention pattern $^{10}$  that an "ideal" 1-layer 1-head model implements (Figure 5c).
(R8) Attention-sharpening regularizers successfully promote hard attention, but errors persist. As mentioned in (R7), attention-sharpening regularization cannot fully eliminate the sporadic errors, which are partially induced by the complexity and redundancy of attention patterns. Moreover, sharpened attention can induce additional failure modes, such as confidently attending to incorrect write positions. An example is demonstrated in Figure 5d, where the attention focuses on an initial write, likely caused by the fact that earlier positions are overemphasized due to the use of causal attention masks. Another example occurs in length generalization, where the attention is correct at positions earlier in the sequence, but starts to confidently focus on wrong positions as it moves towards later positions (Proposition 4).
In a similar spirit to concurrent work on generating Dyck languages (Wen et al., 2023) (a more complex capability which also requires parallel simulation of memory registers), these glitchy solutions point to a concrete obstruction to mechanistic interpretability. Due to factors such as overparameterization, spurious solutions, and the opaqueness of optimization dynamics, learned neural implementations of algorithms generally do not coincide with "ideal", "minimal", or "natural" theoretical constructions. Details for these experiments and further discussion are provided in Appendix B.5.
6 Conclusion and future challenges
We have introduced flip-flop language modeling (FFLM), a synthetic benchmark for probing the fine-grained extrapolative behavior of neural sequence models, based on a one-bit memory operation which forms a fundamental building block of algorithmic reasoning. Despite being able to realize this operation trivially, Transformer models do not extrapolate robustly: they exhibit a long tail of sporadic reasoning errors, which we call attention glitches. Through extensive controlled experiments, we find that many algorithmic mitigations can reduce the frequency of attention glitches, but only recurrence and training on longer-tailed data work perfectly. FFLM provides a concrete and minimalistic setting in which Transformers are far inferior to recurrent sequence models, with respect to multiple criteria (efficiency, stability, and extrapolation).
10While it is well-known that attention patterns can be misleading at times (Jain and Wallace, 2019; Bolukbasi et al., 2021; Meister et al., 2021), they do provide upper bounds on the magnitude of the dependency among tokens. These upper bounds are particularly useful in the case of (1-)sparse attention: a (near) zero attention weight signifies the absence of dependency, which greatly reduces the set of possible solutions implemented.
What does this entail about hallucinations in natural LLMs? The motivating issue for this work is the phenomenon of "closed-domain hallucinations" in non-synthetic LLMs (e.g. the errors demonstrated in Figure 1). We hypothesize that attention glitches occur in the internal algorithmic representations of Transformer models of natural language, and that they account for (a non-negligible portion of) the reasoning errors encountered in practice. To our knowledge, this is the first attempt to attribute model hallucinations to a systematic architectural flaw in the Transformer. However, confirming or refuting this hypothesis is far outside the scope of this paper; the opaque indirections and lack of adequate controls on the training data present significant methodological challenges. Even precisely articulating this hypothesis leaves degrees of freedom which are difficult to resolve; see the discussion in Appendix A.4. We therefore leave these topics for future work.
Paths to hallucination-free Transformers? Our findings suggest that in the near term, there are many mutually-compatible approaches for reducing the frequency of attention glitches: data (particularly with high diversity), scale, and various forms of regularization. Yet, the strikingly outsized benefit of replacing the Transformer with an LSTM network suggests that architectural innovations towards the same ends are well worth examining. Obtaining a practical best-of-both-worlds architecture is a grand open challenge, for which new recurrent designs (Katharopoulos et al., 2020; Dao et al., 2022; Peng et al., 2023; Anonymous, 2023) show great promise. Note that we do not make the claim that recurrent architectures are the only ones which can extrapolate robustly.[11]
Broader impacts and limitations. This work is inherently foundational, and focuses on precise measurements of generalization in an idealized setting; see Appendix A.4 for a discussion of the limitations this entails. By introducing methodologies to isolate, measure, and control the long tail of reasoning errors in neural sequence models, we hope that this work will contribute to the systematic and principled discovery of LLM pipelines with improved factual reliability. Such improvements may result in unintended downstream consequences, such as higher-fluency malicious content generation.
We would like to thank Yuchen Li for helpful discussions and comments on the paper.
Abnar, S., Dehghani, M., and Zuidema, W. (2020). Transferring inductive biases through knowledge distillation. arXiv preprint arXiv:2006.00555.
Ahn, K., Cheng, X., Song, M., Yun, C., Jabbabaie, A., and Sra, S. (2023). Linear attention is (maybe) all you need (to understand transformer optimization). arXiv preprint arXiv: 2310.01082.
Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V., Slone, A., Gur-Ari, G., Dyer, E., and Neyshabur, B. (2022). Exploring length generalization in large language models. arXiv preprint arXiv:2207.04901.
Anonymous (2023). Mamba: Linear-time sequence modeling with selective state spaces. In Submitted to The Twelfth International Conference on Learning Representations. under review.
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. (2022). Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764.
Barrington, D. A. M. and Thérien, D. (1988). Finite monoids and the fine structure of  $\mathsf{NC}^1$ . Journal of the ACM.
In particular, for algorithmic reasoning capabilities corresponding to the recurrent execution of deterministic finite-state machines, the results of (Liu et al., 2023) imply that the Transformer has a "reparameterized" recurrent inductive bias, which parallelizes and hierarchizes any looped recurrence.
Wei, C., Chen, Y., and Ma, T. (2022a). Statistically meaningful approximation: a case study on approximating Turing machines with transformers. Advances in Neural Information Processing Systems, 35:12071-12083.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022b). Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
Wen, K., Li, Y., Liu, B., and Risteski, A. (2023). Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars. In Thirty-seventh Conference xon Neural Information Processing Systems.
Wolf, T., Debut, L., Sanh, V., Chaumont, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019). Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Wu, Y., Rabe, M. N., Hutchins, D. S., and Szegedy, C. (2022). Memorizing transformers. International Conference On Learning Representations.
Wu, Y., Rabe, M. N., Li, W., Ba, J., Grosse, R. B., and Szegedy, C. (2021). Lime: Learning inductive bias for primitives of mathematical reasoning. In International Conference on Machine Learning, pages 11251-11262. PMLR.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016). Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Yao, S., Peng, B., Papadimitriou, C., and Narasimhan, K. (2021). Self-attention networks can process bounded hierarchical languages. arXiv preprint arXiv:2105.11115.
Zeiger, H. P. (1967). Cascade synthesis of finite-state machines. Information and Control.
Zhang, C., Raghu, M., Kleinberg, J., and Bengio, S. (2021). Pointer value retrieval: A new benchmark for understanding the limits of neural network generalization. arXiv:2107.12580.
Zhang, J., Zhao, Y., Li, H., and Zong, C. (2018). Attention with sparsity regularization for neural machine translation and summarization. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3):507-518.
Zhang, M., Press, O., Merrill, W., Liu, A., and Smith, N. A. (2023). How language model hallucinations can snowball. arXiv preprint arXiv: 2305.13534.
Zhang, Y., Backurs, A., Bubeck, S., Eldan, R., Gunasekar, S., and Wagner, T. (2022). Unveiling transformers with lego: a synthetic reasoning task. arXiv preprint arXiv:2206.04301.
Zhao, H., Panigrahi, A., Ge, R., and Arora, S. (2023). Do transformers parse while predicting the masked word? arXiv preprint arXiv:2303.08117.
Zheng, C., Liu, Z., Xie, E., Li, Z., and Li, Y. (2023). Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv.2304.09797.
Zhou, H., Nova, A., Larochelle, H., Courville, A., Neyshabur, B., and Sedghi, H. (2022). Teaching algorithmic reasoning via in-context learning. arXiv preprint arXiv:2211.09066.
A Deferred background and discussion 17
A.1 Flip-flop terminology and history 17
A.2 Additional related work 18
A.3 Why this flip-flop language? 19
A.4 Attention glitches in natural LLMs 19
B Full experimental results 20
B.1 Details for LLM addition prompts (Figure 1) 20
B.2 Extrapolation failures of standard Transformers (Section 4) 21
B.3Effects of training data and scale (Section 5.1) 24
B.4 Indirect algorithmic controls for extrapolation (Section 5.2) 25
B.5 Preliminary mechanistic study and challenges (Section 5.3) 27
B.6 Software, compute infrastructure, and resource costs 32
C Proofs for Section 4.1 32
C.1 Realizability of FFL by small Transformers 32
C.2 Failure of soft attention: attention dilution with bounded Lipschitzness 34
C.3 Failure of hard attention: bad margin for positional embeddings 34
A Deferred background and discussion
A.1 Flip-flop terminology and history
The flip-flop automaton<sup>12</sup> is a two-state machine which remembers a single bit of memory, and enables retrieval of this bit. More precisely, the flip-flop automaton (illustrated in Figure 2(a)) is defined as:
Definition 1 (Flip-flop automaton). A flip-flop automaton  $\mathcal{A} = \{Q, \Sigma, \delta\}$  is defined with state space  $Q = \{0, 1\}$ , input alphabet  $\Sigma = \{\sigma_0, \sigma_1, \bot\}$ , and transition function  $\delta: Q \times \Sigma \to Q$  where
$$
\left\{ \begin{array}{l l} \delta (q, \sigma_ {0}) & = 0, \\ \delta (q, \sigma_ {1}) & = 1, \\ \delta (q, \bot) & = q; \end{array} \right. \qquad \forall q \in \{0, 1 \}.
$$
The semantics of the input symbols can be intuitively be identified with "write 0", "write 1", and "do nothing". This mathematical object is named after a type of electronic circuit which can store a single bit of state information (Eccles and Jordan, 1918, 1919); such physical constructions appear ubiquitously in electrical engineering as the building blocks of memory. The task of interest in Section 5.3 is simulating the flip-flop automaton: the model takes as input a sequence of  $\sigma_{1},\sigma_{2},\dots,\sigma_{T}\in \Sigma$  and learns to output the corresponding states  $q_{t}\in Q$  for each  $t\in [T]$  after processing inputs  $\sigma_{1:t}$ .
Naturally associated with the flip-flop automaton is its transformation monoid, the closure<sup>13</sup> of its state transformations  $\delta(\cdot, \sigma): \bar{Q} \to Q$  under function composition. Identifying each symbol with its state transformation map, we can compute the multiplication table of this monoid ( $f \circ g$  for every pair of transformations  $f, g$ ):
g = σ0	g = σ1	g = ⊥
f = σ0	σ0	σ0	σ0
f = σ1	σ1	σ1	σ1
f = ⊥	σ0	σ1	⊥
This algebraic object is called the flip-flop monoid  $\mathcal{F}$ . Its binary operation  $\circ$  is clearly (1) non-invertible (intuitively: the history of the bit cannot be recovered after a "memory write") and (2) non-commutative (the order of "write" operations matters); it also has (3) an identity element  $\perp$  (which does nothing to the memory bit). By enumeration of smaller objects, it can be seen that  $\mathcal{F}$  is the smallest monoid (in terms of order  $|\mathcal{F}|$ , or fewest number of automaton states  $|Q|$ ) which has properties (1)-(3).
The flip-flop monoid plays a special role in the algebraic theory of automata (Rhodes et al., 2010): flip-flops can be cascaded to represent more complex functions. In particular, the celebrated Krohn-Rhodes theorem (Krohn and Rhodes, 1965) gives a "prime decomposition" theorem for all finite semigroups (associative binary operations), representing them as alternating wreath products of flip-flop monoids and finite simple groups. Further developments (Zeiger, 1967; Eilenberg, 1974) have interpreted this theorem as a structural reparameterization of any finite-state automaton into a feedforward hierarchy of simple "atomic" machines (namely, flip-flops and permutation semiautomata). Basic quantitative questions (e.g. "which functions of  $n$  variables can  $L$  layers of poly(  $n$  ) flip-flops represent?") have proven to be extremely hard to resolve; these are studied by the theories of Krohn-Rhodes complexity and circuit complexity.
It was noted by Barrington and Thérien (1988) that these reparameterizations of finite-state automata entail the existence of parallel algorithms (i.e. shallow, polynomial-sized circuits) for sequentially executing finite-state recurrences (thus, processing formal languages) on sequences of length  $T$ . More recently, Liu et al. (2023) establish implications for shallow Transformer neural networks: they show that they can size-efficiently (with depth  $O(\log T)$  and parameter count  $\Theta(T)$ ; sometimes both
<sup>12</sup>Sometimes, a distinction is made between a semiautomaton  $(Q, \Sigma, \delta)$  and an automaton, which is a semiautomaton equipped with a (not necessarily invertible) mapping from states to output symbols. We do not make such a distinction; we equip a semiautomaton with the output function which simply emits the state  $q$ , and use "automaton" to refer to this dynamical system.
In this case, the closure is the same as the generator set: no functions distinct from  $\sigma_0, \sigma_1, \bot$  can be obtained by composing these three functions. This is not true for a general automaton.
improvable to  $O(1)$ ) realize these parallel algorithms, and that standard gradient-based training can empirically learn  $\ll T$ -layer solutions on a variety of "hard" automata (e.g. composing a sequence of  $T$  5-element permutations; multiplying  $T$  unit quaternions). Here, the role of the flip-flop monoid is essential: it provides a natural way to think about the role of a single self-attention head in a hierarchy of indirections, in order to learn a depth-constrained parallel implementation of a sequential algorithm.
A.2 Additional related work
Relevant challenges in NLP: hallucinations and long-range dependencies. The empirical literature is rife with corroborations that neural language models have trouble with robustly fitting long-range memory and multi-step reasoning (Khandelwal et al., 2018; Sun et al., 2021; Sukhbaatar et al., 2021; Malkin et al., 2022; Saparov and He, 2022; Orvieto et al., 2023; Creswell et al., 2023). Such failures can result in "hallucinations": incorrect outputs which either directly contradict factual input in the context, or contain information absent in the context (Ji et al., 2023).
Hallucination can be attributed to various factors, such as the noisiness in data sources (Dhingra et al., 2019; Dziri et al., 2022), imperfect encoding/decoding (Parikh et al., 2020; Tian et al., 2019), or the discrepancy in training and evaluation setups (He et al., 2019). In particular, the most related to our paper are the characteristics inherent to the model itself. For example, prior work has found that Transformers tend to be biased towards information covered during training (Petroni et al., 2019; Longpre et al., 2021), a potential cause to their poor out-of-distribution performance, and may over-commit to their earlier errors (Zhang et al., 2023)