SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bingbin Liu $^{1}$  Jordan T. Ash $^{2}$  Surbhi Goel $^{3}$  Akshay Krishnamurthy $^{2}$  Cyril Zhang $^{2}$
$^{1}$ Carnegie Mellon University  $^{2}$ Microsoft Research NYC  $^{3}$ University of Pennsylvania
bingbin@cs.cmu.edu, surbhig@cis.upenn.edu, {ash.jordan, akshaykr, cyrilzhang}@microsoft.com
Why do large language models sometimes output factual inaccuracies and exhibit erroneous reasoning? The brittleness of these models, particularly when executing long chains of reasoning, currently seems to be an inevitable price to pay for their advanced capabilities of coherently synthesizing knowledge, pragmatics, and abstract thought. Towards making sense of this fundamentally unsolved problem, this work identifies and analyzes the phenomenon of attention glitches, in which the Transformer architecture's inductive biases intermittently fail to capture robust reasoning. To isolate the issue, we introduce flip-flop language modeling (FFLM), a parametric family of synthetic benchmarks designed to probe the extrapolative behavior of neural language models. This simple generative task requires a model to copy binary symbols over long-range dependencies, ignoring the tokens in between. We find that Transformer FFLMs suffer from a long tail of sporadic reasoning errors, some of which we can eliminate using various regularization techniques. Our preliminary mechanistic analyses show why the remaining errors may be very difficult to diagnose and resolve. We hypothesize that attention glitches account for (some of) the closed-domain hallucinations in natural LLMs.
Recent advancements in scale have yielded large language models (LLMs) with extraordinary proficiency in nuanced reasoning with factual knowledge. Despite these achievements, LLMs are known to produce incorrect outputs, often referred to colloquially as "hallucinations" or "distractions" (Ji et al., 2023). Generally, hallucinations refer to the phenomenon that a model's outputs are syntactically and grammatically accurate but factually incorrect. There are various types of hallucinations, and the focus of this work is the "closed-domain" variety (Saparov and He, 2022; OpenAI, 2023), where the model predictions contain factually incorrect or made-up information according to a given context, regardless of their correctness in the real world.
Perhaps surprisingly, such hallucinations can be observed even on simple algorithmic reasoning tasks. As a warmup, consider the queries shown in Figure 1 (and Appendix B.1), where we prompt LLMs to solve addition problems of various lengths. The responses simultaneously illustrate the following:
1. Nontrivial algorithmic generalization: In cases where the models succeed, it is unlikely that these exact numerical sequences appeared in the training data. To correctly output the first digit of the answer, the LLM must resolve a long dependency chain which generally depends on every digit in the input. Somewhere within these networks' internal representations, implementations of addition algorithms have emerged.
Figure 1: Cherry-picked integer addition prompts, showing how state-of-the-art LLMs can generalize non-trivially on algorithmic sequences, but sporadic reasoning errors persist. The first digit of the correct answer depends on every input; thus, an autoregressive model must propagate a "carry" bit across these long-range dependencies in a single pass. This (and many other algorithmic reasoning capabilities) can be implemented by a Transformer model using internal flip-flops.
2. Sporadic errors ("hallucinations"): These internal algorithms can be brittle and unreliable, especially when processing long inferential chains. Their failures can be subtle and unpredictable.
In this work, we consider the task of processing the flip-flop language, a minimal unit of sequential computation which consists of memory operations on a single bit (see Definition 1) and underlies virtually all syntactic parsing and algorithmic reasoning capabilities (including implementing adders, and far more). A flip-flop language modeling (FFLM) task is defined on sequences of write, read, and ignore instructions: write sets the memory state to a certain value which is later retrieved by read, while ignoring any contents in between. We find that when trained to complete flip-flop sequences, the Transformer architecture exhibits a long tail of reasoning errors (incorrect read retrievals), unlike previous-generation recurrent models such as the LSTM (Hochreiter and Schmidhuber, 1997). We coin the term attention glitch for this phenomenon, and hypothesize that this captures a systematic failure mode of Transformer-based LLMs when internally manifesting long chains of algorithmic reasoning.
- FFLM: a minimalistic long-range dependency benchmark. We propose flip-flop language modeling, a parametric family of synthetic benchmarks for autoregressive sequence modeling, designed to isolate and probe reasoning errors like those demonstrated in Figure 1. We view FFLM as a robust complement to the Long Range Arena (Tay et al., 2020) and some of the tests in BIG-Bench (Srivastava et al., 2022), and recommend measuring glitch rates as a "stress test" for architectural innovations in sequence modeling. $^2$
- Main empirical result: attention attends glitchily. We find that while Transformer models can appear to learn flip-flop languages perfectly on held-out samples from the training distribution, they make a long tail of unpredictable reasoning errors (attention glitches), on both long-range and short-range dependencies. We evaluate various direct and indirect mitigations, including commonly-used regularization techniques and attention-sharpening regularizers — a plug-and-play way to sparsify self-attention architectures. We find that attention sharpening reduces reasoning errors by an order of magnitude, but none of our attempts were successful in driving the number of errors to exactly 0. Meanwhile, recurrent models work perfectly. $^3$
- Preliminary mechanistic analyses. We provide some theoretical and empirical explorations which account for some of the internal mechanisms for attention glitches, and why they are so difficult to eliminate completely.
The challenge of learning long-range dependencies is a long-standing one in the statistical modeling of sequences (Samorodnitsky et al., 2007). The Transformer architecture (Vaswani et al., 2017), a paradigm-shifting sequence model, enables the scalable learning of a feedforward hierarchy of
<sup>1</sup>More precisely, whenever the desired algorithm needs to "store memory" (i.e. contains a non-invertible state transformation); see Section 3.2.
To get started, see our data release: https://huggingface.co/datasets/synthseq/flipflop.
<sup>3</sup>It could be the case that recurrent models may fail at extremely long dependencies (Khandelwal et al., 2018).
meaningful long-range dependencies. Yet, factual errors over long contexts persist in these models; this is the subject of many careful studies in deep NLP (Khandelwal et al., 2018; Tay et al., 2020; Guo et al., 2022; Ji et al., 2023).
The sporadic non-factual outputs of LLMs have been popularly called "hallucinations", especially when there is an expectation that producing a correct answer is disproportionately "easy". Popular approaches for improving robustness to such errors include chain-of-thought generation (explicitly outputting intermediate reasoning steps) (Nye et al., 2021; Wei et al., 2022b) and enforcing self-consistency (Wang et al., 2022). In the emerging taxonomy of LLM pathologies (Saparov and He, 2022; Ji et al., 2023), the hallucinations studied in this work are of the closed-domain variety, under a deterministic notion of factuality (namely, consistency with flip-flop transitions) which is unambiguously reflected by the training data. We provide further discussion on the connections to natural LLM hallucinations in Section 6 and Appendix A.4.
Long-range dependency and reasoning benchmarks. Many datasets and benchmarks have been designed to isolate qualitative issues in language modeling (Tay et al., 2020; Wu et al., 2021; Zhang et al., 2021, 2022; Saparov and He, 2022; Shi et al., 2023; van der Poel et al., 2023; Eldan and Li, 2023). Aside from being focused on the "smallest" and "purest" compositional unit of sequential reasoning (see Section 3.2), FFLM is distinguished by a few factors:
- “ $L_{\infty}$ ” objective: Unlike usual benchmarks, we consider any model with less than  $100\%$  accuracy as exhibiting a reasoning error. Aside from the motivation of completely eliminating hallucinations, we argue that this stringent notion of correctness is needed to avoid error amplification when flip-flops are embedded in more complex networks; see Appendix A.1 and Liu et al. (2023).
- Parametric, procedurally generated, and generalizable: Our empirical study precisely quantifies long-tail errors via a large number of replicates over the randomness of both model initialization and data generation. This methodology is easily adapted and rescaled (by adjusting  $T$ ,  $\mathbf{p}$ , and other difficulty knobs) to probe language models of any size.
We provide an expanded discussion of related literature in Appendix A.2.
2 Background and notation
Modern language models are powered by sequence-to-sequence (seq2seq) neural networks  $f_{\theta} : \mathbb{R}^{T \times d} \to \mathbb{R}^{T \times d}$ , which transduce sequences of vectors according to internal computations determined by the inputs as well as trainable parameters  $\theta$ . When equipped with mappings to and from symbolic tokens (an "embedding layer"  $E : [M] \to \mathbb{R}^d$  (here,  $M \in \mathbb{N}$  is the vocabulary size) and classification layer  $W : \mathbb{R}^d \to \Delta([M])$ , shared across positions),  $W \circ f \circ E : [M]^T \to \Delta([M])^T$  can represent an autoregressive generative model of a joint distribution over tokens  $x_{1:T} \in [M]^T$ , where the output at the  $t$ -th position gives the estimated next-symbol probabilities  $\widehat{\mathrm{Pr}}[x_{t+1} = \cdot | x_{1:t}]$ . The overarching challenge of statistical language modeling is to fit complex distributions such as natural language; recurrent (Elman, 1990; Hochreiter and Schmidhuber, 1997; Wu et al., 2016) and self-attention-based (Vaswani et al., 2017) architectures have shown remarkable capabilities in fitting the seq2seq functions necessary for fluent linguistic parsing and reasoning.
Recurrent inductive biases, attention, and length generalization. To correctly process uniform (i.e. fixed-description-length) algorithmic computations on arbitrarily long sequences, it is natural to embed recurrences within a seq2seq network. Imitating the recurrent nature of the Turing machine, one can hope for RNNs to learn representations of the desired looped computations (Sutskever et al., 2013; Graves et al., 2014; Linzen et al., 2016). However, the key innovation in the Transformer architecture is a non-recurrent self-attention module.<sup>4</sup> Various works have noted that Transformers and RNNs learn qualitatively different solutions, discussing potential ways to reconcile these nuanced discrepancies (Dehghani et al., 2018; Abnar et al., 2020; Liu et al., 2023).
<sup>4</sup>We define self-attention in Appendix C. For in-depth breakdowns of the architectural components of Transformers, see (Vaswani et al., 2017; Phuong and Hutter, 2022; Edelman et al., 2022).
Figure 2: Elementary objects and examples associated with flip-flop languages. (a) the 2-state flip-flop machine (elided transitions are self-loops). (2) A 4-state automaton which processes flip-flop languages (implying the existence of a small RNN). (c) Simple examples of sequential prediction tasks which require processing a flip-flop language.
3 Flip-flop automata and the FFLM task
3.1 Definitions
For any even number  $T \geq 4$ , we define a flip-flop string as a sequence of symbols  $\{\mathsf{w},\mathsf{r},\mathsf{i},0,1\}^T$  which have the semantics of instructions (write, read, ignore) and data (one bit). A valid flip-flop string consists of alternating pairs of instructions and data (e.g. "w 0 i 1 i 0 r 0"), for which every symbol following a  $\mathbf{r}$  instruction must be equal to the symbol following the most recent w; thus, "w 0 i 1 w 1 r 0" is not a legal flip-flop string. These sequences can be viewed as correct execution transcripts of a program which can (perhaps occasionally) write to a single bit of memory, and always correctly reads its contents. All sequences are required to begin with w and end with r.
There are many possible choices of (probabilistic) flip-flop languages, which are distributions over valid flip-flop strings. We define a canonical family of them: let  $\mathsf{FFL}(T,\mathbf{p})$  be the distribution over length- $T$  flip-flop strings, parameterized by  $\mathbf{p} = (p_{\mathrm{w}},p_{\mathrm{r}},p_{\mathrm{i}})\in \Delta (\{\mathbf{w},\mathbf{r},\mathbf{i}\})$ , such that:
(i) The first instruction  $x_{1}$  is always  $\mathbf{w}$ , and the last instruction  $x_{T - 1}$  is always  $\mathbf{r}$ .
(ii) The other instructions are drawn i.i.d. according to  $(p_{\mathbf{w}}, p_{\mathbf{r}}, p_{\mathbf{i}})$  with  $p_{\mathbf{i}} = 1 - p_{\mathbf{w}} - p_{\mathbf{r}}$
(iii) The nondeterministic data symbols (paired with  $\mathsf{w}$  or i) are drawn i.i.d. and uniformly.
We are interested in whether language models can learn a flip-flop language from samples, which we define as processing the read operations perfectly. Two variants of the autoregressive language modeling task can be defined on this distribution:
- Generative ("noisy") mode: Estimate the conditional next-token distribution  $\operatorname*{Pr}[x_{t + 1}|x_{1:t}]$ , for each  $t = 1,\dots ,T - 1$ . In this mode, the sequences can be treated as drop-in replacements for natural text in GPT-style training. Generative FFLMs can be evaluated by checking their completions on prefix "prompts" (e.g. "...w0i1i1i[?]").
- Deterministic ("clean") mode: Predict only the continuations which are deterministic: correctly output  $x_{t+1}$  only for the prefixes  $x_{1:t}$  such that  $x_t = r$ . At the cost of a slight departure from vanilla language modeling, this setting isolates the long-range memory task. It is similar to the non-autoregressive flip-flop monoid simulation problem discussed in Liu et al. (2023), with limited supervision.
These tasks naturally embed the capability of simulating the flip-flop, a machine which memorizes a single bit (see Figure 2a,b for closely related variants).<sup>6</sup> It is easy to see that recurrent networks and
<sup>5</sup>We observe similar behaviors across these two settings (see Appendix B.2), but we report results on the "clean" setting in this paper. Predicting the non-deterministic tokens is irrelevant to the memory task at hand.
<sup>6</sup>A further discussion of the rationale for this specific manifestation of flip-flop sequence processing is deferred to Appendix A.3.
2-layer Transformers (see Proposition 2) can both represent FFLM parsers. The question of whether they do, especially from less-than-ideal data, turns out to be extremely subtle, and is the subject of the remainder of this paper.
3.2 Why focus on the flip-flop?
The most immediate rationale for this synthetic benchmark is that flip-flop simulation (maintaining memory in a sequence) is a direct necessity in many reasoning settings (see Figure 2c). It is a special (depth-1) case of Dyck language processing (Chomsky and Schützenberger, 1959; Yao et al., 2021; Zhao et al., 2023), which is necessary for parsing recursive grammars. It also captures certain structures in code or language tasks, such as tracking semantic changes (Miceli-Barone et al., 2023) or ignoring irrelevant contexts in general (Tafjord et al., 2020; Ho et al., 2020; Shi et al., 2023). The BIG-Bench suite of reasoning benchmarks (Srivastava et al., 2022) contains many tasks which require maintaining a discrete state over a sequence of transitions. Thus, more than a toy model, flip-flop languages are embedded verbatim within many sequence processing tasks. We offer some additional perspectives below.
Algebraic properties and expressive power. Flip-flops are the computational building blocks of memory. The flip-flop monoid  $\mathcal{F}$  (Definition 1), an algebraic encoding of a flip-flop's dynamics, is the smallest monoid whose operation is both non-commutative and non-invertible.  $\mathcal{F}$  plays an essential role in the Krohn-Rhodes theory of automata and semigroups (Rhodes et al., 2010), whose central structure theorem (Krohn and Rhodes, 1965; Zeiger, 1967; Eilenberg, 1974) implies that a constant-depth cascade of parallel flip-flops simulates all group-free finite-state automata. Thus, in a rigorous sense, the robust learning of flip-flops is not only a necessary condition for reasoning, but a sufficient condition for a wide class of algorithmic capabilities.
Intended functionality of attention. One can also appeal to the origin of attention mechanisms (Bahdanau et al., 2014; Luong et al., 2015; Vaswani et al., 2017): attention was specifically designed to attend to (i.e. selectively retrieve and copy) data over long-range dependencies. Indeed, it is easy to verify that a single attention head can perform the required lookup (see Proposition 2). It is thus logical to ask how well a purely attention-based architecture performs this elementary operation.
4 Attention glitches: a long tail of errors for Transformer FFLMs
In our main set of synthetic experiments, we train neural language models to generate strings from the flip-flop language FFL  $(T = 512, \mathbf{p} = (0.1, 0.1, 0.8))$  (for short,  $FFL(p_{\mathrm{i}} = 0.8)$ ), and probe whether the networks robustly learn the language. Although every valid flip-flop string is supported in this distribution, some sequences are far rarer than others; we measure tail behavior via probes of extrapolation, defined here as out-of-distribution evaluations which amplify the probabilities of the rare sequences. To create these "challenging" sequences, we sample  $>3 \times 10^{5}$  sequences from FFL(0.98) (containing unusually many "sparse" sequences with mostly ignore instructions), as well as FFL(0.1) (many "dense" sequences). Training and evaluating the read accuracies of Transformer models of various sizes, as well as a recurrent LSTM model, we find the following (see Figure 3):
(R1) Transformers exhibit a long, irregular tail of errors. Such errors occur on both sparse and dense sequences. Further, a model's out-of-distribution test error varies significantly between random seeds (initializations as well as stochastic minibatches), and even between iterates within the same training run.
(R2) LSTMs extrapolate perfectly. In stark contrast, with 20 times fewer training samples and iterations, a 1-layer recurrent model achieves  $100\%$  accuracy, on 100 out of 100 runs.
Data release. For reproducibility, we publish this synthetic data at https://huggingface.co/datasets/synthseq/flipflop: 16M FFL(0.8) training sequences, 16K FFL(0.8) in-distribution
<sup>7</sup>What this formally entails for representation and generalization is a topic of recent theoretical inquiry (Edelman et al., 2022; Wei et al., 2022a).
In the remaining of the paper, we will use  $\mathsf{FFL}(p_{\mathrm{i}})$  as a shorthand, with  $T = 512$  and  $p_w = p_r = \frac{1 - p_i}{2}$ .
Transformers  $(20\times$  more data & steps)
training iterations
Figure 3: Top: Training curves of recurrent (left) vs. Transformer (center) architectures on FFLM, with best-so-far evaluation errors highlighted for clarity. Transformers fail to extrapolate robustly to the long tail of long-range dependencies, even on this extremely simple task of remembering one bit. The bolded box contains our chosen 6-layer 19M-parameter canonical baseline model. We find that the ability to complete flip-flop language prompts emerges in natural language models, but is not robust (right). Bottom: examples from the sparser FFL(0.98) and denser FFL(0.1) distributions, causing distinct (long-range and short-range) failure modes for the baseline Transformer model.
test sequences, 160K sparse o.o.d. sequences from FFL(0.98), and 4K FFL(0.1)) dense o.o.d. sequences from FFL(0.98). Our data pipelines can be replicated exactly by taking the appropriate prefix slices of these datasets.
As a counterpart to these findings, we observe similar anomalies in real LLMs, when prompted to complete natural textual embeddings (Figure 2, top right) of flip-flop tasks:
(R3) 10B-scale natural LMs can correctly process flip-flop languages, but not robustly. Beyond a certain scale, natural language models can learn to process (natural embeddings of) flip-flop languages from in-context demonstrations. However, this emergent capability is not robust: there exist rare read errors, whose probabilities amplify as the sequence length  $T$  grows. We provide details for the few-shot evaluation protocol in Appendix B.2.1.
4.1 Multiplicity of mechanisms for attention glitches
What failure mechanisms account for these reasoning errors, which occur for both short- and long-range dependencies? The model capability is not a concern as discussed earlier (see Proposition 2). In this section, we discuss how Transformer self-attention modules, when tasked with representing flip-flops, can exhibit multiple (perhaps mutually entangled) failure mechanisms. The accompanying propositions are proven in Appendices C.2 and C.3.
An insufficient explanation: implicit  $n$ -gram models. As a warmup, consider a language model  $\widehat{\mathrm{Pr}}[x_{t+1}|x_{\leq t}]$  which only depends on the  $n$  most recent tokens in the context. Then, if  $n \ll \frac{1}{1-p}$ , the bulk of  $\widehat{\mathrm{Pr}}$ 's predictions on  $\mathsf{FFL}(p_{\mathbf{i}} = p)$  can be no more accurate than random guessing. This recovers one qualitative trend (degradation of accuracy with dependency length) observed in the experiments. However, this cannot fully explain our findings: it fails to account for the incorrect predictions on dense sequences. Furthermore, the Transformers' outputs on  $\mathsf{FFL}(0.98)$  are mostly correct; their accuracies on very long-range dependencies are nontrivial, despite not being perfect. There must therefore be subtler explanations for these errors.
Lipschitz limitations of soft attention. Moving to finer-grained failure mechanisms, a known (Hahn, 2020; Chiang and Cholak, 2022) drawback of soft attention is that its softmax operation can be "too soft"—for any weight matrices with fixed norms, the attention gets "diluted" across positions as the sequence length  $T$  increases, and can fail to perform an intended "hard selection" operation. We provide a formal statement and proof (Proposition 3) in Appendix C.2.
Difficulty of non-commutative tiebreaking. Can we simply robustify soft attention by replacing it with hard attention? We present a brief analysis which suggests that even hard attention can be brittle. In a stylized setting (one-layer models with linear position encodings), we show that self-attention can confidently attend to the wrong index, unless the weight matrices precisely satisfy an orthogonality condition (Proposition 4). This suggests the existence of spurious local optima, which we do not attempt to prove end-to-end; however, we provide supporting empirical evidence in the experiments in Appendix C.3.
5 Mitigations for attention glitches
In this section, we investigate various approaches towards eliminating the long tail of reasoning errors exhibited by Transformer FFLMs. We select the 19M-parameter model (which has  $L = 6$  layers,  $d = 512$  embedding dimensions, and  $H = 8$  heads) from Section 4 as a canonical baseline, and conduct precise evaluations of various direct and indirect interventions.
5.1 Effects of training data and scale
Ideal solution: improving data coverage. Prior work has made clear that data significantly impacts the performance (Schuhmann et al., 2022; Eldan and Li, 2023). Hence, we begin by examining what is perhaps the most obvious solution: removing the need for out-of-distribution extrapolation, by training directly on more diverse examples. Indeed, we verify that this works near-perfectly: