SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In terms of mitigation, various "external" methods (i.e. ones which do not modify the internal representations of the neural network) have been proposed to address some of the above factors, or post-processing model generations (Dziri et al., 2021; Chen et al., 2021), possibly utilizing several forward passes (Wang et al., 2022; Zheng et al., 2023). Another line of work that have gained much popularity and success is to incorporate explicit memory mechanisms, which we discuss next.
Explicit memory mechanisms in Transformers. Prior work has shown that augmenting the neural network with memory modules or knowledge base helps improve the performance on long-range texts (Khandelwal et al., 2019; Wu et al., 2022; Bertsch et al., 2023). An approach particularly effective for large-scale Transformers is to ask the model to output immediate reasoning steps to a "scratchpad" which the model subsequently processes (Nye et al., 2021; Wei et al., 2022b; Zhou et al., 2022; Anil et al., 2022; Shao et al., 2023), similar to writing to and reading from a memory tape. A particular way to interact with the scratchpad is to interlace every other token with an annotation of "as a reminder, this is the state" (Liu et al., 2023; Lanchantin et al., 2023), so that there are no more explicit long-range dependencies. However, this strategy is the same as the recurrent solution implementable by RNNs, and it does not always exist, especially when attention glitches occur in an internal component of the model.
Transformers and algorithmic tasks. Compared to real-world language datasets, synthetic tasks provide a cleaner and more controlled setup for probing the abilities and limitations of Transformers. Specific to algorithmic reasoning, Liu et al. (2023) puts a unifying perspective on the ability of small Transformers to succeed at tasks corresponding to algorithmic primitives. Specific tasks of interest include modular prefix sums (Hahn, 2020; Anil et al., 2022), adders (Nogueira et al., 2021; Nanda and Lieberum, 2022; Lee et al., 2023), regular languages (Bhattamishra et al., 2020; van der Poel et al., 2023), hierarchical languages (Yao et al., 2021; Zhao et al., 2023), and following a chain of entailment Zhang et al. (2022).
Comparison with Transformers Learn Shortcuts to Automata. Liu et al. (2023) study the parallel circuits efficiently realizable by low-depth Transformers. The authors identify shortcut solutions, which exactly replicate length- $T$  recurrent computations ("chains of algorithmic reasoning") in the absence of recurrence, with very few  $(O(\log T)$ ; sometimes  $O(1))$  layers. Their results contain a general structure theorem of representability, and preliminary positive empirics for generalization and optimization, demonstrating that Transformers can learn these shallow solutions via gradient-based training on samples. In contrast, the present work is a fine-grained study of the issue of generalization. Our main empirical contributions are a minimally sufficient setup (FFLM) and a
set of large-scale $^{14}$  controlled experiments, towards providing reasonable scientific foundations for addressing the unpredictable reasoning errors of LLMs.
A.3 Why this flip-flop language?
Liu et al. (2023) (as well as our mechanistic interpretability experiments) use a purer instantiation of flip-flop sequence processing, in which the sequence-to-sequence network is tasked with non-autoregressive transduction: given the sequence of input symbols  $\sigma_{1},\ldots ,\sigma_{T}$ , output the sequence of states  $q_{1},\ldots ,q_{T}$ . This is most natural when studying the Transformer architecture's algebraic representations in their most isolated form.
Our autoregressive sequence modeling setting is a slight departure from this setting; we discuss its properties and rationale below.
- The autoregressive setting "type-checks" exactly with standard state-of-the-art autoregressive (a.k.a. causal, forward, or next-token-prediction) language modeling. This makes it more convenient and intuitive as a plug-and-play benchmark.
- The cost is a layer of indirection: the model needs to associate "instruction" tokens with their adjacent "data" tokens. This is a natural challenge for representation learning, and is certainly a necessary cursor for robust extrapolation on natural sequences that embed similar tasks (like those considered in Figure 2c). It is straightforward to remove this challenge: simply tokenize at a coarser granularity (i.e. treat (instruction, data) pair as a distinct vocabulary item).
- The multi-symbol (and variable-length-symbol, etc.) generalizations of the binary flip-flop language are more parsimonious. If there are  $n$  instead of 2 tokens, this language can be encoded with  $n + 3$  commands. Without the decoupling of "instruction" tokens from "data", the vocabulary size would scale suboptimally with  $n$ . In Figure 12, we provide a quick demonstration that attention glitches persist in the presence of larger vocabularies.
- The conclusions do not change: in smaller-scale experiments, we observe the same extrapolation failures between the autoregressive and non-autoregressive task formulations.
A.4 Attention glitches in natural LLMs
In this section, we expand on the brief discussion from Section 6. At a high level, we hypothesize that attention glitches cause (some) closed-domain hallucinations in Transformer models of more complex languages. However, due to the fact that neural networks' internal representations evade simplistic mechanistic characterization, it is a significant challenge to formulate a rigorous, testable version of this hypothesis. We discuss the subtleties below.
First, we discuss a more general notion of attention glitches, of which the flip-flop errors considered in this papers are a special case. We define attention glitches as failures of trained attention-based networks to implement a hard retrieval functionality perfectly. To formalize this notion, there are several inherent ambiguities—namely, the notions of "hard retrieval" and "perfectly", as well as the granularity of "subnetwork" at which an attention glitch can be defined non-vacuously. The FFLM reasoning errors considered in this work provide a minimal and concrete resolution of these ambiguities. We discuss each of these points below:
- Hard retrieval: To succeed at FFLM, a network's internal representations must correctly implement the functionality of retrieving a single bit (from a sequence of bits, encoded unambiguously by the network), selected via the criterion of "most recent write position". This can be expanded into a richer functional formulation of hard attention, by generalizing the set of possible retrieved contents (a discrete set of larger cardinality, or, even more generally, a continuous set), as well as more complex selection criteria (e.g. "least recent position").
- Ground truth: Of course, to define "errors" or "hallucinations" in reasoning, there must be a well-defined ideal functionality. For FFLM, the notion of "closed-domain" reasoning and hallucinations is evident: the ideal behavior is for a model's outputs to coincide with that of the flip-flop machine on all input sequences. This straightforwardly generalizes to all formal languages, where the model is expected to correctly produce the deterministic outputs of automata
${}^{14}\sim 10^{4}$  19M-parameter Transformers were trained in the making of this paper; see Appendix B.6.
Input	GPT-3.5	GPT-4	Answer
8493 + 2357	10850✓	10850✓	10850
84935834 + 23572898	108008732×	108508732✓	108508732
9991999919909993 + 6109199190990097	16111199100810090×	16101199100890090×	16101199110900090
Table 1: Examples (in Figure 1) of GPT variants on addition: While models tend to succeed at additions with a small number of digits, they (nondeterministically) fail at longer additions.
which parse these languages. By considering expanded notions of "ground truth", it is possible to capture other notions of model hallucinations (such as incorrectly memorized facts). Our work does not address open-domain hallucinations (i.e. where the model output contradicts real-world knowledge (OpenAI, 2023)), which may be unrelated to attention glitches.
- Submodules: Towards attributing implementations and errors to localized components of a network, it is impossible to provide a single all-encompassing notion of "localized component". This is a perennial challenge faced in the mechanistic interpretability literature. Our work considers two extremes: the entire network (in the main experiments, where we probe end-to-end behavior), and a single self-attention head (in Sections 4.1, 5.3 and Appendix B.5, in which we probe whether a single attention head can learn multiplication in the flip-flop monoid). Even when considering the same functionality, attention glitches can be considered for different choices of "submodule".<sup>15</sup> Our results reveal a key subtlety: in the presence of overparameterization (more layers and parallel heads than necessary according to the theoretical constructions), Transformers learn to process flip-flop languages via soft attention.
We expect that to effectively debug the full scope of LLM hallucinations, all of the above choices will need to be revisited, perhaps in tandem.
We hypothesize that the algorithmic reasoning capabilities of real LLMs (i.e. their ability to recognize, parse, and transduce formal symbolic languages) are implemented by internal subnetworks whose functionalities can be identified with generalizations of the flip-flop machine. To the extent that such modules exist, attention glitches (the failure of these modules to represent the flip-flop operations perfectly, due to insufficient training data coverage) cause sporadic end-to-end errors ("closed-domain hallucinations"). In this work, we have treated the case of external attention (where the task is to learn the flip-flop directly).
B Full experimental results
B.1 Details for LLM addition prompts (Figure 1)
These addition problem queries serve as a quick demonstration of (1) non-trivial algorithmic generalization capabilities of Transformer-based LLMs; (2) the brittleness of such capabilities: we directly address this type of reasoning error in this work. Table 1,2 show these queries and results in detail.
We emphasize that these examples were selected in an adversarial, ad-hoc manner; we do not attempt to formalize or investigate any claim that the errors made by larger models are at longer sequence lengths. We also cannot rule out the possibility that some choice of prompt elicits robust algorithmic reasoning (e.g. the prompting strategies explored in (Zhou et al., 2022)). The only rigorous conclusion to draw from Figure 1 is that of non-robustness: even LLMs exhibiting state-of-the-art reasoning continue to make these elementary errors for some unambiguous queries with deterministic answers. It was last verified on May 8, 2023 that GPT-4 (in its ChatGPT Plus manifestation) demonstrates the claimed failure mode.
15 Beyond the two extremes considered in this work, some examples include "a subset of attention heads", "a subset of layers", and "a subspace of the entire network's embedding space".
Input	GPT-3.5	GPT-4	Answer
4491 + 8759	13250✓	13250✓	13250
80087394 + 63457948	143045342X	143545342✓	143545342
5101611078665398 + 8969499832688802	1.4071110911354202e+16X	14071110911354196X	14071110911354200
Table 2: More examples of GPT variants on addition: While models tend to succeed at additions with a small number of digits, they (nondeterministically) fail at longer additions.
B.2 Extrapolation failures of standard Transformers (Section 4)
This section provides full details for our empirical findings (R1) through (R3).
Architecture size sweep. We consider a sweep over Transformer architecture dimensionalities, varying the three main size parameters. We emphasize that these are somewhat larger than "toy" models: the parameters go up to ranges encountered in natural sequence modeling (though, of course, far short of state-of-the-art LLMs).
- The number of layers (depth)  $L \in  \{ 2,4,6,8\}$  .
- The embedding dimension  $d \in \{128, 256, 512, 1024\}$ .
- The number of parallel attention heads per layer  $H \in \{2, 4, 8, 16\}$ . In accordance with standard scaling rules-of-thumb, each head's dimension is selected to be  $d / H$ .
Other hyperparameter choices. We use a sequence length of  $T = 512$ , again to reflect a typical length of dependencies considered by nontrivial Transformer models. We use a canonical set of training hyperparameters for this sweep: the AdamW (Loshchilov and Hutter, 2017) optimizer, with  $(\beta_{1},\beta_{2}) = (0.9,0.999)$ , learning rate  $3\times 10^{-4}$ , weight decay 0.1, 50 steps of linear learning rate warmup, and linear learning rate decay (setting the would-be 10001th step to 0). We train for 10000 steps on freshly sampled data, and choose a minibatch size of 16; consequently, the models in this setup train on 81,920,000 tokens.
Training and evaluation data. Unless otherwise stated, we train our models on online samples (fresh i.i.d. batches) containing moderate-length dependencies ( $T = 512$ ,  $p_i = 0.8$ ,  $p_w = p_r = 0.1$ , or FFL(0.8) for short). We evaluate on the following held-out test sets, which are held the same across all training runs:
(i) In-distribution:  $10^{3}$  sequences from the same distribution FFL(0.8), containing 26508 read instructions.
(ii) Sparse tail:  $10^{5}$  sequences from FFL(0.98), containing 353875 read instructions.
(iii) Dense tail: 3000 sequences from FFL(0.1), containing 345781 read instructions.
When showing training curves, we evaluate errors only on (i) and the first  $1\%$  of (ii) every 100 steps; the purpose of these is only to give a qualitative sense of instability, and affirm the presence of errors. For the main suite of experiments, we focus on reporting the high-precision glitch rate measurements on (ii) and (iii). Final errors on (i) are exactly 0 except for a small number of non-converged runs (2-layer architectures and badly tuned hyperparameters), so we omit these evaluations in favor of the more informative measurements which focus on the sparse and dense tails.
(R1) Transformers exhibit a long, irregular tail of errors. Figure 6 shows training curves for 3 replicates (random seeds) in each setting, while the scatter plot in the main paper shows variability of out-of-distribution accuracy across random seeds for the baseline setup. We find that Transformers sometimes succeed at extrapolation, but erratically.
(R2) 1-layer LSTM extrapolates perfectly. We train a 1-layer LSTM (Hochreiter and Schmidhuber, 1997) network, with hidden state dimension 128 (for a total of 133K parameters), for 500
Transformer training curves: train on  $FFL(p_{i} = 0.8)$ , test o.o.d. on  $FFL(p_{i} = 0.98)$
Figure 6: Examples of training curves over various Transformer architectures, ranging from 46K to 101M trainable parameters. We exhibit 3 (randomly selected) random seeds for each architecture. Lighter curves show raw error percentages, while solid curves denote the lowest error so far in each run. Notice the following: (1) non-convergence of shallow models (despite representability) (2) failure of most runs to extrapolate (i.e. reach  $0\%$  out-of-distribution error); (3) high variability between runs; (4) erratic, non-monotonic progress on out-of-distribution data, even when the indistribution training curves appear flat; (5) a small LSTM outperforms all of these Transformers (see Figure 3). The bolded box represents our 19M-parameter baseline model.
steps with the same optimizer hyperparameters as above. The LSTM model achieves exactly 0 final-iterate o.o.d. error, over 100 out of 100 replicates. In smaller experiments, we found that larger LSTMs (width up to 8192, depth up to 4) also extrapolate perfectly.
Figure 7: Additional training curves. Left: Identical baseline architecture, varying the 5 data seeds and 5 model seeds: models in the same row encounter the same sequence of data, while models in the same column start from identical initializations. Both sources of randomness affect training dynamics and extrapolation, and it is not clear which is more important. Right: Similar findings for models trained in "fully generative" mode (scoring on all tokens); baseline architecture is in the bolded box.
Canonical baseline. We select the 6-layer, 512-dimensional, 8-head architecture (with 19M trainable parameters) as our canonical baseline model: it is large in relevant dimensions $^{16}$  to real Transformers, while being small enough to allow for thousands of training runs at a reasonable cost. To fully understand the variability of this single architectural and algorithmic setup, we train and evaluate 500 replicates in this setting.
Random data vs. random initialization. Recent synthetic probes on the surprising behavior of deep neural nets on hard synthetic tasks (Barak et al., 2022; Garg et al., 2022) obtain additional insights by disentangling the effects of data randomness (i.e. the precise sequence of minibatches) vs. model randomness (e.g. random initialization and dropout). We provide a quick demonstration in Figure 7 (left) that both sources of stochasticity matter. We do not perform a more detailed investigation of their precise influence and roles.
Fully generative setting: similar negative results. As mentioned in Section 3.1, to capture a setting closer to standard autoregressive (sometimes called GPT-style) language modeling, we find a similar failure to extrapolate when models are trained to predict all tokens, rather than only the deterministic ones ( $x_{t+1}$  such that  $x_t = \mathbf{r}$ ). Figure 7 (right) exhibits some training curves for this setting, showing non-extrapolation, variability, and instability. We observe that training (to indistribution convergence) takes slightly longer in this setting, and usually succeeds with the baseline architecture. We do not perform further controlled experiments in this setting.
B.2.1 Evaluating real LLMs on flip-flops
We provide a quick corroboration that while LLMs in practice can perform in-context reasoning when the sequences are unambiguously isomorphic to a flip-flop language. We use the natural language
<sup>16</sup>Except the vocabulary size. In preliminary experiments, we obtained similar findings in the case of token spaces larger than  $\{0, 1\}$ .
User: Hi, let's play a game shall well There are 3 instructions: write, read, and ignore. I will use w, r, l to denote each of them, respectively. Each instruction will be followed by a value (0 or 1), and the game is that: given a sequence of w, r, i and values (the sequence will always start with a w and end with a r), output the value get paired with the last w. Is the game clear?
User.Here is a sequence: w0i0i1i0i0i1i0i1i0i0i0i1i0i1i1i0i1i1r.
ChatGPT: the "w" (write) operation only appears at the beginning, and the paired value is 0. So, despite many "i" (ignore) operations, the value associated with the last "w" is still 0.
(b) Correct on short sequences.
ChatGPT: Yes, I understand the game. To clarify, here's what I understand:
1.  $\mathrm{w}^*$  stands for write and will always be followed by a value (0 or 1).
2. "r" stands for "read".
3. "I" stands for "ignore" and will also be followed by a value (0 or 1). The sequence always starts with a "w" and ends with a "r". My task is to output the value that is paired with the last "w" in the sequence before the final "r". With that understanding, let's play! Please provide a sequence.
ChatGPT: It's very long so it might take a bit of time 'w' for white, 'i' for ink, 'i' for read, 1 for yes/ on /true and 0 for no/off/false. Set white paper. Ink: On. Ink: Off. Ink: On. Ink: Off...
And it goes on like this until you get to the 'r' at the end.
With the 'w' and 'r' commands at the beginning and end respectively, and 'i' commands between them, it's as if you are
turning ink on and off multiple times on a white paper and then reading the results. However, without a pen to move on the paper, we can't get an actual image or text.
This sequence might make sense in the context of a robotic drawing or writing device that's being controlled to put down or lift up a pen to draw or write something on a paper, but without knowing the movement instructions, it's impossible to interpret what the final output should be.
(a) Explaining FFLM to ChatGPT.
(c) Wrong on long sequences (input length 1000).
Figure 8: Examples of interacting with ChatGPT-4 (as of 05/22/2023) by explaining FFLM to it.
example from Figure 2 (top right), and evaluate the capability of popular pretrained LLMs to correctly remember the state of a light switch. Specifically, write instructions in the FFLM task are either "Alice turns the light off" or "Alice turns the light on". The ignore instructions are either "Bob eats a banana" or "Bob eats an apple". All models are prompted with a translated, length-16 FFLM task that's been translated to English in this way before evaluation.
We measure this accuracy as a function of the sequence length for several well-known LLMs: GPT-2, GPT-2-large, GPT-2-x1, Pythia-12C, and GPT-NeoX-20B. Figure 3 shows how well these models perform on this task (i.e. the correctness of the model when prompted with "The light is turned") as the sequence length is varied. Consistent with the findings of this paper, larger models tend to perform best at this task, and the quality of all models deteriorates with increased sequence length. Each point on the plot considers 500 sequences of the indicated length. All models were prompted with a randomly generated, length 16 flip flop sequence to allow the model to learn the task in context. Accuracy is measured according to the frequency with which the model correctly predicts the current state of the light switch, as described in Section B.2.1.
(R3) 10B-scale natural LMs can correctly process flip-flop languages, but not robustly.
Note that it is impossible to quantify the degree to which these sequences are "in-distribution" (it is unlikely that any sequences of this form occur in the training distributions for these LLMs). Much like linguistic reasoning evaluations in the style of BIG-bench (Srivastava et al., 2022), we rely on the emergent capability of in-context inference (Brown et al., 2020) of the task's syntax and semantics. As discussed in Appendix A.4, this layer of indirection, which is impossible to avoid in the finetuning-free regime, can cause additional (and unrelated) failure modes to those studied in our synthetic experiments. Fully reconciling our findings between the synthetic and non-synthetic settings (e.g. by training or finetuning on sequences of this form, or via mechanistic interpretation of non-synthetic language models) is outside the scope of this paper, and yields an interesting direction for future work.
Direct in-context induction of flip-flop languages? Given the conversational abilities of LLMs, another way to interact with an existing pretrained model is to explain the definition of FFLM in natural language, and ask the model to output the correct state for  $\mathbf{r}$ . We test this using ChatGPT (with GPT-4), as demonstrated in Figure 8. ChatGPT seems to understand the rules and can get short sequences correct (up to sequence length 400), but makes errors with unexpected connections on longer sequences.
B.3 Effects of training data and scale (Section 5.1)
Here, we provide more details for the interventions outlined in Section 5.1, which directly modify the training distributions and resources.
Scaling (data, compute, network size) effects on o.o.d. performance
Figure 9: Detailed comparisons of various scaling axes, showing violin plots for selected runs (left) and an annotated scatter plot for all runs (right); + markers show medians for particular configurations. Increasing training data diversity is by far the most effective way to mitigate attention glitches in FFLM. The other scaling axes (increasing the amount of fresh data, increasing the number of optimization steps on the same dataset, and changing the model size) have mixed effects on rare-sequence performance. In particular, we wish to highlight that neither overparameterization nor underparameterization strongly modulates the rate of glitches.