Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
Authors: Mayank Jobanputra
Yana Veitsman
Yash Sarrof
Aleksandra Bakalova
Vera Demberg
Ellie Pavlick
Michael Hahn
Venue: NeurIPS 2025 poster ()
Keywords: transformers
length generalization
empirical analysis
explainability
Primary area: deep_learning
TL;DR: While large-scale pretraining brings remarkable capabilities, it cannot fundamentally rewrite the architecture’s core inductive biases.

Abstract: Transformers have theoretical limitations in modeling certain sequence-to-sequence tasks, yet it remains largely unclear if these limitations play a role in large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models themselves and their pretraining data. We explore how these architectural constraints manifest after pretraining by studying a family of *retrieval* and *copying* tasks inspired by Liu et al. [2024a]. We use a recently proposed framework for studying length generalization [Huang et al., 2025] to provide guarantees for each of our settings. Empirically, we observe an *induction-versus-anti-induction asymmetry*, where pretrained models are better at retrieving tokens to the right (induction) rather than the left (anti-induction) of a query token. This asymmetry disappears upon targeted fine-tuning if length-generalization is guaranteed by theory. Mechanistic analysis reveals that this asymmetry is connected to the differences in the strength of induction versus anti-induction circuits within pretrained transformers. We validate our findings through practical experiments on real-world tasks demonstrating reliability risks. Our results highlight that pretraining selectively enhances certain transformer capabilities, but does not overcome fundamental length-generalization limits.

Born a Transformer - Always a Transformer? On the Effect of Pretraining on Architectural Abilities
Mayank Jobanputra' Yana Veitsman', Yash Sarrof', Aleksandra Bakalova',
Vera Demberg', Ellie Pavlick?, Michael Hahn
Saarlann University BBrown University
fmayank, (hhahn]@lstt sonss.umiiiie umi-saarlam de
Abstract
Transformers have theoretical limitations in modeling certain sequence-to-sequennc tasks, yet it remains largely unclear these limitations play a role large-scale pretrained LLMs, or whether LLMs might effectively overcome these constraints in practice due to the scale of both the models themselves and their pretraining data. We explore how these architectural constraints manifest after pretraining, by studying a family of retrieval and copying tasks inspired by Liu et al. [2024a]. We use recently proposed framework for studying length generalization [Huang et al., 2025] to provide guarantees for each of our settings. Empirically, we observe induction-versuu vettiinn-vruuu-anlte vewsus-aaai-indeaeee -atiianndtiiiion asymmelry, where pretrained models better at retrieving tokens to the right (induction) rather than the left (anti-induction) of a query token. This asymmetry disappears upon targeted fine-tuning if length- generalization is guaranteed by theory. Mechanistic analysis reveals that this asymmetry is connected to the differences the strength of induction versus anti-induction circuits within pretrained transformers. We validate our findings through practical experiments on real-world tasks demonstrating reliability risks. Our results highlight that pretraining selectively enhances certain transformer capabilities, but does not overcome fundamental gength-gnneralization limits!.
Introduction
Transformers [Vaswani et al., 2017] have become the backbone most LLMs [Radford et al., 2019, Brown et al., 2020, Touvron al., 2023]. Given the widespread usage of LLMs, there has been long line of research understand their theoretical underpinnings. Early analyses showed that transformers approximate any seuuencc-to-sqqueeee
function fixed input length [Yun al., 2019] and even simulate Turing machines [Pérez et al., 2019] when given unbounded compute. However, subsequent studies have uncovered fundamental expressibility and learnability limitations, finding classes of problems which transformers cannot express across arbitrary input lengths [e.g. Hahn, 2020, Strobl et al., 2024, Merrill and Sabharwal, 2023, Sanford et al., 2024], which transformers systematically struggle to learn and generalize on [e.g. Hahn and Rofin, 2024, Zhou et al., 2024]. Empirical work confirms that small transformers trained from scratch do instantiate such limitations [e.g. Bhattamishra et al., 2020, Huang et al., 2025, Delétang et al., 2023, Zhou et al., 2024]. For instance, transformers trained from scratch struggle generalize to longer or more challenging examples even on tasks as simple as copying a long string with repeated symbols [Zhou et al., 2024], or retrieving which token followed the last occurrence of some query token in a long context [Liu et al., 2024a] findings in close agreement with theoretical arguments about transformer's 's learning biases [Huang et al., 2025].
'These authors contributed equally.
'https://github com/lacoco- lab/alllllllla haab///lwys_... transformer
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, it remains unclear if such results on expressiveness and generalization have any bearing on large-scale pretrained models. Modern LLMs have many more layers, and many orders of magnitude more parameters than the small-scale transformers typically used for testing theoretical limitations. Model scale alone might make theoretical limitations irrelevant at realistic input lengths. Perhaps even more importantly, LLMS are pretrained on a massive scale, with trillions of tokens of varied data that may impart fundamentally new inductive biases and algorithmic abilities [Rytting and Wingate, 2021] that may be invisible in models trained from scratch on the target task alone [Amos et al., 2024]. Indeed, LLMs' capabilities scale in ways that are difficult to foresee: empirical "laws" relate performance to model, data, and compute budget (Kaplan et al., 2020, Hoffmann et al., 2022] and qualitatively new behaviors such as in-context learning, chain-of oftthhught reasoning, and tool use emerge abruptly at scale [Wei et al., 2022, Ganguli et al., 2022, Bubeck et al., 2023]. Since pretraining reshapes both the model's inductive biases and the sub-circuits it relies on, failures observed in small models might not transfer automatically to their large-scale counterparts.
Scope. We focus on retrieval and copying as fundamental operations that provide a test bed for answering these questions. These operations are critical building blocks for many real-world LLM applications, such as question answering and summarization [Wiegreffe et al., 2025, Fan et al., 2024]. On a mechanistic level, retrieving from context via induction heads is key to performing in-context learning [Olsson et al., 2022, Song et al., 2025, Crosbie and Shutova, 2025], while faithfully copying exact spans makes retrieval- augmented generation more effective YYu et al., 2024]. Understanding the potential impact of architectural limitations on LLMs is thus of great interest for these tasks.
In this work, we revisit the theory of length generalization for these operations, ask how its predictions hold up for pretrained transformers, and diagnose which abilities are amplified by pretraining and which limitations persist. We study family of retrieval and copying tasks² (Figure 1) inspired by Liu et al. [2024a], Zhou et al. [2024]. We organize retrieval into two umbrella categories of unique and non-unique induction, respectively, in which the query token appears exactly once or multiple times respectively. In each variant, the model must return a token located either before or after designated occurrence of that query token. We make an analogous distinction in copying between copying unique tokens or repeated tokens, and between copying in the forward or the reverse direction.
All retrieval and copying tasks we consider are expressible by transformers in principle: however, theory predicts systematic differences in length generalizability,, Specifically, we show (Section 3) that theory predicts transformers to have a wniqueness bias (i.e., unique-token induction is easier than its non-unique counterpart) [Huang et al., 2025, Zhou et al., 2024]; but for it to not have a directional bias (i.e., that forward and reverse variants are equally hard). Theory also predicts that within non-unique induction, retrieving is easier from the first occurrence of the query character than from the last one.
However, given the impressive capabilities of transformer- basedd LLMs, it is reasonable posit that the massive scale of both such models and of pretraining might overcome these limitations. For instance, high prevalence of non-unique copying in natural language might lead transformers to learn specialized circuits or components for handling such situations. Here, we directly test whether these architectural limitations remain relevant to pretrained LLMs. We study 8 pretrained LLMS from Llama-3 and Qwen2.5 families [Grattafiori et al., 2024, Yang et al., 2024]. Our findings show that large-scale pretraining produces a directional bias (Section 4.1). In addition to showing our results on synthetic settings, we show how these limits might surface and the reliability risks they pose for practitioners in more natural settings as well (Section 4.2). While targeted fine-tuning can even out the directional bias born out of pretraining, we find that the fundamental architectural uniqueness bias still binds pretrained transformers (Section 4.3).
2 Background, Notation & Definitions
Task Description. We examine two primary task types, Retrieval and Copying, each instantiated under Unique and Non-unique conditions, as illustrated in Figure 1. In Retrieval, the sequence has the form X = (bos)) 11,... .............................. -N- x.. where each 24 E some alphabet E and xN = q is the query token. The query g appears times in the context, indexed as q1.... with 1 S q1 < ... < < N. Transformer has to predict a single token as the output given the input. For the Unique condition, we
Pi.c., copying of the impur, not the training data, similar to Jelassi et al. [2024]
3 Pability to correctly solve tasks instances longer than those seen during training
2

---

RETRIEVAL
COPYING
Query appears (unique) the context
UF (Unique forward copylng)
Unique tokens
b
d
<eos>
<bos>
b
a
Ans
b
b
<eo5>
(Unique
(Unique Right)
(Unique copylng)
Query appears multiple (non-unique) times context
(Non Unique forward copying)
Repeated tokens
a
b
b
<eos?
d
a
a
a
#
a
Ans
<bos>
a
b
b
b
a
a <eo5>
NLFirstt
NRFirst N
NRLLat
(Non Unique Left/ Right First Retrievall
(Mon Unique Retrieval)
NB (Non Unique beckward copying)
Figure I: Overview of our task variants (formal definitions in Section 2). Retrieval: A '#' marks the separator: the token that follows is the query that may appear once (unique) multiple times (non-unique). With non-unique queries we return the token immediately left/right either the first or the last occurrence, creating 6 sub-tasks in total. Copying: We want model copy the context which consists of either only unique tokens or repeated tokens the forward or reverse direction. Tasks green lie in C-RASP[pos] and length-generalize; those in red do not (proofs in Section 3).
define UL (Unique Induction Left) and UR (Unique Induction Right) respectively when the output expected is the token preceding following the query's single occurrence, i.e., Fig -1 and Teg +1- Under the Non-unique condition, NLFirst (Non-unique Left First) and NRFirst (Non-unique Right First) the first occurrence to predict Fg -1 and g 1 the output, while NLLast (Non-unique Left Last) and NRLast* (Non-unique Right Last) use the last occurrence produce Fg.-1 and gg.+11 respectively.
For Copying, the sequence is X = (bos).: ... (sep) and the expected output is continuation JN+1 ... ..N.N- The decoder is expected to replicate the latter segment based on the first. For Unique conditions all of the tokens in X are unique. UF (Unique Forward copying) and UB (Unique Backwards copying) denote forward and reverse copying of the input segment. For Non-unique settings there is no uniqueness constraint on the x; values and NF and NB also denote the forward and reverse copying of the input segment.
Induction and Anti-Induction Heads. Induction heads were characterized by Elhage al. [2021]
part of a and were later shown to be critical performing in-context learning [Olsson et al., 2022].A previous- token head in layer writes the embedding of the immediately preceding token into the residual stream. An induction head in some layer for positions whose stored "pre- vious token" equals the current to- ken and copies the corresponding value vector forward, enabling the
two-head circuit present in autoregressive transformers
aluntiy
Forward Copying
PREVIOUS TOKEN
White predecesoar resduul um of the neiet toven for every poottion como byyo
Abow
a
INDUCTION HEAD (ac layer
Query: When Preducassor
sbosk
#
Query: My
& then queries
Backward Copying
Onformation availabre for quarying layer L)
-bos-
b
ONTTSINDUCTIOON HEAD
(ot lateeyer >
Pass: My Tredhecssor
Figure Illustration of the induction and anti-induction Circuits. repeated bigram. Our UR retrieval as well as the UF copying task setups are not novel and are meant to test this induction circuit ability and compare with other task setups.
model predict the token after
We hypothesize that, by symmetry, an anti-induction circuit should principle exist and able to copy information backwards. Here, the same previous-token head caches the token immediately to the left, but paired with an anti-induction head that looks back to the earlier occurrence of the same
'The NRLast variant is conceptually equivalent the Flip-Flop task of Liu al. [2024a].

---

token. Because that earlier token received the value of its own predecessor from the previous- token head, the anti anti-induction head can retrieve that predecessor, allowing the model to predict one step to the left. Such a two-head motif should thus enable exact solutions to our UL and UB tasks. Retrieval heads of Wu al. [2025] subsume both induction and indiiinuutiinn heads (see Appendix E for discussion).
3 Theoretical Length Generalization Guarantees for Our Tasks
We derive theoretical predictions for transformers' length- gaagtt-generalzzttion abilities on the retrieval and copying tasks introduced above, using the framework of Huang et al. [2025], which links length- generalizability to expressiveness in C-RASP [Yang and Chiang, 2024]. Of particular interest is C-RASP [Yang and Chiang, 2024], which restricts the usage of positional information and arithmetic operations. Huang et al. [2025] further added positionally-awann the language C-RASP(posJ', Doing so, they established-bott theoretically and empirically-a tight link between the existence of a C-RASP[pos] program for a task and the ability of decoder-only transformers (with absolute or no positional encodings) length-generalize on that same task: in particular, if C-RASp[pos] program exists, a decoder-only model provably generalizes to longer inputs under a specific formal model of training (Theorem in Huang et al. [2025]). More precisely, this guarantee applies to APE Transformers under an idcalized training procedure, where all training data up to some maximum input length N are available and the exact minimizer of a regularized loss is found. Under these specific conditions, the existence of a C-RASP(pnn program for that task guarantees length generalization to larger lengths. While theory has only been worked out for this specific setup, empirically we observe converging results across different PE types and realistic SGD-based training (as shown in Huang et al. [2025] as well as in the experiments in Appendix B). As converse to this length generalization guarantee, extensive experimental evidence in Huang et al. [2025] suggests that, if no C-RASP program exists, such models fail to generalize to longer inputs.
primitives to C-RASP to construct
We therefore adopt the following strategy for providing length generalizing guarantees across our retrieval and copying tasks:
Construct a C-RASP[pos] program = the task length-generalizes to arbitrary n.
We start by considering the "Right" versions of our Retrieval tasks. In the unique case (UR), transformer can solve retrieval using standard induction circuit (Figure 2): an induction head retrieves the symbol that followed the unique occurrence of the query. These operations can be simulated in C-RASP[pos] (Lemma 4), ensuring theoretical length generalization guarantee, In the non-unique case, expressibility and hence the length generalization guarantee- varies:
Theorem 1. NRFirst is expressible in C-RASP(pos). NRLast is not expressible in C-RASP(pos].
The proof is in the Appendix (Lemmas 5 and 6). An extension of the induction circuit performs first, at each Fy, an attention head checks whether this symbol has appeared previously. Given a query w, an induction circuit then retrieves the symbol immediately following the (unique, any exist) first appearance of w. A C-RASP[pos] program formalizes this (Lemma 5). On the other hand, the inexpressibility of NRLast is proven by reduction to the regular language (alble)" provably not C-RASP (Lemma 6). Regarding copying tasks, Huang et al. [2025] shows that UF is in C-R (.Poss, while NF is not. The construction for UF again uses an induction circuit.
NRFirst:
So far, we have considered "right" and "forward" versions-in in fact, in all tasks, C-RASP[pos] expressibility is equivalent for the left (backward) settings:
Theorem 2. Across all tasks, there is no C-RASP[pos] expressibility difference between R vs I. versions, and B versions.
Referred to as C-RAsP[local, periodic] in Huang et al. [2025]. See Appendix A.I.
'While Huang et al. [2025] focuses absolute (APE) or positional encodings (NoPE), our empirical results show that the conclusions also match the behavior of Rotary Positional encodings (RoPE) (See Appendix B). Full theoretical understanding for RoPE length generalization remains an interesting problem. One interesting aspect is that, while training RoPE for UF and UB did not show perfect generalization, we did obtain perfect generalization when instead autoregressively decoding from models trained for UR/UL. In APE, we obtained perfect generalization when training directly for UF and UB, in line with C-RASP[pos] expressivity.

---

That is, C-RASP[pos] expressivity is the same for UL and UR. the same for UF and UB, etc. (Corollary 9). All tasks are annotated Figure with their C-RASP[pos] expressibility. Experiments in Appendix B show that transformers trained from scratch on these tasks exhibit length generalization as predicted by the theoretical results.
(L=f)Bwd)
Ulama (Righ(Fwd)
Qwen 32B (Lef/Bwd)
Owen (RigW/Fwd)
Retrieval
Copying
UUJRR
UB/UF
1.0
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
20
30
40
50
100
20
30
40
50
100
Input Length
Input Longth
NLFirst/RFFiitt
NLLastNRLast
NB/NF
1.0
Non-unique
0.8
Accuracy
0.6
0.4
0.2
0.0
30
100
Inputi Length
Inpuri
Input Length
Figure 3: In-context accuracy for Llama-3 70B and Qwen2. 5-32B across all our tasks averaged three seeds. Across all settings, lengths, model size, and task type, observe Directional Bias: Retrieving the token to the left of the query token is always more difficult compared to the constant. Similarly, copying in the forward direction
the right, provided all other things easier than copying backwards. Detailed prompts, similar performance graphs on other models (including instruction-tunm variants) are in Appendix C.
4 Experiments
4.1 Eliciting Abilities via In-Context Learning
We evaluate pretrained completion well as instruction-tuned variants of the following models: Ll ama3. 1 1-8B, L1ama3. 1-70B, Qwen2.5- 5-7B, and Qwen2.5-335 5-32B. For retrieval as well as copying, each model evaluated across various prompt variations. The prompt variations relate to the few-shot example size (examples are equal in length to the test string or are smaller but clearer) and the instruction templates (details on prompt templates the Appendix C.1) with the templates adjusted for completion as well as instruction-tune
models. All experiments use 1,500 test strings for each length 10, 20, 30, 40, 50, 100 generated under three independent seeds. We cap the input string length at 100 tokens maintain bounded vocabulary comparable to the unique setups; see Appendix C.4 for results at other lengths. The test strings contain spaced letters (including diacritics) and digits, to ensure that every character gets tokenized as independent token and thus, the required uniqueness / repetitions can be maintained within the context, whenever required. Few-shot prompts contain = demonstrations, reserving one slot whenever explicit explanation is required by the template. For each task that belongs to the same group i.e. for Unique (UR, UL) and for Non-Unique (NRLAST, NRFIRST, NLLAST, NLFIRST), have the same strings which we test the in-context ability with different expected answers to avoid the confound of different datapoints.
Directional Bias: Pretraining induces left-right asymmetry. We found persistent directional bias all of pretrained models we tested (Figure 3). For each task, retrieval or copying, the models consistently performs better rightward tasks (UR, NRFirst, NRLast, UF, NF), across models,
5

---

prompt templates, and sequence lengths. The drop in performance for the leftward tasks cannot be attributed to lexical cues such as the words "right" or "forward" (explored in earlier work [He et al., 2025]) because the performance asymmetry persists even when we strip all natural-language instructions and provide only input-output pairs for the model to infer the task in-context. C- RASP[pos] predicts symmetric performance in transformers trained from scratch within a task family (Theorem 2). We further find that models trained from scratch do not exhibit such a left-right asymmetry, confirming it is not inherent to the architecture (Appendix B). We therefore hypothesize that pretraining instills this bias because induction is more natural task than anti-induction and thus more induction head circuits get formed during pretraining as opposed to anti- anti-induction circuits; Sections 4.3 and 4.4 probe this explanation in more detail.
Uniqueness Bias: Unique tasks are easier to elicit than non-unique tasks. Unique retrieval and copying setups both perform relatively well in rightward/ forward direction. In contrast, in the non-unique settings, eliciting correct retrieval in any direction is challenging across task setups. We observe an interesting difference to what is expected for transformers trained from seratch: First, the underperformance of NRFirst in pretrained models does not align with what is expected for transformers trained from scratch (Theorem 1). Copying, by contrast, is easier to elicit than retrieval, even though copying requires predicting many tokens and hence offers more opportunities for error. Remarkably, Non-unique unique-forware (NF) remains near perfect throughout. As this experiment only considers input lengths up to 100 for comparable results vis-à-vis the unique settings, interesting question is how copying performs at longer lengths. The next section therefore stress-tests in-context copying on longer paragraphs, seeking reveal if the theoretically predicted difficulty of non-unique copying (NF) appears in realistic settings.
Takeaway: Pretraining introduces a Directional Bias favoring retrieving and copying the forward/right direction over the backwards/left. Pretraining also selectively amplifies the abilities of a transformer, as evident from the near perfect performance on copying repeated tokens, and poor performance certain retrieval tasks, even though the latter are better Iearnable in a length- generalizable way for the architecture.
4.2 Testing Uniqueness Bias and Directional Bias in Natural Settings
Testing Uniqueness Bias by Copying Lorem Ipsum Paragraphs. In Section 4.1, forward copying showed near-perfect performance irrespective of uniqueness (UF and NF). We now ask whether the theoretically predicted asymmetry between unique and non-unique copying still impacts LLM behav- ior, by having models reproduce longer, naturalistic but semantically neutral text. We here do not focus on evaluating exact copying accuracy, instead focusing on understanding the nature of LLM mistakes. While entirely incorrect outputs are easily de- tectable, subtle differences in nearly accurate copies represent the most challenging errors for users to identify. Therefore, we focus exclusively on the
Unambiquous
Ambiguous
LLama
1.00
Accuracy
cases where the generated output is within 75% of the length of the expected output, targeting scenar- ios where the model clearly understands the copying task and produces plausible copies of the input. To confirm that prompting setup is valid, uate the models on copying exact spans from recent arXiv papers (April 2025), finding perfect perfor- mance.
0.87
0.50
703 Jivtrast
Qwen 2.5
eval-
1.00
Accurory
0.90
78
75 Instruct
325
525 Instruck
We design copying benchmark suppress mantic cues that could help the model break ambigu- ous copy chains. Specifically, each input consists of a randomized content is unlikely to have appeared verbatim in the training corpus. Each paragraph has close to 500 tokens. See Appendix C.5.1 for results at longer lengths (upto 5k tokens).
Figure 4:
Failures in accurate copying of Lorem Ipsum paragraphs are associated pri- marily with ambiguous transition indices.
paragraph. Such
We classify each token in the input as either unambiguous (uniquely predictive of the subsequent token) ambiguous (potentially followed by multiple subsequent tokens and therefore is not perfectly

---

predictive of the subsequent token). We perform token-level alignments using the Needleman- Wunsch algorithm [Needleman and Wunsch, 1970], to detect token-level insertions, deletions, and substitutions. What we are left with is an alignment sequence insert, delete and substitute. We group consecutive entries in this alignment list if they are of the same type and call them aligned spans if the entries consist of only 'match' and misaligned spans if the entries grouped consist of any of the other types. We refer to indices where an aligned span finishes and misaligned span starts transition indices. These transition indices mark critical points where an error in copying started propagating during autoregressive generation. It should be noted that, even if there are just few such misaligned spans, they can each be very long, especially for the ones consisting exclusively of 'insert'. Such hallucinated insertions can at times snowball into a long error chain from where the model never reorients itself to produce a correct continuation (Examples in Appendix C.5).
sequence consisting of 4 kinds of entries: match,
We categorize the transition indices as unambiguous or ambiguous based on the nature of the token at that index. Since an induction head would suffice to correctly predict the subsequent token following an unambiguous token, we hypothesized that transition indices would never be unambiguous. Conversely, ambiguous tokens were expected to be the source of where all of the misaligned error span began, as it has similarities to the failure modes expected in NF (Non-unique forward copying). Empirical results (Figure 4) confirm this hypothesis. Across all model variants tested (sizes, families, completion well instruction tuned models), consistent pattern emerges, the transition indices almost unilaterally always are ambiguous and cause the "glitches' we see in the final copied output. While these glitches appear across models, their exact nature appears to be random. We did not find any pattern except for the fact that they were tied to ambiguous tokens.
Testing Directional Bias in Git Commit History: Direction-Aware Copying, While the Lorem Ipsum setup shows how glitches in forward copying might cause reliability issues, we now show how failures in backward copying impact real world tasks. We use a com- pact Git history manipulation task exemplify when the right-over-left asymmetry seen in Section 4.1 surfaces in practice. Given a truncated git log (newest-roldest com- mit hashes), the model must list commits that need reverting git revert(forward or- git cherry-pick(revene pick(reverse order) onto release branch.
der) or must be cherry-picked Llama-3 models reproduce the forward list al- most perfectly but accuracy drops when the or- der flips (Figure 5) mirroring the asymmetry seen in the synthetic experiments (Section 4. 4.1).
revert
cherrypick
Accuracy
1.0
I
Qwen models are not able to do the task all, hence we exclude them from this analysis. The
0.5
reversal presented here does not happen here at the token level, instead at the commit order level and is thus in principle more similar to a gener- alized copying setup. However, the theoretical results and expectations transfer to the general- ized scheme (See Appendix for detailed dis- cussion A.2.1).
0.0
8B
70B
8B-IT
70B-IT
Figure 5: Git Commit History Manipulation also has the forward backward asymmetry seen in Section 4.1.
Even though pretraining selectively amplifies certain abilities of the transformer, fun- damental limitations inherent to the architecture have marked impact even after the pretraining and need to be kept in mind while deploying these models in real-world tasks.
Takeaway:
4.3 Fine-tuning Eliminates the Directional Bias, But Not the Uniqueness Bias
Our experiments have uncovered three failure modes: (i) a Directional Bias in pretrained models, (ii) poor performance on certain retrieval tasks that are provably Uniqueness Bias: sporadic "glitches" on copying tasks that are not bangth-generalizable. Each of these could stem either from architectural limitations or from inductive biases imparted during pretraining. To disentangle the two, we carry out task-specific supervised fine-tuning and see which failures are removed. Since we wish to know whether the model has truly learned we evaluate on test sequences twice as long as those seen in training, following Huang et al. [2025]. A fully correct model should therefore achieve 100% accuracy on this test set; even single error
and (iii) the
genggghhggeeruiiabblee solution,
7

---

then flags reasoning flaw that could undermine reliability in real- applications, an evaluation scheme similar to that of Liu al. [2024a].
Experimental Setup. We fine-tune a pretrained language model 2 1.5B (APE) on each retrieval task plus unique forward and back- ward copying. Fine-tuning uses the standard next-token objective but masks the loss to the answer span only, mimicking inference, i.e., for retrieval tasks, just the answer token, and, for the copying tasks, all tokens after the EOS delimiter in the input, indicating the end of the input se- quence. To ensure that APE mod- els can generalize on longer positions than those seen during training, we ap- ply the positional offset trick of Huang et al. [2025]: for every training input a random constant A E [0, 200 mmmn] is added to all position indices, ensuring that all the positional embeddings are trained. All models see training examples whose input lengths are sampled uniformly from [emin: 100], where mii is the length of the shortest well-formed instance of the task. We confirmed that, prior to this fine-tuning, the base model's performance on each task was effectively zero. Details of hyperparameters and datasets in Appendix D.
Task PoSA
not C-RMSP/PPS
1.0
0.4
1.8
1.R
findt Wifinst
Milast WRinst
Figure 6: On the vut-of-distribution dataset, tasks belonging to C-RASP[POS] (UL, UR, UF. UB, NLFIRST, NRFIRST) retain perfect accuracy, whereas NLLAST, NRLAST, NF, and NB, which provably not in C-RASP(POS), suffer drop in performance. The Directional Bias within task setups is largely absent from the fine-tuned models.
Our testbed setup for all of our tasks closely follows the setup of Bhattamishra et al. [2020], Huang et al. [2025]. with an in-distribution bin (i.e., lengths [fmin, 100)) and an out-of-distribution evaluation bin (i.e., lengths [101, 200).. Similar to their setup, we report accuracy on our test sets only on seeds where training loss converges to O. For our out-of out-of-distribution evaluation bin on Retrieval tasks, we enforce that all instances of the query token q appear exclusively in the first half of the context C. This way distance between the relevant tokens which the model needs to attend increases, thereby rigorously testing the model's ' ability to generalize to longer contexts. In such settings, superficial heuristics should be insufficient, and thus expect that only models with robust length generalization can perform reliably. We find that C-RASP(pos] perfectly tracks length generalization results in our fine-tuning experiments: Every task which belongs to C-R C-RASP(pos], and is thus predicted to -ength-generalize, does so, whereas tasks not in C-RASP[pos] do not the Directional Bias of pretrained models disappears with fine-tuning, confirming the fact that it was an artifact of pretraining. We next seek to understand what leads to the development of this asymmetry by analyzing induction and anti induction circuits (defined in Section 4.4) both in pretrained models, where this asymmetry exists, and in our fine-tuned models, where it doesn't.
Additionally,
4.4 Source of the Directional Bias: A Mechanistic Perspective
We hypothesize that the source of directional bias in pretrained models lies the difference between the presence and strength of induction and anti-induction circuits as described in Section 2. We posit that, in pretrained models, anti-induction heads are less common, but that fine-tuning can boost these, removing the asymmetry. We use Unique copying (UF and UB) setups to evaluate our hypothesis.
Induction and anti-induction heads are causal in both fine-tuned and pretrained models. We conduct a patching experiment to confirm that removal of induction heads hurts performance in the UF task, while removal of anti-induction heads hurts performance in the UB task. Indeed, both in fine-tuned and pretrained models, removing induction heads is catastrophic for UF and has a negligible effect in UB, and removing anti- inti-induction heads cripples backward copying while leaving forward copying intact. On fine-tuned checkpoints, the effect is amplified to the extreme: removing relevant heads drops the accuracy to zero, while removing another type of heads has no effect (Figure 7b). Thus, induction and anti-induction heads have causal effect on model performance (Details of patching experiment in Appendix E). Additionally, the induction and anti-inductions heads get amplified via fine-tuning (Figure 7a). Fine-tuming strengthens the heads that matter: induction heads gain attention weight for forward copy, anti induction heads for backward copy.
? aattps://uuggiingaaco.o wttp://www....tat..t....aacaaa/e 1ww.s__fiaaal..__1111_11 hf
8

---

Unique Copy
Unique Forward
1.0
1.0
0.8
Artention
0.5
Sequeenc Length
0.4
Liema
fine-tuned GPS-2
GPT-2
Unique Backward Copy
Unique Backward Copy
1.0
Induction Head
Ant--iduction
0.8
Anti Induction Head
Fine-TTuned
0.6
Induction
GPT-2
0.2
0.0
Index
(a) Fine-tuning strengthens the relevant heads
(b) Induction and Anti-Induction heads causal
Figure 7: (a) Top- 1op-100-ereent attention heads ranked by their attention score, i.e. the mean attention weight that target-string tokens allocate either the same source token (anti-induction) the next source token (induction). Dashed lines: pretrained checkpoints; solid lines: fine-tuned. Shaded bands cover source lengths 3-10, Fine-tuning eliminates the pretraining imbalance; for forward copying (UF) fine-tuning strengthens induction heads, for backwards copying (UB) fine-tuning boosts anti- induction heads. (b) Patching experiment that ablates each family of heads while measuring task accuracy. For UF, removing induction heads (red - green) is fatal, whereas ablating anti-inductio heads inconsequential (red - blue); the roles reverse for UB. This holds across models & sequence lengths confirming the causal status of the two circuits.
The directional bias in pretrained transformers can explained by the difference strength and prevalence of induction vs. anti-inductio heads. Fine-tuning can be used to amplify either of these heads and thus remove the asymmetry.
Takeaway:
Discussion
Related Work. Length generalization has been studied extensively [Anil et al., 2022, Abbe et al., 2023, Zhou et al., 2024], including attempts to improve it using scratchpads [Abbe et al., 2023, Hou al., 2024] or specific positional embedding schemes [Press al., 2022, He et al., 2024, Cho et al., 2024]. Transformer limitations: besides copying and retrieval, transformers struggle with problems including sensitive functions [Hahn and Rofin, 2024] and state tracking [Merrill and Sabharwal, 2023]. These limitations can be related to failures L.MMs on tasks such as multiplication [Frieder al., 2024. Satpute al., 2024, Amiri al., 2025] and maintaining state information [Toshniwal et al., 2022, Kim and Schuster, 2023, Merrill et al., 2024, Zhang et al., 2025]. Induction heads have been studied extensively since Elhage et [2021] and their existence has been linked in-context learning and copying [Olsson et al., 2022, Chen et al., 2024a, Singh et al., 2024, Crosbie and Shutova, 2025]. Retrieval heads [Wu al., 2025] generalize this notion by conditionally copying information from arbitrary positions, subsuming our definitions of induction and anti-induction heads.
Takeaways. While failures in length generalization in copying and retrieval tasks may not affect typical user interaction, they pose reliability concerns in specialized applications. For instance, AI coding assistants like GitHub Copilot [Chen et al., 2021] routinely handle non-semantic identifiers (commit hashes, variable names, function names). Glitches in copying such strings could lead to bugs the deployment of critical code and would therefore be highly undesirable. Pretraining of LLMs has been shown to impart biases, be it for specific tokens or specific entities [He et al., 2025] and in-context performance can be brittle based on the choice of tokens used. This can lead to reliability risks in domains requiring high precision such as medicine [Li and Chong, 2024, Bélisle-Pipon 2024]. In contrast, the left-right asymmetry we uncover indicative of latent ability that difficult to elicit in pretrained models without explicit fine-tuning. We ask practitioners to be aware of the

---

existence of such biases and consider task-specifer fine-tuning to eliminate this kind of asymmetry if is considered undesirable for downstream application.
On the other hand, glitches in copying large semantically neutral text (SHA keys, DNA sequences) would fall under the umbrella of repeated copying. To achieve complete reliability and compensate for persistent architectural limitations, practitioners might need to resort to external tool usage in LLMs. Similar how tool usage used compensate for mathematical weaknesses in LLMs [Gao et al., 2023, Chen et al., 2024b, Gou et al., 2024, Wang et al., 2024], designing principled policies that decide when use tools like programmatic copy-pasting could be an interesting research direction.
Limitations. We focus on two open LLM model families, not covering the entire model landscape, especially closed source systems models based recurrent architectures, which might show different sets of abilities or limitations [Sarrof et al., 2024, Grazzi et al., 2025, Siems et al., 2025]. Second, due resource limitations, fine-tuning experiments in Section 4.3 are limited to single, 1.5B sized model. We also do not analyze the underlying pretraining data of the models to get an estimate of what the correct in-distribution bin should be for any of our models. Larger contexts should in principle push the out-of of-disttibuution boundary to higher lengths as the in-distribution size increases. It remains an open question whether length generalization would even matter for larger models with a huge in-distribution context size. Finally, our theoretical results linking C-RASP[pos] to enngh-generalizann applicability to RoPE has only been tested empirically.
in Huang et al. [2025] have only been proven formally for APE, and
Conclusion
We asked whether large-scale pretraining can erase the transformer's native limits, using a foundational family of retrieval and copying tasks as testbed. We find that pretraining boosts certain abilities, but transformers remain bound by the same length-generrliization limits they were "born" with: Pretraining amplifies induction circuits, making right-/forward-orries and copying notably casier, but does not overcome the difficulty inherent to non-unique retrieval and copying. tine-tuning can rebalance induction and anti-induction heads and restore full, theory-aligned generalization, but only by explicitly teaching the model its missing capacities. Our results thus suggest that, while large-scale pretraining brings remarkable capabilities, it cannot fundamentally rewrite the architecture's 's core inductive biases.
retrieval
Acknowledgements
We thank anonymous reviewers for their encouraging and constructive feedback. This research is funded in part by the Deutsche Forschungseemensccaat (DFG, German Research Foundation) Project-ID 232722074 SFB 1 1102 "Information Density and Linguistic Encoding": Project-II 471607914 GRK 2853/1 "Neuroexplicit Models of Language, Vision, and Action"; and Project-ID 389792660 TRR 248 "Foundations of Perspicuous Software Systems". We would like to thank Mark Rofin, Blerta Veseli, Brian DuSell, Ekaterina Shutova, Kayo Yin, Xinting Huang, Yifan Wang for discussions and feedback on the draft.
Contributions
MJ contributed to Sections 4.1, 4.2, 4.3, scaling experiments, paper refinement and the Appendix. YV contributed to Sections 4.1, 4.2, 4.3, 4.4, and paper refining. YS contributed Sections 4. 1.1, 4.2, the Appendix, scaling experiments, paper drafting and refining. SB contributed to Section 4.4. VD and EP provided feedback, supervision, and refined the paper. MH supervised the project and contributed to Section 3, paper drafting, and refining.
References
Abbe, S. Bengio, A. Lotfi, and Rizk. Generalization on unseen, logic reasoning and degree curriculum. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
A. Amiri, X. Huang. M. Rofin, and M. Hahn. Lower bounds for chain-of of ofothhtght reasoning in hard-attention transforeers. arXiv preprint arXiv:2502.0233 2025.
10

---

1. Amos, J. Berant, and A. Gupta. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In The Twelfth International Conference on Learning Reprexentations, 2024. URL attps://openrevier net
C. Anil, Y. Wu, A. Andreassen, A. Lewkowezz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and
B. Neyshabur. Exploring length generalization in large language models, Advances Neural Information Processing Systems, 35:38546-38556 2022.
A. Bakalova, Y. Veitsman, X. Huang, and M. Hahn. Contextualize-hee--aggrgeeee then-aggregate: Circuits for in-context learning in gemma-2 2b. arXiv preprint arXiv:2504.00132. 2025.
S. Bhattamishra, K. Ahuja, and N. Goyal. On the ability and limitations of transformers to recognize formal languages. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7096 7116. Association for Computational Linguistics, 2020. doi: 1118653/V1/0220 EENIO2....MLP..... WAMNLP-MAIN.SS MAIN.576. URL attpss//doo org/10. 18653/v1/2020 emnlp-main. mtp-mmai..777 576.
Bhattamishra, M. Hahn, P. Blunsom, and Kanade. Separations in the representational capabilities of transformers and recurrent architectures. CoRR, abs/2406.09347, 2024. doi: 10.4855WARXXI:224000094477
URL https://doo org/10 org/ 48550/ /ariv.. 2406. 09347.
B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, Hadsell, M. Balean, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 2020, virtual, 2020. URL https: / http///poceednn...///e neur neurips- cc/ppppep////e/// http:////////////e/// hhash// 1457
Abstract.
S. Bubeck, V. Chadrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. Lee, Y. L. S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gith 2023.
J.-C. Pipon. Why we need to be careful with Ilms in medicine. Frontiers in Medicine (Lausanne). 11: 1495582, December 4 2024. doi: 10.3389/fmed.200) 200.mme.d..2...499..0 1495582.
M. Chen, J. Tworek, H. Jun, Q. Yuan, P. D. Pinto, Kaplan, H. Edwards, Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Chen, Sheen, Wang, Yang. Unveiling induction heads: Provable training dynamics and feature leamning in transformers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
W. Chen, X. Ma. X. Wang, and W. Cohen. Program thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2024b.
H. Cho, J. Cha, Awasthi, S. Bhojanapalli, Gupta, and Yun. Position coupling: Improving length generalization arithmetic transformers using task structure. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL ht tps /atp:://oop/eiieeeeeee net/ / forumridie diruummiiiiiili idddiIIMM.....
J. Crosbie and Shutova. Induction heads as an essential mechanism for pattern matching in in-context learning. 2025. URL httph:t/p///t //apxii///////////////aa/aa org/
G. Delétang, A. Ruoss, Grau-Moya, T. Genewein, L. K. Wenliang. E. Catt, C. Cundy, M. Hutter, S. Legg, Veness, and A. Ortega. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
wtwplee...eeeeeeeeee heft//df?/// ? httpiiiddwwxxeeeee
N. Elhage, N. Nanda, C. Olsson, Henighan, N. Joseph, B. Mann, Askell, Bai. A. Chen, T. Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.
W. Fan, Y. Ding. L. Ning, Wang, Li, D. Yin, T.SS. Chua, and Q. Li. survey on rag meeting Ilms: Towards retrieval- retrieval-ungmented large language models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, page 6491-6501, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704491. doi: 10.1145/3635528.3677 1470. URL https://doi. org/10. 10.11....0 1144//137522--... 367717777777777777000
S. Frieder. L. Pinchetti, R.-R. Griffiths, T. Salvatori. T. Lukasiewicz. P. Petersen, and J. Berner. Mathematical capabilities of chatgpt. Advances in Neural Information Processing Systems, 36, 2024.
11

---

D. Ganguli, D. Hernandez. L. Lovitt, A. Askell, Y. Bai, A. Chen, T. Conerly, N. Dassarma, D. Drain, N. Elhage. al. Predictability and surprise in large generative models. In Proceedings the 2022 ACM Conference on Fairnexs, Accountability, and Transparency, pages 1747-1764, 2022.
L. Gao, A. Madaan, S. Zhou, U. Alon, Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764 10799. PMLR, 2023.
Z. Gou, Z. Shao, Y. Gong, Y. Yang, M. Huang, N. Duan, W. Chen, et al. Tora: A tool-integrated reasoning agent for mathematical problem solving, In The Twelfth International Conference on Learning Representations. 2024.
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Daale, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama herd of models, arXiv preprint arXiv:2407.21783, 2024.
R. Grazzi, Siems, J. K. Franke, A. Zela, Hutter, et al. Unlocking state-tracking in linear mns through negative cigenvalues. In ICLR, 2025.
M. Hahn. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8: 8:156-171, 2020.
M. Hahn and M. Rofin. Why sensitive functions hard for transformers? In Proceedings of the 2024 Annual Conference of the Association for Computational Linguistics (ACL, 2024), 2024, arXiv Preprint 2402.09963.
M. Hanna, O. Liu, and A. Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in pe-ttrinee language model. Advances in Neural Information Processing Systems, 36:76033-76066 2023.
Y. He, B. He. Z. Ding, A. Lupidi. Y. Zhu, S. Chen, C. Zhang, J. Chen, Y. Ma. V. Tresp, and I. Horrocks. Supposedly equivalent facts that aren't 't? entity frequency in pre-training induces asymmetry in Ilms, 2025. URL aattp:///ariiy orgg abaaab//253 22362.
Hc, Feng. S. Luo, K. Yang, L. Wang. Xu, Z. Zhang, H. Yang, and He. Two stones hit one bird: bilevel positional encoding for better length extrapolation. In Proceedings of the 41st International Conference on Machine Learning, ICML'24, JMLR.org, 2024.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T.
Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks. J. Welbl, A. Clark, al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 30016-30030, 2022.
K. Hou, Brandfonbrener, S. M. Kakade, S. Jelassi, and Malach. Universal length generalization with turing programs. CoRR, abs/2407.0333 10,10000,.......0 2024. doi: 10.48550ARXIV:2400..00 18650/arXiv. 2407 03310.
URL https:://obl org/10 10.
X. Huang, A. Yang. S. Bhattamishra. Y. Sarrof. A. Krebs, H. Zhou, P. Nakkiran, and M. Hahn. A formal frame- work for understanding length generalization in transformers. In The Thirteenth International Conference on Learning Representations, 2025. URL https: htappps:////rv...eeeeeeeeeasmest wrg/abs//////eeooo//ooooee a2s/2410....../.../ .22140.
Jelassi, S. d' d'Ascoli, C. Kamiing--Enriche Y. Wu, Y. Li, and F. Charton. Length generalization arithmetic transformers. CoRR, Abs/2306.15400, 2023. doi: 10:4550/ARXIV.2300.55lA URL https: /// /ooi org/ 10. 20.44850/arre arXiv. 2306 15400.
S. Jelassi, D. Brandfonbrener, S. M. Kakade, and E. Malach. Repeat after me: Transformers are better than state space models copying. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: htttp:://openeai http:///ppeeeve.ene..oo fet/ouumnd duRRoGeoQT.
G. Kaplan, M. Oren, Y. Reif, and R. Schwartz. From tokens to words: On the inner lexicon of LLMs. In The Thirteenth International Conference on Learning Representations, 2025, URL httpss://pporrrviee... net/
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001 008311, 2020.
N. Kim and S. Schuster. Entity tracking language models. A. Rogers, J. BoyddGrrbbe and N. Okazaki. editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). pages 3835 3855, Toronto, Canada, July 2023, Association for Computational Linguistics. doi: 10. 006653/v1/2223 acl- Long 213. URL attps://aclantholl/ org/ 2223/222..r acl-long, 213/
12

---

http:/orrommdddee///eeeee D. Li and J. Chong. Laterality: potential pitfall in applying multimodal large language models to radiology. Radiology, 31422), 2024.
B. Liu, J. Ash, S. Goel. A. Krishnamurthy, and C. Zhang. Exposing attention glitches with flip- flop language modeling. Advances in Neural Information Processing Systems, 36, 2024a.
N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 2024b.
W. Merrill and A. Sabharwal. The parallelism tradeoff:: Limitations of Log-precision transformers. Transactions of the Association for Computational Linguistics, 11:53 11:531-545, 2023.
W. Merrill, J. Petty, and Sabharwal. The illusion of state in state-space models. In Proceedings of the 4 International Conference on Machine Learning, pages 35492-35506, 2024.
B. Needleman and D. Wunsch. A general method applicable the search for similarities in amino acid sequence of two proteins. Journal of molecular biology, 48(3):443 453, 1970.
C. Olsson, N. Elhage, N. Nanda, Joseph, N. DasSarma, Henighan, Mann, Askell, Y. Bai, Chen, ct al. In-context learning and induction heads. CoRR, 2022.
O. Press, N. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. International Conference on Learning Reprexentations, 2022. URL httpss/////eeerrreiii net/Eforum2id-R8B0PPOO0o
Pérez, Marinković, and P. Barceló. On the Turing completeness modern neural network architectures. In International Conference on Learning Representations, 2019.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188)9, 2019.
C. Rytting and D. Wingate. Leveraging the inductive bias of large language models for abstract textual reasoning. Advances Neural Information Processing Systems, 14:17111-17122, 2021.
C. Sanford, J. Hsu, and M. Telgarsky. Representational strengths and limitations of transformers. Advances Neural Information Processing Systems, 36, 2024.
Y. Sarrof, Y. Veitsman, and M. Hahn. The expressive capacity of state space models: A formal language perspective. In The Thirty-eighth Anmual Conference Neural Information Processing Systems, 2024. URL https://openreviee
A. Satpute, N. GieBing, A. Greiner-Petter, M. Schubotz, O. Teschke, A. Aizawa, and B. Gipp. Can lms master math? investigating large language models on math stack exchange. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 2316-2320, 2024.
J. Siems, T. Carstensen, A. Zela, F. Hutter, M. Pontil, and R. Grazzi Deltaproduct: Improving state- tiite-raakkigg in linear mas via householder products. In ICLR 2025 Workshop on Foundation Models the Wild. 2025.
A. K. Singh, T. Moskovitz, F. Hill, S. C. Chan, and A. M. Saxe. What needs to go right for an induction head? mechanistic study of in-context learning circuits and their formation. In International Conference Machine Learning, pages 45637-45662. PMLR, 2024.
J. Song, Z. Xu, and Y. Zhong. Out-of ditt----isttibutionn generalization via composition: lens through induction heads in transformers. Proceedings of the National Academy of Sciences, 122(6))24171822 2025.
L. Strobl, W. Merrill, G. Weiss, D. Chiang, and D. Angluin. What Formal Languages Can Transformers Express? Survey. Transactions of the Association for Computational Linguistics, 12:543-561, 05 2024. ISSN 2307-387X. doi: 10.1 11e/:///_aae00e00011 00663, URL
wwt.ooe11ceaaacaeeee.a a_00663.
Q. Team. Qwen3 technical report, 2025. URL http::s/aaaxw...rrrbbbbb//////// www.aawwwww....////////.///////. 09388.
S. Toshniwal, S. Wiseman, K. Livescu, and K. Gimpel. Chess as a testbod for language model state tracking. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 11385 11393, 2022.
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, F. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 13377. 2023.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems, pages 5998-6098099 10886608 2017.
13

---

K. Wang, A. Variengien, A. Commy. B. Shlegeris, and J. Steinhardt. Interpretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL http:///rxii. wtg/abx/2211.00593 2022.
K. Wang. H. Ren, A. Zhou, 7. Lu, S. Luo, W. Shi, R. Zhang, L. Song, M. Zhan, and H. Li. Mathcoder: Seamless code integration in Ilss for enhanced mathematical reasoning. In ICLR, 2024.
J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201. 11903, 2022. URL hRL.ttpp: httpa://arxii httpaaiwwiirr./raa/ee./ee arggaba/2222 11903.
S. Wiegreffe, O. Tafjord, Y. Belinkov. H. Hajishirzi, and A. Sabharwal. Answer, assemble, ace: Understanding how LMs answer multiple choice questions. In Thirteenth International Conference on Learning Representations, 2025. URL https:/ / wew.:/eneeeettoommddn/ni/it
W. Wu, Y. Wang. Xiao, H. Peng, and Y. Fu. Retrieval head mechanistically explains long long-context factuality, In The Thirteenth International Conference on Learning Representations, 2025. URL https:// nst/forum?l id SeerrummiddyyyBBpop ByBUUGG...
A. Yang and D. Chiang. Counting like transformers: Compiling temporal counting logic into softmax trans- formers. In First Conference on Language Modeling, 2024, URL https: http:/:///pereeeee.eeee. wwwpe/rwwweeeeeeeeeeeeeeeeeeeeee wwt.:ormm//o FmbPg4UJ9K.
Q. A. Yang, B. Yang. B. Zhang. B. Hui, B. Zheng. B. Yu, C. D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Yang. J. Tu, J. Zhang, J. Yang. J. Yang. J. Zhou, J. Lin, K. Dang. K. Lu, K. Bao, K. Yang. L. Yu, M. Li, M. Xue, Zhang,
Zhu, R. Men, R. Lin, Li, Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y-C., Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, Z. Qiu, S. Quan, and Z. Wang. Qwen2.5 technical report. ArXiv, 2bs/2412.15115,,
2024. URL https: : // /api. semanticscho ar org/Corpus1D:27488842) 274859421.
H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and 7. Liu. Evaluation retrieval-angmented generation: A survey. In CCF Conference on Big Data, pages 102-120. Springer, 2024.
C. Yun, S. Bhojanapalli, A. S. Rawat, S. J. Reddi, and S. Kumar. Are transformers universal approximators of sequence-to-sequeee functions?, 2019.
Zhang, W. Du, D. Jin, J. Fu, and Z. Jin. Finite state automata inside transformers with chain-of of-thhughtthh mechanistic study on state tracking. arXiv preprint arXiv:2502.20122 2025.
H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. M. Susskind, S. Bengio, and Nakkiran. What algorithms can transformers learn? A study in length generalization. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httpp////peneeviee..eeee/rrorrrs HHHnH...
14

---

A Detailed Theoretical Proofs
A.1 C-RASP[pos] as a Framework for Predicting Length Generalization
C-RASP[pos] (pos for "positional information") was introduced as C-RAsP(periodiic,een by Huang et al. [2025]* as a formalization of RASP-L. and the RASP-I. conjecture of Zhou et al. [2024]. It extends C-RASP [Yang and Chiang, 2024] by incorporating positional information. Huang et al. [2025] connected expressibility C-RAsP[pos] to length generalization for transformers both theoretically and empirically:
1. They proved that transformers, when trained with an idealized model of learning, generalize on problems expressible in C-RASP[pos}.
2. Across a battery of algorithmic and formal language problems, they showed that problems expressible in C- C-RASP[pos] show length generalization when transformers are trained with SGD.
3. For problems in this battery that are not expressible in C-RASP[pos], they showed that length general- ization was empirically unsuccessful.
These results unify and explain variety of prior results; for instance, the link between C-RASP[pos] and length generalization explains why transformers show persistent glitches in FlipFlop (empirically observed by Liu et al. (2024a]). it explains why length generalization on copying is difficult in the presence of repetition [Zhou et al., 2024, Jelassi el al., 2023], and why length generalization C-RASP[pos] offers single framework for understanding length generalization, with well-understood methods for understanding expressivity. It more fine-gree tine-grained than various other methods understanding transformers' expressivity: For instance, all tasks considered in Zhou et al. [2024]. Huang et al. [2025] are in TC and in principle expressible by transformers, but length generalization varies substantially in line with C-RASP[pos] expressivity. Thus, C-RASP[pos] provides more fine-grainet perspective by focusing on length generalizability.
addition is difficult [Zhou et al., 2024]. Importantly.
C-RASPI [pos] can thus be viewed formal model of the space of problems that transformers represent across different input lengths and are likely length-generalize on. transformers can in principle represent proh- lems outside C-R Asp[pos], but these are then likely to not show length generalization hecause representations at increasing lengths will require increases in model size or parameter porm - if learning tends to find simpler solutions, one will expect length generalization to fail [Zhou et al., 2024, Huang et al., 2025]. For instance. construct transformer performing copying over strings of any fixed length N, e.g. by hard-coding matching pairs positions |Bhattamishra et al., 2024], but any construction that works across all lengths up to < N will require a growth of model size (in sense made formal by Huang et al. [2025]) as N increases, i.c., length generalization is unlikely.
As Huang et al. [2025] formalizes the RASP-L conjecture of Zhou et al. [2024]. we expect that the same conclusions would likely be reached using the RASP-L language from Zhou et al. [2024]: the advantage of C-RASP(pos] as compared to RASP-L is that its expressiveness is provably understood (including impossibility proofs), and is theoretically linked to length generalization (Theorem 7 Huang et al. [2025]).
For self-containedness, we recapitulate the operations used C-RASP[pos] here:
Definition 3. Let Σ be alphabet.
Let & be set of unary relations N (1, where each & satisfies
(Periodicity) for some A > 0, it holds that d(1) = o(i + A) all
Let V be a set of binary relations v N x (1. here each (ii. 5) satisfies"
(Depends only on Distance) (b( only depends on jj
(Restricted Bounded Distance) (i (t. = T} is finite, for any j E
Here, 1 stands for "fulse" and T stands for "true".
A C-RASP[pos] program P is defined as sequence Pt.... Pr. of operations. each of the following types:
Wee use a shorter name just for convenience.
These properties referred to as translation invariance and locality in Huang et al. [2025].
15

---

Boolean- Valued Operations
Count- Valued Operations
Initial
P(i) = Qx(() for E E
Counting
C(i) Lj S i, +(i,)) P() for d E V (T)
Boolean
P(i) == PP
Conditional
C(i)
Ca(1)
P(i) == Pi (
Addition
C(i) (1()
Constant
P(i) ==
Subtraction C(i) (it
Positional
P(i) == ó(i)
Constant
C(i) :=
for @ E +
Comparison
P()) =- Ci(1) S C2(1)
The most important construct is the Counting operation, which returns integer indicating for how many positions both P(j) and (3. hold. the special returns "true"). the operation just counts how often P(j) holds for j S Another important operation is Initial operation, where Qu (t) is true if and only if the i-th symbol in the string equals a. We refer to Huang et al. [2025] and Yang and Chiang [2024] for detailed definitions of the semantics.
where 1 T (i.e., predicate that always
The most important example for function & E y is the "predecessor" predicate checking if = i-
T
if =i-= else
(i,
(I)
1
We will follow Huang et al. [2025]. Yang and Chiang [2024] in using, as syntactic sugar, Bij S (i.i))
shorthand for # S i, (i and # [j S i] P(j) for # [j S T] P()).
In this paper, retricval task, the next output symbol when copying string. To formalize this, assume that every program contains Boolean-valued operations named NEXT, (1) for each E E such that NEXT, (1) holds and only the correct next symbol is a. Thus, we say that task is expressible in C-RAS C-RASP[pos] if there is a program defining NEXT, (i) for each E E such that, for each in the span of the output (a single position for a retrieval task, the full output string for copying task), NEXT, (f) holds (i.e, returns "truc") at the final position if and only if the correct next symbol Generation stops when none these operations return "true".
interested in tasks that, given prefix, produce token; either the overall output symbol in
A.2 Expressiveness Results
We show:
Lemma 4. UL. UR are expressible in C-RASP(pos].
Proof. A UR program is given in Huang et [2025]; the UL program analogous; overall:
C-RASP[pos] program for UR, UL
PRED. 3 S ز for cach a E E
1)Q.(i)
(I)
CBIGRAM == 3/3 < i)Qu() A PRED.(J)
(2)
for each a E
(EXT. V (Q)(1) A (BIGRAM....) EE
(UR version) (3)
for each a E
NEXT. = V (Q-(1) CHIGRAM...))
(UL. version) (4)
oEE
for each a E
This program is direct formalization of the (anti-)inductioon head circuit, and basic to the other C-RASP(pos) programs constructed below. The first line checks for each Σ, at each position whether position i holds the symbol a. The second line checks, for each bigram ab E Σ whether it appears at some position the context up to the i-th position. The third line then states that, in UR, the correct next token is a and only the bigram aa has appeared in the context, where a is the symbol at position i. The fourth line is the analogous instruction for UL: Here, predicted if and only if the bigram appears in the context.
L
16

---

Lemma 5. NRFirst and NLFirst are expressible in C-RAsplpos].
Proof. We first show construction NRFirst; builds on construction from Lemma 4:
C-RASP[pos] program for NRFirst
(SLEFTMOST()
V (Qa(1) aE>
1
(1)
PREDa : S = - - QQ.
ISLEFTMOST( )
(2)
for each E E
CBIGRAM.. Jij 1 [Qa(s) (RED..
(3)
for each a, E
NEXT (0) = V (1) A (1.
(4)
for a
where a holds at the final position if and only if the desired completion is the symbol a. The first line in the program checks, at each position, if it is the leftmost representative of the symbol that it holds: That is, whether there is symbol E Σ that occurs at position but at earlier position. For each E Σ, the second line checks, at position i, whether the preceding position i 1 holds the symbol Previous- Token head. Additionally it also uses the computations made the first line check whether position i 1 was the leftmost representative with such property. For each possible bigram ab E 2 X 2, the third line checks whether it has appeared or not. Finally, the fourth line says that a is the correct next token if and only if the current position holds symbol such that appears a bigram in the context. This last line the that simulates the operation of the Induction head itself.
simulating the operation
The program for NLFirst is very similar, it only differs the fourth line:
C-RASP[pos] program for NLFirst
ISLEFTMOST()) V (1) A 3 s (0.00)
(1)
3E PRED.. = J ú 1] (Q. A ISLEFTMOST. for cach a E
(2)
CBIGRAM ab = j/j s [|[[o[)) A PRED.
33)
for each a, E
(EXT. = V (Q.(1) A CBIGRAM, (1) 3E
(4)
for each E E
where (E holds the final position and only the desired completion the symbol
Lemma 6. NRLast and NLLast are expressible in C-RAsP(pos).
We show this by reducing them to Lemma 36 in Huang et al. [2025]. Assume, for the sake contradiction, that aRLa was expressible in C-RASP(pos). This program would discriminate between the two languages
Proof.
(ui) "wi"" (sep)w
(retrieving "1")
(5)
and
(w0)wrl1)) w0i" (sep)w (retrieving "0")
(6)
Such program could be turned into a program discriminating between the two languages
(w0) w1 i) wli"
(7)
and
(w0 wl i) w0i" By a simple transformation of the alphabet, this in turn would amount to discriminating between the two languages
(8)
(albli) as"
(9)
17

---

and
(albli)*bi
(10)
As described in the proof of Lemma 36 in Huang al. [2025]. C-RASP[pos] program discriminating between these two languages could be used create C-RASP/pos] program for recognizing the language
(alb)t) bi ((a(6)))
(11)
which in impossible by Lemma 35 in Huang al. [2025]. The reasoning holds for Nasast.
Lemma 7. UF, UB expressible in C- C-RASP(pos].
Proof. This task can be viewed as iterated application of UL and UR, respectively, and can thus be done with the same program. The only complication is that (i) at the beginning of copying, at (sep), the program needs to retrieve the symbol that had followed (bos), and that (ii) generation should stop once (scp) has been generated second time. These conditions can be encoded into C-RASP/pos].
Lemma 8. NF, NB not expressible in C-RASP[pos].
Proof. Huang et al. [2025] (Theorem 12) show that NF is not in C-RASP[pos] by showing that all problems expressible C-RASP[pos] have logarithmic communication complexity in the model where Alice and Bob have access to the first and second halves of the string, respectively. Because NF has sublinear communication protocol in this model, it is in C-RAsP(pos]. The argument applies NB.
We conclude from all lemmas in this section that there is no theoretical difference between left and right versions of retrieval and copying tasks:
Corollary 9 (Repeated from Theorem 2). Across all tasks, there is no C-R C-RASp[pos] expressibility difference between R L versions, and F vs B versions.
Proof. Immediate from lemmas this section.
A.2.1 Generalized reverse copying
Let the token alphabet be faI,.... Tx ( where represents separators. Ew Σ {0} are all tokens that can be part a word. Define the language L == (E D)* " Every æ can written uniquely as wy Dwy ... Ew, with w, E Σ* (i.e. each w; represents word.) and also as x = L ..... lms where each 4 E and is the token level representation of the string, We can think of copying this string E L in ways.
Peok(ti fm) fm
Pword ( (wi -... Dwn) = 0 -ww
Prok(fs ... tm) is the same backward copying setup, while Pword (wn 0 [.w)) more closely matches how a typical user would ever want to interact with an LLM to reverse string if ever.
Corollary 10. Pword is not expressible in C-RASP(pos).
Proof sketch. Theorem states that Non- uniqun Copy backwards which is the same as Piok Ptak-lie -iis outside C-RASP(poS] by communication complexxty argument. To transfer this inexpressibility to Pword consider the following reduction.
Reduction. Embed any token string .....tm fm E E' into T. by inserting separators:
4 D Di E
Because each "word" now has length 1, applying Pword yields
Ponnd (4. Clm) L., Dh.
and deleting the D symbols recovers Prok ...... ). Hence an algorithm for Pword would give one for Ptok with constant overhead, contradicting Theorem
Intuition. Pword reverses a list of chunks, whereas pook does the same when every chunk is single token.
When each word has length the two coincide, so solving word-orde reversal would solve Non Non-unique copy-back bopybbackwards special case-which creates contradiction, and thus Pword should also not be solvable.
18

---

Problem
Model Size
LR
Max Steps
UL, UR, UF, UB
2 layer; 4 head; 64 dim
le-3
30k
NLFirtt, NRFirtt
4 layer; 4 head; 64 dim
le-3
30k
NLLast, NRLast, NE, NB
4 layer: ; 4 head; 256 dim
le-4
30k
Table Hyperparameters for training with APE on transformers trained from scratch
Problem
Model Size
LR
Max Steps
UL, UR, UF, UB NLFirst, NRFirst
2 layer; 4 head; 64 dim
le-3
30k
2 layer; 4 head; 64 dim
1c-3
30k
NLLast, NRLast, NF, NB
4 layer; ; 4 head; 256 dim
le-4
30k
Table 2: Hyperparameters for training with RoPE on transformers trained from scratch
B From Scratch Training Experiments
Here, we validate theoretical predictions about the success of transformers in length generalization on all retrieval and copying tasks, by training small transformers from scratch. We particularly validate that results hold both for APE (as in GPT-2, as originally targeted by the theory of Huang et al. [2025]) and RoPE (as found in various other modern L.Mss.. We adopt the methodology from Huang et al. [2025] to demonstrate all our experiments when training transformers from scratch. Like Zhou et al. [2024], we sample independent training batches on the fly instead of using finite-size training set. In contrast, each test set contains 2000 samples that are sampled the beginning of each experiment.
We train models with the default CausalLMLoss from the transformers library. the length of inputs is sampled uniformly from minimum up to maximum length in the specified range. This is true for training data where the range is (lmin, 50) where lmin is the minimum length possible for a given task (varies from 2 - 4, depending on the task variant and the minimum length of characters required to make the string valid). Our test sets have the size- (Imnn, 50), (51, 100), (101, 150). At cach step, the model outputs the correct continuation - for retrieval retrieved, for copying the subsequent next token from the input sequence. The models are
the correct token trained on whole sequence of tokens. We train decoder-only transformer from scratch, using implementations from HuggingFace transformers for APE (GPT2LMHcaddMode) and RoPE (LlamaForCausalL.M) models for maximum 30K steps with batch size of 64. We stop training early once the model's accuracy reaches 100% on the in-distribution test set (the one in range [Lniin 50). The model is trained with dropout rate of 0.0, and we use AdamW, with a weight decay rate of 0.01 (choices based on Huang et al. [2025]).
We train
For experiments with APE, at training time, we add random offsets to position indices so that all position embeddings are trained (following Huang et al. [2025]). The affsets are sampled uniformly at random in the range [0. 150 - feurr] where feurr is the length of the current string. For RoPE, such an offset is not required in training" 10; instead a scaling factor of 32.0 set; the RoPE scaling type used was linear.
preliminary experiments, we found that different model architectures, while achieving 100% accuracy on in di-diiiribution data, may perform differently on out-of-distribution data. To draw a conclusion about how the model performs on a problem in general, we determine the hyperparameters as follows: We consider configurations of (1, 2, 4) layers, [1, 2, 4) heads and model dimension of (16, 64, 256), and learning rate of [0.001, 0.0001), We sweep all the configurations by iterating over every combination and choose the one that achieves the highest accuracy on [51, 100] among those configurations whose accuracy on [min, 50) is 100%. When there are multiple such options, e.g.. their accuracy on [51, 100) is 100%, the one with the simplest architecture is selected (when estimating complexity, we assume the following priority: number of layers > number of heads > model dimension). The final hyperparameters we used for each task are shown in Table I and After we determine the hyperparameter configuration, we run the experiments with multiple random seeds and report the average accuracy of 3 successful runs (those runs where the model achieves 100% accuracy on in-distribution data).
UF, UB vs UR, UL: While training UF and UB with RoPE, we did not observe perfect length generalization when directly training the task, unlike what observed with APE. However, note that UF and UB can be thought repeated applications of UR and UL, for which we did obtain perfect length generalization even with RoPE- showing that length-generalizable RoPE transformers do exist for UF and UB. !l We thus use the RoPE transformers for UR and UL to model UF and UB. We take separate versions of transformers on UR and
IIn fact, relative positional encodings such as RoPE are by definition invariant to the addition of constant offset to the indices
We leave to future research to determine whether this phenomenon reflects some deeper discrepancy between RoPE and APE, or superficial optimization difficulties.
19

---

UL (without any separator before the query token) and autoregressively use the UR and UL models to generate full continuation till EOS is reached get correct solutions for UF and UB. Therefore format the input = (bos) (sep)ar ... x (ccos((spp) Upon seeing the separator, the model predicts æ1, and then 22 and on till we get to NN, and then an (eos) is predicted. We do the same with UL to get predictions for UB.
In C-RASP[Pos]
Not in C-RASP[Pos]
(Unique Forward)
UB (Unique Backward)
100
100
100
100
100
60
40
40
40
40
40
20
20
20
20
20
Bin 1
Bin
Bin
Bin
Bin 2
Bin
Bin
Bin NLFirst
Bin 3
Bin 1
Bin 2 NRLast
Bin
Bin
Bih
Bin 3
UL
NRFirst
NLLast
100
100
100
100
100
80
80-
80
80
80
60
60
60
60
60
20
20
20
0
0
Bin
Bin
Bin
Bin
Figure Transformers trained from scratch with APE, perfectly aligns with C-RASP[pos] prrdsippssppreiccction predictions.
C-RASP[Pos]
Not in C-RASP[Pos]
UF (Unique Forward)
UB (Unique Backward)
NF
NB
100
100
100
100
100
80
80
40
40
40
40
40
20
20
20
20
20
0-
Bin 1
Bin
Bin
Bin
Bin 2
Bin
Bin
Bin NLFirst
Bin 3
Bin 1
Bin 2 NRLast
Bin
Bin
Bin
Bin 3
UL
NRFirst
NLLast
100
100-E
100
100
80
80
80
80
80
60
60-
60-
60
60
40
0
Bin
Figure 9: Transformers trained from scratch with RoPE, aligns with C-RASP[pos] predictions, even though the length generalizability guarantees given by C-RASP(pos] only apply APE. should be noted that the accuracy on UF, UB plotted here are not obtained by directly training, but by applying UR, UL repeatedly as described the paragraph B. The bins here correspond the evaluation bins of sizes (Imin, 50), (51, 100), (101, 150)).
C Additional experimental details and results
C.1 Prompting: In-Context Retrieval Templates (Section 4.1)
Table 3 below lists the three prompt factors and their possible values. A Cartesian product of all prompt settings across the categories given below taken generate 20 prompt templates.
Separator Examples
The input string is given in the following format based on the category.
SEP: 21 I22...o-Ill
NOSEP 21, 22....nn1120
20

---

Table 3: Prompt Template Grid Factors
Factor
Options
Separator
SEP, NOSEP
Few-shot
SMALL, SAME
Template
BARE, SIMPLE RULE, SIMPLE RULE EXPL., MATH RULE MATH RULE EXPL.
Few-Shot Pools
SMALL Examples drawn from a held-out pool with lengths smaller than the input string X.
SAME Examples sampled from the same evaluation set as the input string X (excluding X itsclf).
Instruction Templates
The different possible instruction templates for each of the tasks are given below. In every case, the prompt ends with the unresolved query line, prompting the model to emit the answer token.
BARE This template provides only the few-shot examples without any additional instructional or explanatory text. It requires the completion model to infer the task directly from the presented examples. Example for U R::
The task is introduced through concise English English-languae description defining the resolution rule. The idea of the setup is the whether the model can follow explicit instructions based linguistic clarity. The format across the tasks as follows.
SIMPLE REPPL__RULE RULE
Each line is written as 'context query: target'. The vertical bar "Ia separates the context from the query token. All strings below follow this rule: (rule_ simple} (examples)
Table 4: Simple Rules for each of the tasks
Task
Template
The answer is the token immediately to the left of the single instance the query token
UR
The answer is the token immediately to the right of the single instance of the query token.
NLLast
When the query appears multiple times, the answer is the token just to the left of its last appearance.
NRLast
When the query appears multiple times, the answer is the token just to the right of its last appearance.
NLFirst
When the query appears multiple times, the answer the token just to left of its first appearance.
NRFirst
When the query appears multiple times, the answer is the token just to the right of its first appearance.
SIMPLE RULE EXPLAINED This template enhances the simple English rule format by providing an additional worked example explanation. It begins by stating the simple rulc, then illustrating through example how the rule actually applied. Following the same, the model presented with some few-shot examples to reinforce the rule understanding.
This template explicitly formulates the task using formal mathematical definitions and notation. It introduces variables to clearly define the input string, context, query token, token indices, and continuation token. The mathematical notation precisely indicates conditions under which the continuation token should be selected, leaving no ambiguity regarding the required solution path. The template follows.
MATH RULE
Let $X - x_1 Aldots x_n$ with $n Age 4$ and $x_1 Vin \Sigmat (token vocabulary). The final token $x_n$ is the query token $q$. In the context $x_1 Aldots x_ {_(n-1)) $q$ appears $t$ times at indices $q_1, Vldots, q_t$ ($1 Ale q_1 Ale q_t Mie n-18). Continuation taken $x_[(n+1>}$ is defined by: (rule_math) Examples: {examples)
MATH RULE EXPLAINED Similar to the SIMPLE, SIMPLE_RULE BULE_EXPLAINED this template begins with a detailed mathematical formulation of the task and is then followed by worked-out example and then finally some sew-shot set of strings.
21

---

Table Math Rule Template
Task
Template
UL
[ and In+1 2gg -1
UR
t t= and In+1 = 2q: +1
NLLast
t and Ix+1 Zz -1
NRLast
t> 1 and In+1 = Zge
NI NLFirst
1> and Tx+1 = Zig
NRFirst
t> and In+1 Zig +1
C.2 Prompting: In-Context Copying Templates (Section 4.1)
Template variants. For our In-context Copying task suite, we use three lightweight prompt styles. There is no Cartesian grid as in retrieval, each of the prompts had statistically little variance, and eliciting this ability from our LLMs
challenging in retrieval.
BARE - few-shot examples only, no instructional text.
OBEY one-line English rule (rule [ill_ssmppe) introducing the few-shot block.
HINT - short hint (rule hint) that highlights the ouput- relation; mainly useful for backward copying.
Rule strings injected by each template. The phrases referenced above drawn from Table 6. Note that unique and non-unique regimes share identical wording because the operation (copy reverse) is independent of token repetition once input fixed.
Table 6: rule_ rule_simple phrases for the four copying tasks.
Task
rule_simple
UF
The output is exactly the same sequence as the input.
UB
The output is the input sequence written in reverse order.
NF
The output is exactly the same sequence as the input. The output is the input sequence written in reverse order.
NB
The corresponding rule_hi cule_hint strings swap the lead-in "The output ...." with "In every example the output to make the instruction less imperative.
Few-shot sampler. All copy prompts use the same held-out 5-hos pool (length L=5, N 1500) that is disjoint from the evaluation set. Each prompt is built by picking five random lines from this pool. followed by the query built from the current test string.
End-to-end prompt bare template example (UF)
<bos> r
s
<eos>
<bos> H
H
<eos>
<bos>
<eos>
<bos>
a
<eos>
<bos> m
N
r
m I
r D
<eos>
<bos> Z
i B
The unresolved line after is the model's query: generation proceeds with greedy decoding (1' 0).
C.3 Longer Vocabulary (Word-Level) Experiments (Section 4.1)
To ensure that the Uniqueness and Directional Biases observed in our primary character-level experiments are simply artifacts of synthetic setup, we conducted a parallel of experiments using naturalistic word-level vocabulary. case of LLMs, the distinction between token-level and word-level task may be less sharp than it appears. Kaplan et al. [2025] shows that LLMs form an "inner lexicon" where simple or common words are processed as cohesive semantic units. Given their results, this word-level task formulation can be considered analogous the character-leve task.
22

---

Experimental Setup The setup for the word-level vocabulary mirrors the main experiments in Section 4.1 but replaces the character-based vocabulary with vocabulary of 300+ unique English words. This allowed construct longer, more complex sequences with lengths up to 300 tokens. fow-shot prompting strategy, and task structures (c.g. UR vs. UL, UF vs. UB) remain identical. The exact vocabulary we used to generate the test strings in all of the experiments is as follows.
other aspects of methodology.
Character-level vocabulary
a, a, a, a, å, 8, b, c, 5, 6, e, 6, 8, d, e, è,
a, ē, ê, a, Φ, f, 8,
8,
8, h, h, i, 1, j, ]. x,
1, n, 6, 6, 6, 6, P. 9,
8, u, ú, ú,
ü, ü, Y, I, y, 2,
A, h, h, A, A, h E, B, C, G, ć,
C, c, 8, D, E,
E, E, E, E, F, G, H, I, I, 1, 1, I, I, 1, I, J.
K, L, L, M, N, 8, 0, 0, 0, 0, P,
Q, R, 8, T, U,
0, 0, 0, 0, V, W, X, Y, 2,
0, 1, 2, 3, 4, 6, 7, 8,
Word-level vocabulary
apple, ant, arrow, anchor, artist, animal, angle, apricot, arch, armor, axis, avenue ball, bat, book, bridge, bottle, bucket, bench, bread, bell, button, brush, branch cat, car, cup, cloud, clock, candle, coin, chair, circle, crown, castle, cookie dog, door, desk, drum, duck, doll, diamond, dish, dress, dream, drop, dust egg, ear, eye, engine, elbow, earth, envelope, exit, echo, edge, event, energy fish, fan, fork, flower, flag, feather, fire, frame, forest, farm, fruit, fence goat, game, glass, glove, gate, garden, gift, grape, guitar, gold, gear, group hat, hand, horse, house, hill, hammer, heart, honey, hook, horn, hug, hope ice, iron, ink, island, idea, image, item, ivory, icon, input, isaue, idol jar, jam, jet, jewel, jungle, jacket, juice, job, joke, joy, judge, jump kite, key, king, knee, kitchen, knife, kitten, knight, kick, kettle, kind, koala lamp, leaf, lion, lock, ladder, lake, lemon, line, letter, lip, light, lunch man, map, moon, milk, mouse, mirror, mountain, market, meal, music, magnet, match net, nose, nest, name, nail, night, number, note, neck, nurse, noise, nation owl, oil, oven, orange, ocean, order, orbit, open, option, owner, object, office pen, pig, pot, plate, plane, pumpkin, pearl, park, path, piano, point, paper queen, quill, quiz, quilt, quiet, quick, quote, quest, queue, quake, quart, quark rat, ring, rain, river, rope, road, rose, rock, rule, room, root, radio sun, sock, star, ship, shoe, stone, sugar, song, salt, sand, seed, snake top, toy, tree, train, table, tube, tiger, tool, time, tent, team, towel urn, use, unit, uncle, under, upper, uniform, union, urban, urge, ultra, usual van, vase, veil, voice, valley, visit, value, vest, vote, view, vine, victory wax, web, wall, wind, water, wheel, wave, wolf, wing, worm, word, wood xylophone, xerox
yam, yard, yarn, yawn, year, yellow, yogurt, yolk, youth, yield, yeti, yoga zoo, zebra, zone, zero, zip, zinc, zeal, zest, zigzag, zoom, zombie, zodiac
C.4 Additional results and plots for in-context prompting (Section 4.1)
C.4.1 Additional results: character-level vocabulary
As evident from all the figures here (smaller completion models Figure 10), (smaller instruct models Figure 11), (bigger instruct models- Figure 12), the Directional Bias persists across models sizes and types. While in the smaller models the gaps smaller for the retrieval tasks, nevertheless there still pronounced gap within the copying task variants.
Amongst the prompts there was high variability for all our retricval setups, but close to none for the copying setups. Our bare template performed the worst in most retrieval setups, and the highest accuracy metrics were achieved when the instruction template was MATH B_____RR_______ EXPLLAEED,, with the few shot setting SMALL with the presence of separator. For copying, just the BARE setup was enough get high accuracy numbers without the need to specify any instructions (especially for the forward copying cases). Providing instructions did improve the performance especially for the Qwen2.5-Instee Instruct models.
We also carried out the same experiments with newer Qwen3 [Team, 2025) models with "thinking" variants to see if our findings still hold. The results from Figures 13, 14. 15. 16 show that even the model trained to explicitly generate chain-of th-thotght for solving task still have the same Directional and Uniqueness biases.
23

---

-a-mmama 3 8B (Lot/Bwd)
lima 3 (RighuFwd)
-- -e-Qwan 25 78 (Leh/Bwd)
Qwan 2.5 78 (Righi/wad)
Retrieval
Copying
UU/UR
UBAJF
1.0
0.8
Unique
Accurapy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
10
20
30
40
50
100
Impult
NLFrrs/NRFirst
NILas//NRLast
NB/NF
1.0
Non-unique
0.8
Accuracy
0.6
0.0
10
20
20
100
Impul Length
Input Length
Input Length
Figure 10: In-context accuracy for Qwen2.5-o 5- and Llama-3 3-ma-3-883 8B across all our character-level tasks averaged seeds and prompt variations per task. The same patterns as discussed in Section 4.1 are visible.
-Lium 88
(RighoFwd)
Retrieval
Copying
UUUR
UB/UF
1.0
Unique
Accuracy
0.6
0.2
0.0
20
30
40
50
100
10
20
30
40
50
100
Length
NUFrrstNRFFist
NLLasWNRLast
NEINF
Non-unique
0.8
Accuracy
0.4
0.0
30
100
100
Input Langth
Inpurt Lorgth
LnpurLonngh
Figure 11: In context accuracy for our character-level tasks averaged over seeds and prompt variations per task. The same patterns as discussed in Section 4.1 are visible.
and Saama-3-8B-Intaaee across all
24

---

- a 708 Instruct (Leivtted)
--- Liama 708 Instruct (RightFwe)) --- -Qwwen 2.5 328 Instruct (Lefv/sww)
Qwen 2.5 328 Instruct (Righ-/Fww)
Retrieval
Copying
0.8
Unique
Acouracy
0.6
0.4
0.9
10
20
30
40
50
100
10
20
30
40
50
100
Langh
Lenghh
NLFirs//RFiist
NLLastNRLLstt
1.0
Non-unique
0.8
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
100
10
20
30
50
100
30
20
30
40
50
100
Lengtht
Input Length
Irpur
Figure 12: In context accuracy for Qwen2. Twen2.5-321 5--25--Innttuue Instruct and Llama-3 --t-aa---------s-u- across all character-level tasks averaged over seeds and prompt variations per task. The same patterns as discussed in Section 4.1 are visible.
- Owen 3 14B (LelvBwd)
Owen 148 (KightFFwd)
Retrieval
Copying
UUURR
UBAUF
Unique
Accuracy
30
40
50
100
10
20
100
Impur Length
Input Length
NLFirst/RFFist
NLLast/NRLast
NE/NF
1.0
Non-unique
0.8
Acouracy
0.6
0.4
0.2
0.0
10
20
30
50
100
40
50
1.00
10
20
50
100
Impul Length
Input Length
Inpul Length
Figure 13: In-context accuracy for Qwen3-144 -1BB across all our character-level tasks averaged over seeds and prompt variations per task. The same patterns discussed in Section 4.1 are visible.
25

---

-vvQuen 4B (LeftBwd)
Qven 4B (RightFww))
Retrieval
Copying
UD/RR
UBAFF
1.0
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
1.00
10
20
30
40
50
100
Impuli Length
Input Longth
NLFirs//RFirst
NiLass/NRLast
NB/NF
1.0
Non-unique
0.8
Accuracy
0.6
10
20
100
40
50
100
10
20
100
Impui Length
Input Length
Input Length
Figure 14: In-context accuracy for Qwen3-4B across all our character-level tasks averaged over seeds and prompt variations per task. The same patterns as discussed in Section are visible.
atvRRd)
Thinking (Righ/Fwdd)
Retrieval
Copying
UL/UR
UBAJF
1.0
0.8
Unique
Acouracy
0.6
0.4
0.2
0.0
40
50
100
20
50
100
Imput Length
Inpul Length
NLFrss/NRFirst
LRst/NRLast
NB/NF
Non-unique
Accuracy
10
20
30
100
40
50
1000
10
20
50
100
Input Length
Length
across all character-level tasks aged over seeds and prompt variations per task. The same patterns as discussed in Section 4.1 are visible.
Figure 15: In context accuracy for
26

---

Owen 148 Thinking (Lefväwd)
14B Thinking (Righ/Fwu)
Retrieval
Copying
UUURR
UB/UF
1.0
0.8
Unique
Acouracy
0.6
0.4
0.2
0.0
40
50
20
50
100
Input Length
Input Length
NLFrrs//RFrrst
NiLass/NRLsst
Non-unique
Accuracy
10
20
30
100
10
30
40
50
1000
10
20
30
50
100
Length
Length
Figure 16: In context accuracy for Qwen3-14B 1113--4B-Thinkinn across all character-level tasks averaged over 3 seeds and prompt variations per task. The same patterns as discussed in Section 4.1 visible.
C.4.2 Results: word-level vocabulary
find the core asymmetries persist even more naturalistic word-level setting (see Figures 17, 18. 19, 20, 21, 22, 23, 24). Although length generalization improves slightly for the larger models, the pretraining induced Directional Bias and the architecture-ding -erivverrrrreeennneeee Uniqueness Bias remain clearly evident across all LLMs.
C.5 Promping: Lorem Ipsum Copying (Section 4.2)
Dataset generation. To stress-test copying while keeping text realistic, we construct a 1 1,500-example corpus of synthetic Lorem /psum paragraphs. Generation starts from one hase paragraph produced by the lorem Python library and then applies light, probabilistic perturbations increase the non-determinism of the texts.
Sentence count. Each sample contains exactly 45 sentences (average ~350 tokens), enforced by itcrative expansion/shufflng until target length is reached.
Word/discourse noise. With probability 0.3 random sentence is repeated up to four times; with probability 0.5 random words inside sentence duplicated; with probability 1.0 words within sentence shuffled (except the leading capital).
Token budget. Sequences are truncated to 500 BPE tokens to stay within context limits while preserving paragraph-level coherence.
Prompt variants. We identified successful prompts for copying naturalistic text as follows. We downloaded research papers put up ArXiv in April 2025, and checked if the prompt results in perfect accuracy (not even single mistake). Our motivation was that ArXiv includes challenging text with equations, and, due recency, should not have appeared the LLM training data. More specifically, we segregated the sections of the papers into paragraphs, making sure the total length of the paragraph to be copied has close to 500 tokens (while ensuring that we do not end the paragraph in the middle of a sentence). We then pre- validated our prompts dataset containing 500 samples each containing 500 tokens, and the smaller models Llama3. 1-8B and Qwen2. 7B perfectly copied given text (not single mistake). We proceeded with our minimal prompts, shown in Table 9, which achieved perfect copying accuracy. Each of our prompts ends with unresolved <start> token, prompting the model to emit a verbatim repetition of the preceding paragraph.
Ratio of Ambiguous vs. Unambiguous Tokens. Table reports, for each model, the average number of ambiguous (tokens with multiple possible continuations) and unambiguous (tokens with single continuation)
27

---

---Liama Bld))
Liama (Right/Fw))
Quen 32B (Lob)/Bd)
2.5 ghi/FFud
Retrieval
Copying
UUJUR
UB/UF
1.0
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
200
300
10
20
30
40
50
100
200
300
Input
Input
NL NisiiisiRiiistt
NLLasuNRLast
NB/NF
1.0
Non-unique
0.8
Accuracy
0.6
0.4
0.2
0.0
20
40
100
200
300
10
40
100
200 300
50 100 200 300 Input Length
Input Length
Length
Figure 17: In context accuracy for Qwen2. 3-B and Llama 3- 4-70B across all our word-level tasks averaged over 3 seeds and prompt variations per task. The same patterns discussed in Section 4. are visible.
-e-Liama 3 8B (Let/Bwdd)
Liama 9B (RighuFwd) --
-e-Qwen 25 78 (Lef/Bwd)
Owen 2.5 7B (Right/Pad)
Retrieval
Copying
UL/UR
UBAJF
100
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
200
300
10
20
30
40
50
100
200
300
Impul Length
Length
NLFirs//RFiist
NiLass/NRLast
NB/NF
1.0
Non-unique
0.8
Accuracy
0.6
Impui Length
Inpur Length
Input Length
Figure 18: In-context accuracy for Qwen2. 2- 5-7B and Llama-2 8B across all our word-level tasks averaged over 3 seeds and prompt variations per task. The same patterns discussed in Section 4. are visible.
28

---

-*-Likma 88 Instruct (lo))
amm 86 Instruct (RighoFwd)
-B- Qwen 2.5 instruct (1 (LahBwwd)
Owen 2.5 78 insorctt (RighoFwc)
Retrieval
Copying
UUUR
UB/UF
0.8
Unique
0.2
Ingur Langth
Inpurt Length
NLFirsiNRFistt
NLLasiNRLees
NE/NF
1.0
Non-unique
Accuracy
0.6
0.2
0.0
50 100 Imput Length
200
300
10
20
30
40
50
100
200
300
10
30
40
50
100
200 300
Lnnghh
Inpun
Figure 19: In context accuracy for 5ren2.5-7B. Instructt and Llama-3--00
across all
word-level tasks averaged over 3 seeds and prompt variations per task. The same patterns as discussed in Section 4. are visible.
-L-mma (Lef/Red)
(RightFwl)
(LahhBoo))
(RighWFFod)
Retrieval
Copying
ULJUR
UBNUF
1.0
Unique
Accuraoy
0.8
0.2
0.0
300
100
Langhh
Impur Langth NBINF
NLFirst/RFFist
NLLasUNRLast
1.0
Non-unique
0.8
Accuracy
0.6
0.4
0.2
0.0
Length
Imput
Figure 20: In context accuracy for ---e2..-3-B--nae and Llama-3-700 --B Instruct across all our word-level tasks averaged over seeds and prompt variations per task. The same patterns as discussed in Section 4.1 are visible.
3--OB-Instruct
29

---

- Qwen 3 14B (LefVBwd)
Qwen 148 (RighUFwe)
Retrieval
Copying
UU/UR
UBNUF
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
200
300
10
20
30
40
50
100
200
300
Impult
Input
NLFrrs/NRFiist
NiLas//NRLast
NE/NF
1.0
Non-unique
0.8
Accuracy
0.6
0.0
300
Impui Length
Input Length
Input Length
Figure 21: In-context accuracy for Qwen3-14B across all our word-level tasks averaged over 3 seeds and prompt variations per task. The same patterns as discussed Section 4.1 are visible.
-+ -e-Qwen 3 4B (LefvBwd)
-
Qven 3 4B (RightFwed)
Retrieval
Copying
UU/UR
100
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
200
300
10
20
30
300 Length NB/NF
40
50
100
200
Impul Length
NLFirs/NRFirst
NRLss
1.0
Non-unique
0.8
Accuracy
0.6
0.2
0.0
200
300
Impur Length
Inpur Length
Input Length
Figure 22: In-context accuracy for Qwen3-4B across all our word-level tasks averaged over 3 seeds and prompt variations per task. The same patterns as discussed in Section 4.1 are visible.
30

---

--- Owen 48 Thinking (LefvBwd)
- Qwen 4B Thinking (RighUFwd)
Retrieval
Copying
UL/UR
UBAFF
0.8
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
200
300
10
20
30
40
50
100
200
300
Impul
Length
NLFiss/NRFirst
NiLas//NRLast
NE/NF
1.0
Non-unique
0.8
Accuracy
0.6
0.0
300
Impui Length
Input Length
Input Length
Figure 23: In context accuracy for (wen3-4B-Thine across all our word-level tasks averaged
over seeds and prompt variations per task. The same patterns discussed in Section 4.1 visible.
- -wen 3 14B Thinking (LefvBwd) -
Qwen 14B Thinking (Right/Fwd)
Retrieval
Copying
UL/UR
UBNJF
100
Unique
Accuracy
0.6
0.4
0.2
0.0
10
20
30
40
50
100
200
300
10
20
30
40
50
100
200
300
Length
NLFirs/NRFiist
NRLstt
NB/NF
1.0
Non-unique
0.8
Accuracy
0.6
0.2
0.0
200
300
Impur Length
Inpur Length
Input Length
Figure 24: In context accuracy for Dxen3-11BBThinkinne across all our word-level tasks averaged over seeds and prompt variations per task. The same patterns discussed in Section 4.1 visible.
31

---

tokens per paragraph, aggregated across all random seeds. While different model families employ distinct tokenizers, so their absolute counts vary, yet the relative balance between ambiguous and unambiguous tokens remarkably consistent. Each paragraph contains roughly 500 tokens, but repetition of sentences (to make the dataset appropriate for Non-unique copying) reduces the number of unique token types to only about 70. However, amongst those, both token types occur frequently in our data and the fact that copying "glitch" can be traced exclusively unambiguous tokens is remarkablc.
Table 7: Average per-paragraph bigram counts by model
Model name
Ambiguous tokens
Unambiguous tokens
Total tokens
Llama3_800 Llama3_700 1_0 Qwen2. 5_7B, Qwen2. 5_32B
31.97
39.65
71.62 74.72
32.59
42.14
Example of Hallucinated Chain. We provide here truncated real sample (from the total 500 sequence length the original paragraph) of an input sequence to be copied:
neam que Non etincidunt dolorem tempora magnam.
the corresponding output we get for same imput is:
neam que Non etincidunt dolorem tempora magnam velit neque. Non etincidunt dolorem tempora magnam.
As can be seen, a long chain of hallucination starts at the point tempora magnam. However, earlier the truncated history, there were cases where magnam was followed by velit.
Our algorithm proceeds with this example as follows. We first tokenize both sequences to get the following lists of tokens in the input and output. Here, the symbol G denotes a leading space for any token:
Input: ['Gne?, 'am?, 'que', '.,, 'GNon', 'čet?, 'inc?, 'idunt', 'édol', forem?, 'Gtemp', 'ora', 'Gmagn', 'am', (.
Output: ['Gne', 'am', 'que', ',, 'GNon', 'Get', 'inc', 'idunt', "dol', 'orem', 'Gtemp', 'ora', "Gmagn', 'am', 'évelit', 'Gne', 'que', 'Get', 'inc', 'idunt', 'Gdol', 'orem', 'Gtemp', fora', 'Gmagn', fam", {.
'Glonn,
The alignment obtain between input and output tokens is:
[('match' 0, 0), ('match' 1, 1). ('match' 2, 2), ('match' 3, 3),
('match' 4, 4), ('match² 5, 5), ('match' 2 6, 6), ('match' 7, 7),
(smatch' 8, 8), ('match' 9, 9), ('match' 10, 10), ('match' 11, 11),
('match² 12, 12), ('match' 13, 13), ('insert', None, 14), ('insert', None, 15),
('insert', None, 16), ('insert', None, 17), ('insert', None, 18),
('insert', None, 19), (finsert', None, 20), ('insert', None, 21),
('insert', None, 22), ('insert', None, 23), ('insert', None, 24).
( ('insert?, None, 25), ('insert', None, 26), ('insert', None, 27)
After grouping the alignment by operation, we get:
[('match' (0, 13)), ('insert', [14, 27))]
Finally, we analyze the transition index between these groups, which here is index 13. The token index is the word am, which appeared earlier in the context and was followed by other tokens and not a period (.). Since this transition index at am is ambiguous, we classify the source of this hallucination chain as ambiguous.
C.5.1 Additional results for longer lengths
We conducted additional experiments with the same Lorem- Lorem-Ipsum Ipsum setup described above for a single seed even longer input lengths (up to 5000 tokens) and report the results in Table 8. We observe the same effects we see at lengths of 500, that is, lower success rate for ambiguous bigrams than for unambiguous ones. These results longer inputs align perfectly with and amplify findings in Section 4.2.
32

---

Table 8: Model Performance Longer (2k/3k/5k) Input Lengths
Unambiguous
Ambiguous
Model
2k
3k
5k
2k
3k
5k
(lama3-70B-Instmunn
0.99
0.99
0.99
0.96
0.94
0.76
-lama3-8B-Instreee
0.96
0.98
0.99
0.89
0.69
0.44
gwen2.5-32B-IIm.
1.00
0.99
0.99
0.95
0.94
0.93
(wen2.5--BBInsttre
0.99
0.99
0.99
0.75
0.43
0.19
Table 9: Prompt templates for copying Lorem Ipsum styled text. input stands for the test paragraph. All templates terminate immediately after the final <start> token, leaving the closing tags for the model to complete. The example paragraphs along with their copies are written in full in the templates for few-shot examples (omitted here for brevity)
Name
Template body
ECHO-TAG
RULE Paragraphs repeated **exactly** is, whenever
Echo seen.
Paragraph:
<start> Once upon time
<end>
Echo:
<start> Once upon a time
<end>
Paragraph:
<start> input <end>
Echo:
<start>
VERBATIM-ENT
Paragraph:
<start> input <end>
Provide a verbatim copy of the paragraph above exactly as it appears, with no modifications. Use the tag <end> to mark the end of the response.
<start>
P-TAG-THE, END
Please repeat the following paragraph exactly as it is, without any changes:
input
Put the repeated paragraph between <paragraph> and </paragraph> tags. Use the phrase "THE_END" to mark the end of the response.
<paragraph>
C.6 Prompting: Git Commit Revert ys Cherry-Pick (Section 4.2)
Dataset generation. We synthesize commit logs by creating randomized seven-character hexadecimal hash, an action/topic message drawn from 12 verbs and subsystems. history snipper lists commits newest,
oldest. We export four depths 10, 15, 20, 25) and three random seeds, yielding 4 3 x 1, 5.00 18, 000 total examples. For each snippet consisting of this generated git commit history (listed newest - oldest) we store two reference answers:
33

---

revert:
the lines shown (newest,-v,oldes) matching the order a developer would pass to git (thus somewhat similar forward copying)
cherrypick: the same lines reversed (oldest, +,newest), the order expected by git cherry-pice (thus somewhat similar to backwards copying)
All files are distributed in json1 format with keys snippet. revert, and cherrypick.
Prompt variants. We adopt exactly two natural- natural-languue language templates; one for the cherry-pick orientation, one for the revert orientation. Each prompt describes the task in plain English, includes up to three few-shot examples (hand-crafted, = 6), shows the new history under ==- NEW HISTORY =-- delimiter, and terminates with ANSWER
where the model must emit the ordered commit list only. An example given below.
Prompt template examples Cherry Pick
Below is part of Git commit history (newest at the top, oldest at the bottom).
Task
List every commit line that must be cherry-picked onto the release branch. Output them **from oldest to newest** (the order 'git cherry-pick' expects). One complete commit 1ine per output line.
Do **not** output anything else.
{ block }}
=== NEW HISTORY ==
{ snippet }}
ANSWER
<start>
Prompt template examples Revert
Below is part of a Git commit history (newest at the top, oldest at the bottom).
Task
List every commit 1ine that must be reverted to roll the codebase back. Output the lines **from newest to oldest**, One complete commit line per output line. Do **not** output anything else.
{{ few_shot_blen block }}
NEW HISTORY
{{ snippet
ANSWER ==
<start>
Because correct output demands global re-ordering rather than token-level copy, we found no benefit in adding separator toggles or mathematical re-statements.
Model coverage. Preliminary sweeps revealed that Qwen2.5 7B/32B models performed really poorly < 10% this task, farther shorter depths & = 5. hence exclude them from our analysis. Performance degrades further with longer historics, indicating difficulty with list reversal rather than context window length.
Additional prompting details for Instruction-tune model variants. We keep the prompts very similar to the completion models and mainly change the output formatting instructions (i.c., Put the answer between <target>, <\target> tags). Additionally, both Quen and Llama family of models require and an additional SYSTEM PROMPT, which we provide as follows:
You are very careful and precise assistant. always follow the instructions and solve tasks yourself. You never generate code.
With our initial experiments, we observed that unless we specify explicitly, instruction tuned model variants generated code solve both copying and retrieval tasks.
34

---

D Fine- Sane-Tuning Details (Section 4.3)
In general, the random seeds only affect the data shuffling and the positional offset during fine-tuning: the rest of the things remain unaffected.
D.1 Retrieval Tasks UL/UR
Examples.
Some sample inputs with the query token (bolded) and their UL/UR answers
Input
UL answer
UR answer
ns0w6up9v8 | lu
6
p
qyx283zd9411w8
13
Z
Dataset.
The training/test dataset statistics for the UL/UR tasks follows:
UL
UR
# train/val/test samples train/val sample lengths [min, max] test sample lengths [min, max]
45k/5k/Sk
45k/5k/5k
[4, 100]
[4. 100]
[101, 200]
[101, 200]
Hyperparameters. dataset.
The table below contains all the hyperparameters for fine-tuning models the UL/UR
Hyperparameter
Value
Model name
GPT-2-XL
Tokenization
Character-level
Maximum train sequence length
100
Maximum inference sequence length
200
Epochs
30
Batch Size
64
Learning rate (AdamW)
le-5
Weight decay (AdamW)
0.01
Warmup ratio
0.15
Random seeds
3. 71, 92, 435, 541, 591, 24050, 29214
Maximum gradient norm
1.0
D.2 Retrieval Tasks NLFFiss/RRFis//LLaaiiis
Examples.
Some sample inputs with the query token (char after 'II') and their NLFFI//NRFis//NLLL///RR
Last targets.
Input
NLFirrt
NRFirst
NLLaat
NRLast
q50008b6v503 lo 5
0
5
c8r5r5r3r6r0 Ir
8
5
6
0
Dataset.
The training/test dataset statistics for the NLFins//RFFis//LLLLs//LLLsst tasks are as follows:
# train/val/test samples
100k/5k/4.7k
train/val sample lengths [min, max] test sample lengths [min, max]
[4, 100] [101, 200]
35

---

Hyperparameters. Firs/NRRins//LLLL//RRLL... dataset.
The table below contains all the hyperparameters for fine-tuning models on the NL-
Hyperparameter
Value
Model name
GPT-2-XL
Tokenization
Character-level
Maximum train sequence length
100
Maximum inference sequence length
200
# Epochs
15
Batch Size
64
Learning rate (Adam W)
le-5
Weight decay (AdamW)
0.01
Warmup ratio
0.15
Random seeds
3, 47, 71, 100
Maximum gradient norm
1.0
D.3
Copying Tasks UF/UB/NF/NB
Examples.
Some sample inputs and their UF/UB/NF/NB targets.
Input
Config
Target
Syb5DEH1h0>
UF
Syb5DEH1h0
Syb5DEHih0>
UB
OhiHED5byS
LnvTsiggMt>
UF
LavTsiggMt
LnvTsiggMt>
UB
tMgq1sTvnL
9975813713>
NF
9975813713
9975813713>
NB
3173185799
525671167>
NF
525671167
525671167>
NB
761176525
Dataset. The training/test dataset statistics UF/UB/NF/NB tasks are as follows:
UF/UB/NE/NB
# train/val/test samples train/val sample lengths [min, max] test sample lengths [min, max]
50k/5k/5k
[4. 100]
[101. 200]
The train/val sample lengths are considered without the delimiter the combined input/target length is 100. not just input length.
Hyperparameters. F/BB dataset.
The table below contains all the hyperparameters for fine-tuning models on UF/UB/N-
Hyperparameter
Value
Model name
GPT-2-XL
Tokenization
Character-level
Maximum train sequence length
100
Maximum inference sequence length
200
# Epochs
15
Batch Size
64
Learning rate (AdamW)
le-5
Weight decay (AdamW)
0.01
Warmup ratio
0.15
Random seeds
33,
71, 77, 91
Maximum gradient
1.0
36

---

D.4 Results
We provide the training loss curves for each fine-tuning task in Figure 25.
UL
NRFirst
NLLast
UR
NRLast
NFwdCopy
FwdCopy
NLFirst
NBwdCopy
8
BwdCopy
Train Loss
4
2
5000
10000
15000
20000
25000
Steps
Figure 25: Averaged training loss curve for GPT 2-- model across our tasks. For all the tasks, we achieve 0 training loss. Yet, the generalization behavior varies in line with theoretical predictions (Figure 6).
E Further Details on Mechanistic Interpretability Analysis (Section 4.4)
Dataset. We randomly generate 100 strings containing letters of the English alphabet in lowercase. To prompt base models to perform the tasks, we add 10 examples before the actual query for few-shot learning. We tokenize each symbol individually, adding preceding white space to cach letter to get more natural tokenization. We add a BOS token in the beginning, separate few-shot examples with dot, and separate the input and output of each example with comma. Fine-tuned models the same dataset, except that tokenization and the choice of separators are inherited from fine-tuning setup, and they also have no BOS token and not require few-shot examples.
Patching details. remove an induction head, we use path patching methodology, widely adopted in the literature [Hanna et al., 2023, Wang et al., 2022]. An induction head can be viewed as a path consisting of two edges connecting different positions inside an attention head: one edge connecting two adjacent positions in the input string, and one edge connecting the second of these with position in the target string. We remove paths inside attention heads following Bakalova et al. [2025]. Removing all induction heads thus done two forward passes:
In all heads, replace the V activation of a token at position / in the input string with zero whenever is queried by the token in position P + 1 in the input string. In the same forward pass. for each head, K and V activations at position P 1 the input string. Intuitively, this intervention prepares K and V activations at each position that carry information about the preceding token.
In a second forward pass, each head, replace the K activation of the token in position P + in the input string with the activation saved in the previous step whenever it is queried by token in position P in the target string. Intuitively, this intervention ensures that head cannot attend to position on basis of information about token immediately preceding the key position. That precisely the behavior of induction heads [Olsson el al., 2022], i.e., induction heads are removed by this intervention.
When removing anti-induction heads, the first step is the same, but the second step is different: In each heads, we replace the V activation of the token in position P in the input string with the activation saved in the first step whenever it queried by token in position + the target string. Intuitively, this intervention ensures that an attention head attending to a prior occurrence of the same token cannot retrieve the immediately preceding token.
The above description applies to Unique Forward Copy: in Unique Backward Copy, the position indices in the target string appropriately reversed.
37

---

Unique Forward Copy
Unique Backward Copy
Artention Score
Induction Head
2.5
Anti-Induction Head
0.0
0
10
20
30
40
10
20
30
40
Layer
Layer
Figure 26: Difference in sum of attention scores per layer between fine-tuned and base models. The higher the difference, the more amplified is this head in this layer during fine-tuning.
Layer localization. In fine-tuning, useful heads may migrate upward in depth (Figure 26). In fine-tuned models, induction (resp. anti anti-induction) strength peaks in the cop-hhir layers for forward (resp. backward) copy. Pretrained models exhibit the same pattern but with induction heads situated later than anti-induction heads, mirroring their superior robustness.
Relation to Retrieval Heads. Recent work by Wu al. [2025] introduced the notion of retrieval heads. experiments to find these retrieval heads Wu et al. [2025] focus primarily on zero-shot settings compared to the typical few shot settings for induction head studies [Elhage et al., 2021, Olsson et al., 2022, Song et al., 2025, Crosbie and Shutova, 2025] including setup. These retrieval heads are found by looking at tokens being retrieved from Needle Haystack like test beds [Liu et al., 2024b]. There is needle, span needs to be copied the output given query. These answer spans needles consist of tokens that are sufficienlly unique in the haystack (context) and thus cannot be figured out on the basis of semantics alone. head is called retrieval head if it pays maximal attention to tokens being copied from a needle more number of times than any other head. For example, if the number of tokens be copied from needle is 10, and the maximal retrieval paid maximal attention to
short answer
be 0.7, that would imply that the head in question out of the 10 tokens while they were being copied. Thus even though Retrieval heads are highly connected to the way we calculate our induction and anti-induction circuits, the exact methodology is different. However retrieval heads should have a strong connection with the induction heads, when they perform forward copying, and the anti-induction heads when they perform reverse copying. The exact correlation however left for future work.
that calculated
F Compute Resources
We all the prompting experiments using 4B/7B/8B parameter models single H100 GPU and 14B/32B/70B parameter models 4xH100 GPUs with full precision without any quantization. Our fine- tuning experiments also use single H100 GPU for all the experiments. All our experiments should be reproducible with approximately 2500 H100 GPUh.
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction assert that (1) fundamental length- gengraaaeeerllizationn limits of small transformers may persist in large-scall scale pretrained LLMs (2) pretrained models cxhibit notable induction-vs sistoon----att--iie anti-induccton asymmetry; (3) this asymmetry can be traced to the relative strength of underlying attention circuits; and (4) targeted fine-tuning can rebalance these circuits and mitigate architectural limits, but does not entirely eliminate them. Through a mixture of synthetic as well real world experiments in section 4.1, 4.2 we show point 2. Through our finetuning experiments in section 4.3 we justify point 4. We justify point with experiments and discussion in section 4.4. Finally in totality the cumulation of of our experiments and theoretical results justifies point 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations of our work in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors encouraged create separate "Liimitaions"" section in their paper.
The paper should point out any strong assumptions and how robust the results are violations of these assumptions (c.g., independence assumptions, noiseless settings, model well well- specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection,
outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Question:
Answer: [Yes]
39

---

Justification: We base our theory the extension of Huang et al. [2025]. In the main body of the paper we provide informal theorems and intuitions behind our ideas (Section 3). Detailed definitions, proofs, and remarks follow in Appendix Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referee roos-reffeenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core
paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide technical details of tasks, in-context and fine-tuning experiments in Section 4. Specifically, we address the setup the copying tasks in subsection 4.2 and fine-tuning setup in subsection 4.3. Additional details on prompting and fine-tuning are provided in the Appendix C and D respectively.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on contribution, reproducibility can accomplished various ways. example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either provide access
make possible for others to replicate the model with the same dataset, the model. general. releasing code and data often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
new model (e.g., large language model), then there should either be
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We attach the code and data used all across our experiments in the supplementary section. We also have provided sufficient details all throughout the paper in each section to help the readers reproduce any result.
40

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.c// cc/public/ https//oooeenninonnneeeeeeeeeeee
for more details.
While we encourage release of cnde and data, we understand that this might not possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-souree benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :ttt::://nigg //nips/nppp..// cc ec/publie/ quides/Codeeuuben sgionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided exact hyperparameters in the appendix, along with all relevant training and test details for all experiments considering training in the appendix in sections D, B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide exact error bars in our experiments whenever relevant. use at least seeds for each experiment of ours to confirm their validity.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
assumptions made should given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of crrors is not verified.
For asymmetric distributions, the authors should he careful not to show in tables or figures
symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
41

---

8. Experiments compute resourees
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer:[Yer
Justification: We provide details the compute resources used all our experiments in our Appendix in Scction
Guidelines:
The answer means that paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e-g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics https:/ htapa://eeuris.... wwweeerpp..cppaaci///eeeeeeeeeee hccpaallcc/hhiiaauieelieeee
Question:
Answer: [Yes]
Justification: There is no deviation from the NeurIPS Code of Ethics guidelines mentioned. We conform to all the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics. the authors answer No, they should explain special cireumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considcration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper as such connects theory of Transformers and contextualizes them LLMs. Although we do not foresee any immediate negative positive societal impacts of our work. Long term, we hope that our insights that can be used to make better LLMs that when deployed in some critical domains avoid the kind of errors expose in LLMs
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
42

---

11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: We do not foresee any misuse of the new benchmarks we release to expose some limitations in LLMs, and thus do not see the need to provide any safeguards related to the same.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us- should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g.. code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the creators of models and code we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets released, the license, copyright information, and terms of use in the package should he provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
existing datasets that are re-packaged, both original license and license of derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out to asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We contrihute some new small benchmarks related to better understand limitations in L.Mss, and attach the henchmarks, along with how to re-generate them in the supplementary material. We also attach all the code used our experiments the supplementary section.
Guidelines:
The answer NA means that the paper does not release Rescarchers should communicate the details of the dataset/codd/moddl as part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
43

---

Answer:
[NA]
Justification: conduct no such experiments.
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification:
We conduct no such experiments.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it an important, original. or non standaad component of the core methods in this research? Note that if the LLM used only for writing. editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [Yes]
Our paper is about studying LLMs, and hence all of our experiments involved running LLMs understand their behavior. Additionally we used LLMs to assist refining our writing throughout the paper. I.LSs were not used any other part of the paper, including our theoretical results, coming up with our experimental design, creating our datasets and writing code.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: htt///neuripppee wttperippp....ooeeeeeeeeeeeoeeee should or should not be described.
for what
44


Reviews:
[{"review_id": "Zw5lyvvwZ5", "reviewer": "Reviewer_JBEt", "summary": "The paper studies whether large-scale pretraining helps remove certain limits of the transformer architecture, like the length generalization limit. The benchmark task being used for evaluating such capabilities is the \"retrieval and copy\" testbed. The major finding is that pertaining helps boost certain capabilities, such as achieving near perfect performance on copying, but it shows less strong performance with retrieval. It is also hypothesized that pretraining introduces bias like favoring forward/right induction over the reverse. Such asymmetries can be alleviated in finetuning as inductions in both directions can be amplified. The conclusion is that pretraining can't fundamentally remove the essential inductive bias of the transformer architecture.", "questions": "Please refer to the \"strengths and weaknesses\" above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper tackles a fundamental problem. It is crucial to understand the limitations of the transformer architecture e.g. which tasks are more difficult than the rest and which inductive biases were accountable for them. More importantly how can we leverage those learnings to improve the performance of pretraining and downstream tasks.\n- The choice of the copy and retrieval task looks like a simple yet fundamental one and helps test the directional inductive biases\n\nWeaknesses\n- A seq_len limit of 50 tokens is too low for any real world application which often takes at least thousands of tokens. A more practical setup is needed to understand the full picture. \n- Fig 3 is not enough to conclude that the asymmetry was introduce by pretraining. A comparison against models without pretraining is needed for attribution\n- The connection between the \"copy and retrieval\" task with prestraining ppl or downstream tasks performance is unclear. More specifically how does the learning from the model's behavior on this task generalize to other tasks and if any constructive guidances can be distilled from it. The significance of the findings needs more elaboration\n- The paper can benefit from better presentation and story telling. e.g. a contribution summary at the end of the intro section, as well as  the reason to choose copy and retrieval as a representative task would be helpful.", "was_revised": true, "final_justification": "Supplemented experiments on longer seq_len making the results more convincing", "rebuttal": "We thank the reviewer for their suggestions and comments about the paper. As there were no explicit questions from the reviewer, we address the potential weaknesses below:\n\n> A seq_len limit of 50 tokens is too low for any real world application which often takes at least thousands of tokens. A more practical setup is needed to understand the full picture.\n\n- The seq_len limit of 50 tokens applies specifically to Section 4.1. We'd like to clarify that Section 4.1 is not about practical setups, but the point here is to cleanly check theoretical predictions about uniqueness and left/right, forward/backward (a)symmetries. We swept lengths from 10 to 50 tokens across multiple model sizes and families and observed the same asymmetry at every length. Because the pattern was already stable in this range, we chose 50 as a convenient cutoff. We note that a positive result (i.e., no difference across task setups) at short lengths would have motivated further scaling, whereas a negative result (a clear pattern towards asymmetry in one direction over the other) on such basic tasks is itself striking.  Based on our observations in Section 4.1, we were then motivated to examine longer lengths, which led to the more practical experiments described in Section 4.2. By extending the testing lengths and making the tasks more realistic—such as copying Lorem Ipsum–style paragraphs and performing git revert and cherry-pick on a list of commits—we were able to replicate the findings from Section 4.1. These larger-scale experiments again demonstrated the same asymmetries between left/right or forward/backward directions (as in the git commits case), as well as the relative ease of unique or unambiguous tasks compared to non-unique or ambiguous ones (as in the copying of lorem ipsum paragraphs).   \n\n- To further strengthen our point, we have conducted some additional experiments with the Lorem-Ipsum setup at even longer scales (up to 2000 tokens) and report the results in the table below for 2 of our models. Once again, **we do see the same effects we see at lengths of 500, i.e., a lower success rate for ambiguous bigrams than for unambiguous ones.** Running these models for longer and longer lengths is computationally expensive, and as we consistently see these effects at lengths up to 2000, we have concluded our experiments here. If copying had been perfect at such lengths, it would have warranted a deeper analysis at longer lengths. However, since we already observe failures in copying across models and settings at ~500 tokens, we consider it a noteworthy finding. We find this especially interesting as previous work has shown that models perform perfect accuracy while copying ordinary text (with semantic cues) [1] at thousands of tokens length, whereas our Lorem-Ipsum setup demonstrates failures on copying text without semantic cues at much more moderately sized lengths. \n    | Model         | Unambiguous | Ambiguous  |\n    |---------------|-------------|------------|\n    | llama3-70B-Instruct    | 0.99  | 0.96\n    | llama3-8B-Instruct      | 0.96  | 0.89\n    | qwen2.5-32B-Instruct | 1.0 | 0.95\n    | qwen2.5-7B-Instruct | 0.99 | 0.75\n\n\n- Overall, we appreciate the reviewer's comment and will report these results at higher lengths more explicitly in the paper.\n\n> Fig 3 is not enough to conclude that the asymmetry was introduce by pretraining. A comparison against models without pretraining is needed for attribution\n\n\n2. We would like to clarify our argument that the asymmetry is introduced by pretraining: We agree that the left-right asymmetry is shown most clearly in comparison against models without pretraining, and we would like to point out the from-scratch results in Appendix B. These show that no left-right asymmetry is present when training from scratch. Further important support for the asymmetry as a pretraining artifact is given in Section 4.4, where we give a mechanistic explanation to both fine-tuned and pretrained behavior, and attribute the asymmetry to an imbalance between induction and anti-induction circuits in pretrained models. Taken together, we agree with the Reviewer that the point is not made by Figure 3 -- rather, it follows from comparison to from-scratch models and from mechanistic interventions. We will revise to ensure this argument is made transparent.\n\n> The connection between the \"copy and retrieval\" task with prestraining ppl or downstream tasks performance is unclear. More specifically how does the learning from the model's behavior on this task generalize to other tasks and if any constructive guidances can be distilled from it. The significance of the findings needs more elaboration\n\n3. We believe that copy and retrieval tasks are indeed fundamental to reasoning, as solving subproblems in the reasoning chain often involves some sort of retrieval and copying of the input [2]. However, we agree that this can be made more explicit, and we will add a more detailed discussion in the Appendix.\n\n> The paper can benefit from better presentation and story telling. e.g. a contribution summary at the end of the intro section, as well as the reason to choose copy and retrieval as a representative task would be helpful.\n\n4. Thank you for your suggestions, we will incorporate them into the paper.\n\n\n[1] [Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-Context Models](https://aclanthology.org/2024.emnlp-main.269/) (Liu et al., EMNLP 2024)\n[2] [Query-Focused Retrieval Heads Improve\nLong-Context Reasoning and Re-ranking](https://openreview.net/forum?id=KG9IxJ9ZAL) (Zhang et al., 2025.)"}, {"review_id": "aqZmzaArHb", "reviewer": "Reviewer_KxkG", "summary": "In the present paper, the authors investigate whether Transformer architectures might display unexpected length-generalization limitations compared to those predicted theoretically (e.g., C-RASP[pos]), after large-scale pretraining. The authors focus on copying and retrieval tasks, comparing model performance on “rightward” (induction) versus “leftward” (anti-induction) variants, and highlighting a robust right-hand bias in pretrained LLMs (LLaMA-3, Qwen2.5), even though theory predicts symmetric behavior. This asymmetry is linked to induction vs. anti-induction heads, and it disappears with targeted fine-tuning. However, some additional experiments show how “glitches” arise in non-unique copying due to ambiguous transitions, and these cannot be easily removed in a fine-tuning stage. The work concludes that pretraining amplifies certain capabilities but does not eliminate architecture-induced biases.", "questions": "1 - Isn’t a rightward bias natural given the autoregressive training? Why is this presented as an architectural failure rather than an expected outcome?\n\n2 - Is there a connection between the glitch phenomenon and the induction/anti-induction head asymmetry? Do the authors believe that predicting multiple tokens at the same time (e.g. with diffusion-based methods) could remove this bias?\n\n3 - Does the existence of a C-RASP[pos] program prove that a task will generalize under all training regimes and model types? \n\n4 - Why was GPT-2 used for fine-tuning instead of LLaMA/Qwen? Do the authors expect the results to hold for modern LLMs? Can the authors provide a baseline for comparison showing the bias of GPT2 before fine-tuning?", "limitations": "Discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The asymmetry in performance is consistent and well-demonstrated across tasks and models.\n- The paper convincingly links asymmetry to the prevalence of induction vs. anti-induction heads, across models, instruction vs. non-instruction tuning, and multiple prompt templates.\n- The authors also provide a formal lens to understand the phenomenology, using the C-RASP[pos] framework. \n- The authors convincingly discuss that these subtle biases can manifest in practical contexts (Git workflows) or copying long neutral text.\n\n**Weaknesses**\n- The paper does not fully address whether this “bias” is surprising, especially given that Transformers are trained autoregressively (left-to-right). \n- While reading the paper, it remains ambiguous whether the glitches in non-unique copying are related to the directional asymmetry or are a separate phenomenon entirely. In this case, it is not completely clear what this behavior adds to the \"unexpected bias from pretraining\" picture. \n- To a non-expert reader, it is unclear if the presence of a C-RASP[pos] program implies a necessary and sufficient condition for generalization, or just an empirical trend.\n- The explanation/analysis of the induction head mechanism appears to be slightly rushed if one does not assume prior familiarity with mechanistic interpretability.\n-  The fine-tuning experiments (showing that the bias can be removed) are conducted on GPT-2 model rather than the same models (LLaMA, Qwen) used earlier, making comparisons less direct.", "was_revised": true, "final_justification": "I believe this work shows an interesting pre-training artifact in LLMs, that should be accounted for in specific applications. Overall, the presence in the main of two parts that are not well-connected, and the lack of rigor in the extension of the proposed theory to the settings where the experiments are run limit somewhat the readability and impact of this work. For these reasons, I will maintain my initial score.", "rebuttal": "We thank the reviewer for their feedback and suggestions on clarifications of explanations. We would like to jointly respond to question and weaknesses below.\n\n> from Weaknesses: The paper does not fully address whether this “bias” is surprising, especially given that Transformers are trained autoregressively (left-to-right).\n    \n> from Questions: Isn’t a rightward bias natural given the autoregressive training? Why is this presented as an architectural failure rather than an expected outcome?\n    \n1. We would like to clarify that we consider the rightward bias to be *an artifact of pretraining*, and *not an architectual failure*. Indeed, we agree with the reviewer this it is a natural artifact of autoregressive training on language. We consider it to be especially interesting as the theory predicts equal difficulty of both directions for the architecture for precisely autoregressive generation of output by Transformers. So, despite the bias being intuitive, we consider finding a more precise reason to explain it to be worthwhile. We believe that explaining this bias mechanistically using induction and anti-induction circuits gives an important empirical insight relevant for downstream applications.\n\n    \n> from Weaknesses: While reading the paper, it remains ambiguous whether the glitches in non-unique copying are related to the directional asymmetry or are a separate phenomenon entirely. In this case, it is not completely clear what this behavior adds to the \"unexpected bias from pretraining\" picture. \n\n> from Questions: Is there a connection between the glitch phenomenon and the induction/anti-induction head asymmetry? Do the authors believe that predicting multiple tokens at the same time (e.g. with diffusion-based methods) could remove this bias?\n\n2. The copying glitches and the left-right asymmetry are separate phenomena. The copying glitches are tightly related to the inherent architectural difficulty of non-unique copying and retrieval. In contrast, the left-right asymmetry is a bias arising from pre-training. Theory describes the existence of the copying glitches as inherent to Transformer architecture, while asymmetry is removable by fine-tuning. We believe that fine-tuning the model to remove the above mentioned asymmetry is already a good way to address the issue in more practical settings. Non-autoregressive methods lie beyond the scope of our paper, which focuses on Transformers. Confirming the existence or absence of both the glitches and the left-right asymmetry in such architectures is an interesting future work direction that might be a promising solution on par with fine-tuning.\n    \n> from Weaknesses: To a non-expert reader, it is unclear if the presence of a C-RASP[pos] program implies a necessary and sufficient condition for generalization, or just an empirical trend.\n    \n> from Questions: Does the existence of a C-RASP[pos] program prove that a task will generalize under all training regimes and model types?\n\n3. Thanks for this question. We would like to point out that, in Section 3, we sketch the theoretical length generalization conditions, including required positional encodings and training procedures. To provide more detail, we will expand this discussion in Section 3 along the following lines: *the theoretical length generalization guarantee for C-RASP[pos] by Huang et al. 2024 applies to APE Transformers under an idealized training procedure, where all training data up to some maximum input length N are available and the exact minimizer of a regularized loss is found. Under these specific conditions, the existence of a C-RASP[pos] program for that task guarantees length generalization to larger lengths. While theory has only been worked out for this specific setup, empirically we observe converging results across different PE types and realistic SGD-based training (as shown in Huang et al. 2024 as well as in the experiments in Appendix B).* In the future, we would be excited to see the extensions of the existing theory to more PE types and more practical regimes. \n\n> from Weaknesses: The fine-tuning experiments (showing that the bias can be removed) are conducted on GPT-2 model rather than the same models (LLaMA, Qwen) used earlier, making comparisons less direct.\n\n> from Questions: Why was GPT-2 used for fine-tuning instead of LLaMA/Qwen? Do the authors expect the results to hold for modern LLMs? Can the authors provide a baseline for comparison showing the bias of GPT2 before fine-tuning?\n\n4. Regarding the baseline of GPT-2 before finetuning: In experiments, we found that the in-context performance of GPT-2 was at chance level (i.e. 0), hence, we did not include those results in the paper. We will consider making these results explicit in Figure 6. Regarding the choice of GPT-2: Due to computational budget constraints as well as their usage of RoPE, we did not fine-tune larger, 7-8B and 70B models used in the Section 4.1 at the time of the submission. \n    \n> from Weaknesses (no corresponding Question): The explanation/analysis of the induction head mechanism appears to be slightly rushed if one does not assume prior familiarity with mechanistic interpretability.\n\n5. Thank you for your feedback. We will strive to improve this."}, {"review_id": "11axWDqqw0", "reviewer": "Reviewer_gWjY", "summary": "The paper investigates whether fundamental length-generalization limits of small transformers carry over to large pretrained LLMs. Using a family of synthetic retrieval and copying tasks formalized in the C-RASP[pos] framework, the authors derive theoretical expressivity guarantees. They then measure in-context performance across multiple LLMs and uncover a consistent right-forward versus left-backward asymmetry, trace this mechanistically down to induction versus anti-induction head prevalence, and show that supervised fine-tuning rebalances these circuits to restore theory-aligned generalization.", "questions": "- For the git cherry picking task, some performance drop might be due to the LLM's unfamiliarity with this command. Would be good to have a more straight forward task.\n- Generality of this paper would benefit from evaluating on SOTA closed-source systems.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength:** The paper combines solid theory with extensive empirical validation on both synthetic and naturalistic tasks, connecting C-RASP[pos] expressivity to observed LLM behavior and mechanistically attributing asymmetry to circuit strength.\n\n**Weakness:** Real-world evaluations remain limited to specific tasks at moderate context lengths. The paper can also benefit from some more discussions on the initial asymmetry.", "was_revised": true, "final_justification": "The authors conducted a series of theory-guided experiments for understanding retrieval and copying tasks. All of my questions have been addressed.", "rebuttal": "We are thankful to the reviewer for their comments and aim to address the concerns regarding testing lengths, asymmetry discussion, and evaluation choices below. \n\n\n#### Response to Weakness:\n\n> Real-world evaluations remain limited to specific tasks at moderate context lengths. \n1. While we acknowledge that real-world applications involve processing of larger text sequences, we believe it to be important and interesting that the mentioned effects can already be observed at smaller lengths. Additionally, Section 4.2 does extend our experiments to larger context lengths, showcasing a more naturalistic test bed.\n\n\n#### Response to Questions:\n\n> The paper can also benefit from some more discussions on the initial asymmetry.\n\n2. We are unsure what the reviewer means by *initial asymmetry*. If this refers to the from-scratch behavior in comparison to the behavior of pretrained LLMs, we report those results in Appendix B. Otherwise, we would be grateful if the reviewer could clarify this in the discussion.\n\n\n> For the git cherry picking task, some performance drop might be due to the LLM's unfamiliarity with this command. Would be good to have a more straight forward task.\n\n3. While we acknowledge that the model's familiarity with the *git cherry pick* command might affect the performance, we provided an explanation of the task which uses the same terms as the terms mentioned for the *git revert* task, and also provided three few-shot examples in the prompt to further explain what the task is, which taken in conjuction should alleviate the effect of token unfamiliarity, as we observed in our experiments in section 4.1 as well. Despite this, we still find a gap in performance between these tasks, matching the theoretical predictions.\n\n> Generality of this paper would benefit from evaluating on SOTA closed-source systems.\n\n4. The focus of our paper is to understand the role of architectural limitations and biases from the pretraining data in shaping LLMs' abilities. These details are not known for closed-source systems -- thus, evaluation of the same setup in closed-source setting, even if results are similar, cannot be directly attributed to the model's architecture or pre-training regimen. We thus believe that open systems are most relevant to testing our research questions."}, {"review_id": "RKVZqeGiuC", "reviewer": "Reviewer_akQM", "summary": "The goal of this paper is to quantify whether and how the length generalization limitations predicted by theory persist in large pretrained transformers, particularly in tasks involving retrieval and copying. The authors propose variants of forward/backward copying and left/right retrieval tasks, and assess the models' length generalization capabilities as predicted by C-RASP. They also empirically validate the accuracy of C-RASP predictions by finetuning small models on each task. Subsequently, they evaluate pretrained models in-context on these tasks and reveal a stark contradiction between empirical performance and theoretical predictions. Additionally, they observe that pretrained models perform significantly better on the forward versions of the tasks compared to their backward counterparts. This discrepancy is later attributed to the relative strength of induction circuits over anti-induction circuits, likely shaped by patterns in the pretraining data.", "questions": "1) In addition to my earlier point in the weaknesses section: if the goal of Section 4.1 is to cross-check the C-RASP length generalization predictions on pretrained language models, it’s important to note that the C-RASP framework assumes learned absolute positional embeddings. However, the models evaluated in this section use RoPE. This discrepancy is not clearly acknowledged or discussed in the paper, and it likely introduces a confounding factor into the evaluation.\n\n2) The finding in Section 4.2 and Figure 4 closely resembles the results of Jelassi et al. [2024], who demonstrated that transformers rely on  bigram lookup mechanisms for copying. It would be helpful to more explicitly acknowledge (or possibly differentiate) this.\n\n3) The discussion on “token-level alignments” (lines 217–222) and the “Git Commit History” experiment were difficult to follow and could benefit from clearer explanations.\n\n4) Is there a particular reason the performance of the GPT-2 model prior to fine-tuning isn't reported? Including this baseline would help quantify the relative gains achieved through fine-tuning on each task.\n\n5) The task names (e.g., UL vs. UB) are somewhat confusing when reading the text. As a suggestion, perhaps adding a prefix, like Copy-U-L or Retrieve-U-B could improve readability and help readers quickly identify tasks.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper is well-organized, well-written, and easy to follow. The scope is clearly defined, the tasks are thoroughly explained with helpful schematics, and the use of paragraph titles and takeaway boxes summarizes the main points of each section.\n\nThe experiments are clearly presented and rigorously documented. I also appreciated the variety of in-context prompting variants discussed in the appendix. The main results, particularly those in Figures 3 and 4, are highly informative with clear practical implications.\n\n**Weaknesses:**\n\nMy main concern is that one of the key motivations of the paper is to examine whether theoretical limitations on _length generalization_ actually impact the performance of large pretrained models in practice. However, I believe this question is most meaningful when tested on relatively long sequences, whereas the experiments in Section 4.1 focus on rather short ones.\n\nModel performance mainly depends on whether the sequence lengths used during testing were also present during training. As a result, performance (especially on shorter sequences that are more common in training data) may reflect the influence of the training distribution more than any fundamental architectural limitations.\n\nThe observation in Section 4.2, where models fail to copy because of ambiguous tokens, aligns with the theory prediction and is performed on longer sequences. This also supports the idea that the sequences used in Section 4.1 may have been too short to properly test the predictions of the length generalization theory.\n\nTherefore, the observed gap between theoretical predictions and empirical performance might be more attributable to the nature of the training data than to the scale of the models or the web-scale data they were trained on, as suggested in the introduction (lines 37–41). I believe this point should be clarified more.", "was_revised": true, "final_justification": "The paper is clear and well-presented, with informative results and thorough experiments. Its main motivation is to assess how transformers' theoretical limitations in length generalization manifest in large-scale pretrained models. However, this question may not be fully addressed, as the tested sequences are relatively short. Instead, the authors observe a left–right asymmetry in models' recall capabilities that is likely influenced by the training data. While this finding is interesting, another minor concern is the significance of the contribution for NeurIPS.", "rebuttal": "We thank the reviewer for their detailed feedback regarding clarifications on theoretically predicted generalization behavior, experimental setups, and writing improvements. We intend to address the above mentioned weaknesses and questions in separate sections below.\n\n\n#### Response to Weaknesses\n> My main concern is that one of the key motivations of the paper is to examine whether theoretical limitations on length generalization actually impact the performance of large pretrained models in practice. However, I believe this question is most meaningful when tested on relatively long sequences, whereas the experiments in Section 4.1 focus on rather short ones. \n1. Indeed, we acknowledge that the sequence limit of 50 tokens in Section 4.1 does not exhaust the potential real world sequence lengths. The important point in Section 4.1 is that the key failures already appear at shorter lengths, which are potentially more frequent in the training data and particularly practically relevant. As the reviewer mentioned, we also test longer sequences up to 500 tokens on the Lorem-Ipsum-style copying task in the Section 4.2. \n    \n\n> Model performance mainly depends on whether the sequence lengths used during testing were also present during training. As a result, performance (especially on shorter sequences that are more common in training data) may reflect the influence of the training distribution more than any fundamental architectural limitations.\n\n> The observation in Section 4.2, where models fail to copy because of ambiguous tokens, aligns with the theory prediction and is performed on longer sequences. This also supports the idea that the sequences used in Section 4.1 may have been too short to properly test the predictions of the length generalization theory.\n\n> Therefore, the observed gap between theoretical predictions and empirical performance might be more attributable to the nature of the training data than to the scale of the models or the web-scale data they were trained on, as suggested in the introduction (lines 37–41). I believe this point should be clarified more.\n\n2. While we agree with the reviewer that the influence of the training distribution does impact performance in many ways, that is precisely why we found our results exciting: our task setups allowed us to disentangle the effects of pretraining from architectural limitations. Through our experiments in Section 4.1, we were already able to observe the effects of pretraining — certain abilities were selectively amplified more than others, despite there being little to no theoretical difference between them.\nSince we were able to shortlist which abilities the training distribution amplifies the most, we moved on to the experiments in Section 4.2. We are glad the reviewer agrees that the failure modes observed in Section 4.2 indeed align with theoretical predictions.\nIt was only in Section 4.3, through our fine-tuning experiments and mechanistic insights, that we were able to conclude that the directional asymmetry (e.g. left/right or forward/backward) was due to pretraining, while certain other failures (e.g. in retrieval models, or repeated copying) could be attributed to architectural limitations. In these experiments, we tested our length generalization setup by training models on shorter sequences and evaluating them on longer ones. We were able to eliminate failures caused by pretraining, but the effects of architectural limitations remained.\nThis helped us finally disentangle the effects of pretraining from those of architectural limitations. As suggested by the reviewer, we will make this point clearer in the main text of the paper.\n\n\n#### Response to Questions\n> In addition to my earlier point in the weaknesses section: if the goal of Section 4.1 is to cross-check the C-RASP length generalization predictions on pretrained language models, it’s important to note that the C-RASP framework assumes learned absolute positional embeddings. However, the models evaluated in this section use RoPE. This discrepancy is not clearly acknowledged or discussed in the paper, and it likely introduces a confounding factor into the evaluation.\n\n1. Indeed, the C-RASP[pos] framework assumes a Transformer architecture using APE. In the footnote of the Section 3 we note that while Huang et al. results rely on this assumption, we find that empirically these results extend to the RoPE setting, which are more extensively discussed in the Appendix B, providing the from-scratch results for both APE and RoPE Transformers. However, we agree to make this note more explicit in the body of the paper.\n\n> The finding in Section 4.2 and Figure 4 closely resembles the results of Jelassi et al. [2024], who demonstrated that transformers rely on bigram lookup mechanisms for copying. It would be helpful to more explicitly acknowledge (or possibly differentiate) this.\n\n2. Although we do cite the results of Jelassi et al. [2024], we agree that we could make this discussion more explicit, and we promise to do so.\n\n> The discussion on “token-level alignments” (lines 217–222) and the “Git Commit History” experiment were difficult to follow and could benefit from clearer explanations.\n\n3. We acknowledge that the token-level alignment and the Git Commit task can be described more clearly. We had hoped that the additional details and example traces provided in Appendix C.3 would help clarify our method. Nonetheless, we will make a concerted effort to further improve the explanations in the main body of the paper.\n\n> Is there a particular reason the performance of the GPT-2 model prior to fine-tuning isn't reported? Including this baseline would help quantify the relative gains achieved through fine-tuning on each task.\n\n4. We found that the baseline performance across tasks was at a chance level (i.e. the baseline was 0), so, we have not included those in-context results. We will consider making these results explicit in Figure 6. \n\n> The task names (e.g., UL vs. UB) are somewhat confusing when reading the text. As a suggestion, perhaps adding a prefix, like Copy-U-L or Retrieve-U-B could improve readability and help readers quickly identify tasks.\n\n5. Thank you for your suggestions. We will carefully consider ways to improve readability of the paper, including potential task names clarifications."}]

Author rebuttal:

Meta-review: The paper evaluates whether large-scale pretraining overcomes Transformers’ length generalization limits (using C-RASP) when evaluated on synthetic retrieval and copying tasks. It finds a consistent bias favoring forward induction over backward anti-induction, linked to circuit asymmetries shaped by pretraining. Fine-tuning restores symmetry, but core architectural constraints remain. The reviewers have expressed various concerns over the experiments, but also over the significance of the contribution. Given the general title, I do agree with the reviewers that I expected more and agree with the reviewers that there seems to be a wrong scope. Therefore, I will recommend acceptance since this is an interesting observation and analysis on inductive bias, but I **strongly recommend** the authors to revise the title and make it more descriptive of their work in the camera-ready version.
Decision: Accept (poster)