Extrapolation by Association: Length Generalization Transfer In Transformers
Authors: Ziyang Cai
Nayoung Lee
Avi Schwarzschild
Samet Oymak
Dimitris Papailiopoulos
Venue: NeurIPS 2025 spotlight ()
Keywords: Language models
LLM
Transformers
Length Generalization
Transfer
Primary area: deep_learning
TL;DR: We show that transformers achieve length generalization when training on shorter main task and longer auxiliary tasks together.

Abstract: Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalization arises. In this paper, we investigate length generalization—the ability to extrapolate from shorter to longer inputs—through the lens of \textit{task transfer}. We find that length generalization can be \textit{transferred} across related tasks. That is, training a model with a longer and related auxiliary task can lead the model to generalize to unseen and longer inputs from some other target task. We demonstrate this length generalization transfer across a diverse suite of algorithmic tasks, including arithmetic operations, string transformations, and maze navigation. Our results show that transformer models can inherit generalization capabilities from similar tasks when trained jointly. Moreover, we observe similar transfer effects in pretrained language models, suggesting that pretraining equips models with reusable computational scaffolding that facilitates extrapolation in downstream settings. 
Finally, we provide initial mechanistic evidence that length generalization transfer correlates with the re-use of the same attention heads between the tasks. Together, our findings deepen our understanding of how transformers generalize to out-of-distribution inputs and highlight the compositional reuse of inductive structure across tasks.

Extrapolation by Association: Length Generalization Transfer in Transformers
Ziyang Cai*
Nayoung Lee
University of Aacconin-MMadisee
University of Wisconsin-Madisst
Avi Schwarzschild Carnegie Mellon University
Samet Oymak University of Michigan
Dimitris Papailiopoulos
University of Seeconsin-Madisee Microsoft Research
Abstract
Transformer language models have demonstrated impressive generalization capa- bilities in natural language domains, yet we lack fine-grained understanding of how such generalization arises. this paper, we investigate length generaliza- tion-the ability to extrapolate from shorter to longer inputs throuuh the lens task association, We find that length generalization can be transferred across related tasks. That is, training model with a longer and related auxiliary task can lead it to generalize to unseen and longer inputs from some other target task. We demonstrate this length generalization transfer across diverse algorithmic tasks, including arithmetic operations, string transformations, and maze navigation. Our results show that transformer models can inherit generalization capabilities from similar tasks when trained jointly. Moreover, we observe similar transfer effects in pretrained language models, suggesting that pretraining equips models with downstream
reusable computational scaffolding that facilitates extrapolation settings. Finally, we provide initial mechanistic evidence that length generalization transfer correlates with the re-use of the same attention heads between the tasks. Together, our findings deepen our understanding of how transformers generalize to out-of-de of-distribuutin inputs and highlight the compositional reuse of inductive structure across tasks.
Introduction
A central theme of transformer language models is their ability to generalize. By scaling data and model size, large language models develop emergent abilities that exceed expectations [Wei et al., 2022]. They can also transfer knowledge across domains and tasks [OpenAI, 2024, Brown et al., 2020, Sanh al., 2022]. While is widely believed that language models are not simply parroting or memorizing their training data, we still lack a fine- tne-grained understanding of how language models apply skills learned during training to potentially unseen problems.
The distribbttiin (OOD) generalization capabilities of language models have garnered much attention in the literature [Anil al., 2022, Zhang al., 2024, Yang al., 2024]. In this work, we study canonical example of OOD generalization, length generalization, which is the ability to generalize from shorter to longer inputs [Zhou al., 2023]. There long line of work focusing on improving length generalization of arithmetic tasks in transformers, which has spurred innovations in positional encoding schemes and transformer architecture [Cho al., 2024, McLeish al., 2024]. Closely related is the concept of compositional generalization, where the model combines previously learned skills solve new problems [Yang et al., 2024, Xu al., 2024].
*Corresponding author. zcai75 @ wisc.edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we study new mechanism underlying length generalization: extrapolation by associ- ation. We hypothesize that, when faced with a problem outside its training distribution, language models can use related skills to solve it. Specifically, we ask: Can generalization to longer inputs in one task transfer to another task that is only trained on short examples?
To showcase the length generaliza- tion transfer capabilities in transform- ers, we choose three distinct groups of synthetic tasks. The tasks in each group are related such that they rep- resent similar algorithmic procedures. Within each group, we train multiple tasks together, and crucially, we train an "auxiliary task" at longer length and "main task" at shorter length. Using this setup, we observe that the shorter main task generalizes to the length of the longer auxiliary task when trained together. See Figure for the tasks and respective lengths used each experiment.
General ization Transfer
Trained Separately:
Trained Together:
Aux
Main
Aux
Accuracy
Main
Langth
Len
Figure 1: Trained separately, each task fails to generalize to longer inputs. When trained jointly, the main task inherits the generalization range of the auxiliary task.
Contributions
1. We present the phenomenon of length generalization transfer, in which transformer models trained on related tasks exhibit extrapolation behavior not present when trained on the target task alone, providing new insights on the effect of multitask training on length generalization.
2. We show that the same phenomenon replicates in pretrained language models, and that natural language pretraining transfers length generalization capabilities to synthetic downstream tasks.
3. We provide mechanistic evidence that transfer correlates with shared internal computa- tion specificalll the reuse of attention heads across tasks.
2 Related Works
Length generalization concerns extrapolating to longer sequence lengths than those seen during training [Dubois al., 2019, Hupkes et al., 2020, Newman et al., 2020, Anil et al., 2022]. Previous approaches include architectural modifications such specialized positional embeddings [Press et al., 2021, Li al., 2023, Ruoss et al., 2023, Kazemnejad et al., 2024, Sabbaghi et al., 2024, Cho et al., 2024, Zhou et al., 2024, McLeish et al., 2024], looping [Fan et al., 2024], novel attention mechanisms [Duan et al., 2023, Li al., 2025], and input format augmentation [Zhou et al., 2023, 2024]. Beyond arithmetic, Yehudai et al. [2021] studies length generalization in graph tasks. In contrast, our work examines novel mechanism from which length generalization emerges: transfer from related tasks. Finally, closely related to our work, "task hinting" [Awasthi and Gupta, 2023] trains sorting and increment-by in length generalization performance.
Length Generalization.
tasks with simpler auxiliary tasks, showing improvements
Compositional Capabilities. To explain emergent capabilities language models, many works study compositional generalization to understand whether transformers can gain abilities beyond those in the training set. Yu et al. [2023], Zhao et al. [2025] and Hosseini et al. [2024] design benchmarks testing the ability to combine Iearned skills to solve compositional math problems. Ahuja and Mansouri [2024] derive provable guarantees for length and compositional generalization conditioned on training set diversity. Some works use synthetic tasks to probe compositional generalization.
Ramesh et al. show transformers achieve compositional generalization on unseen combinations using series of bijections and permutations applied to strings, while Abedsoltan et al. [2025] show similar results on families of parity functions.
For the specific task of reverse addition, works like Quirke and Barez [2023] and Quirke et al. [2025] identify computational circuits responsible for compositional subtasks and show transferability of such circuits to the related task of subtraction.

---

Experimental Settings
Models. For from-scratch experiments,
transformer models with 6 heads and 6 layers, following the Llama architecture [AI@ Meta, 2024], which uses Rotary Positional Embeddings (RoPE) [Su et al., 2023] for position encoding. For experiments with pretrained models, SmolLM [Allal et al., 2024], which provides access to intermediate checkpoints during pretraining, allowing us investigate how length generalization transfer evolves over time.
Tasks. We evaluate length generalization transfer across three categories of algorithmic problems:
arithmetic, string manipulation, and maze solving. Our tasks include:
Arithmetic Tasks
reverse add - Compute the sum of two integers, presented in reversed order.
carry Compute digit-wise sums mod 10, without carry propagation.
carry only Output a binary mask indicating carry positions during addition.
reverse subtract - Compute the reversed digit-wise difference between two numbers.
n 3 3 COT multiply - -Multiply an n-digit number by 3, with chain-of-thought steps.
String Manipulation Tasks
string copy Return the input string unchanged.
MOAR Multi-Query Associative Recall) [Arora et al., 2023] Given a repeated query substring, retrieve the next character following each occurrence.
capitalize Flip the case of all alphabetic characters (lower upper).
reverse Reverse the character order of the input string.
capitalize-reverae Apply both reversal and case-flipping to the input string.
Maze Tasks
trace Simulate depth-first search from start node goal node in maze.
shortest path
Return the optimal (shortest) path between a start and goal node.
Task Groups. We construct task groups by pairing a main task, trained on short sequence lengths, with one more auxiliary tasks, trained on longer sequences. The main goal is to evaluate whether training on related auxiliary task improves the main task's ability to generalize longer inputs, despite never seeing such lengths during training. The list of task groups are:
Main Task (Train Length)
Auxiliary Task(s) (Train Length)
reverse add (16)
no carry & caaryy only (32)
reverse add (16)
reverse subtract (32)
reverse add (8)
n n 3 CoT multiply (16)
string copy (16)
MOAR (32)
capitalize-reverse (16)
capitalize (32), reverse (32)
DFS trace (32)
shortest path (64)
Data sampling and Task Length. Since we train under a multi-task setting, at each iteration, a task sampled uniformly random from a predefined task group. For the selected task, an individual training example is constructed based on a single governing parameter: length, which determines the size complexity of the problem instance. The length of each example is sampled uniformly from specified range for that task. All training data is generated on-the-fly during training.
Since the notion of length varies across task types, define length for each task as:
Addition Tasks: the maximum number of digits in both operands.
String Tasks: the number of characters in the input string.
Maze Tasks: the number of nodes in the input maze graph. See Section 4.3 for further details.

---

Arithnetic
String Tasks
Carry Only:
2090455-1829948-00000011
Reverse String:
No Carny:
4090466441829558--8119999
Capitalize String:
Auxiltary Task
2050455-78225588-8878.53
Capital Reverse:
PayIIcvTTkk
Pain Task
nx3 COT Multiply:
60844671
030422830+000000000
[03042288]]] 000-22228|-0022698855000
[0325817163]
Hulti-query KY04FL8T-04FLY
Auxiltary Task
Auxiltary
Reverse Add:
2050465
Copy string:
ONFetcebxY-ONFatteee
Mainn Task
Naze Tasks
adjacency
Shortest
13) [30],
Path:
[0]1P]]155) (12) [61]]29] [28]-1993
In adjacency List
Out: ALl paths Ln DFS
[281
DFS Trace:
(15): (8[ts]ss)))) (62) [[1]]29]. [28]=[55]]
52 [22] [6]][aa][[0[[[(]]1118]
58 (3)
[35]
Task
Overview of the tasks used in our length generalization transfer experiments, spanning three domains: arithmetic, string manipulation, and maze solving. Each group consists of a main task trained on shorter sequences and one more auxiliary tasks trained on longer ones. We study whether generalization to longer inputs can be transferred from the auxiliary to the main task.
Figure 2:
Training and Evaluation. Each example consists of an input-output pair. We use loss mask to train only on output tokens (and for MQAR, only on answer characters). At test time, we evaluate using exact match accuracy on fixed test set of 1024 examples. For each configuration, we report results across random initialization seeds but the dataset is kept the same. Full experimental configurations and hyperparameter details are provided in Appendix C.
Length Generalization Transfer in Algorithmic Tasks
In this section, demonstrate that while length generalization is often difficult for algorithmic tasks, it can emerge through transfer when the model is co-trained on longer auxiliary tasks. Figure illustrates the three categories of tasks arithmetic operations, string transformations, and maze navigation.
4.1 Arithmetic Tasks
Reverse addition has become popular synthetic task for studying length generalization [Lee et al., 2023. Shen et al., 2023, Zhou et al., 2023, 2024, Cho et al., 2024, McLeish et al., 2024, Lee et al., 2025] in Transformers. The task involves calculating the sum of two randomly sampled integers, and length generalization in this task involves training on examples up to some fixed length, and generalizing on test data beyond the training lengths. Here, we adopt the reverse add format proposed by Lee et al. [2023], where the operands and the sum reversed for faster learning. For the auxiliary tasks, consider (1) reverse subtract, which computes the difference between

---

two operands, (2) no carry, which computes the digit-wise sum mod 10, ignoring the carries, and carry only, which computes the locations where a earry happens in the addition.
1.00
1.00
Accuracy
0.75
Accuracy
0.75
Reverse Add (main)
0.50
Reverse Add (main)
Reverse Sub (aux)
0.25
No Carry (aux)
0.25
Only Carry (aux)
0.00
0.00
20 Length
10
40
Length
(a) Aux:
carry
carry only
(b) Aux:
subtract
1.00
1.00
Reverse Add (main)
Accuracy
0.75
Accuracy
0.75
0.50
0.50
0.25
Reverse (main)
0.25
x COT Mult (aux)
0.00
0.00
15
30
40
Length
Length
(c) Aux: COT multiply
(d) reverse add (No Aux Tasks)
Figure 3: Length generalization results for addition-related task groups. The main task is reverse add, with performance shown when trained with different auxiliary tasks. Each model is trained with 5 random seeds; best-performing runs are shown in bold. The dashed vertical line indicates the maximum training length for each task. When trained alone (d), the model fails to generalize beyond training length. Co-training with related auxiliary tasks (a-c) enables extrapolation to longer inputs.
As shown Figure 3, models trained only on reverse add (Figure 3d) struggle to generalize beyond the training length. However, when co-trained with longer auxiliary tasks (Figures 3a, 3b, 3c), the model successfully extrapolates, often matching the auxiliary task's generalization range. This provides empirical evidence that length generalization
transfer across tasks.
is worth noting that the generalization behavior is not entirely robust: different random seeds yield noticeably different outcomes, suggesting unstable training dynamics. We discuss this instability further in Section 6.2.
4.2
String Tasks
We now turn to string operations, where we observe similar transfer effects on two task groups. The tasks include: string copy. which returns the input unchanged; MOAR (Multi-Quen) As- sociative Recall) [Arora et al., 2023], where the model retrieves the next character given random substring: reverse, which reverses character order; capitalize, which inverts letter case; and capi tawlitareeeeessee combining case inversion and reversal.
Figure shows that when trained on main tasks alone (Figures 4b, 4d), the model does not generalize beyond the training range. On the other Aadiig training with auxiliary tasks enables substantial extrapolation (as shown in Figures 4a and 4c).
4.3
Maze Tasks
Lastly, we examine maze-solving tasks as testbed for length generalization transfer. We define maze as spanning tree over square grid, generated using Wilson's 's algorithm [Wilson, 1996], which
5

---

1.00
1.00
Copy String (main)
Accuracy
0.75
Accuracy
0.75
0.50
0.50
0.25
Copy String (main)
0.25
MOAR (aux)
0.00
0.00
10
20
10
20
30
40
50
Length
Length
(a) Main: string ccpy Aux:
MQAR
(b) string copy (No Aux Tasks)
1.00
1.00
Capitalize-Reverse (main)
Accuracy
0.75
Accuracy
0.75
Capitalize Capitalize-Reverse Reverse (main)
Capitalize (aux)
0.50
Reverse (aux)
0.25
0.25
0.00
0.00
Length
Length
(c) Aux:
capitalize & zeverse
(d)
capital (No Aux Tasks)
Figure 4: Performance plots for string tasks. When trained alone (b, d), models fail to generalize beyond their training range. Co-training with auxiliary tasks (a, c) enables substantial length extrapolation.
ensures uniform sampling via loop-erased random walks. For each problem instance, randomly sample a start and end node, and the model tasked with producing a path from start to end. Mazes represented adjacency lists, with each node and its neighbors encoded individual tokens (e.g., [1], [2), ..., [64]). Input/output formatting examples are shown in Figure 2 and Section C.2.
A challenge in defining length generalization for mazes is that increasing grid size introduces unseen node tokens at test time. To avoid this, we fix the grid size and instead vary the number nodes included in the spanning tree. Specifically, we define the input length as the total number of nodes in the maze graph and generate partial mazes by stopping Wilson's ' algorithm early. For example, to construct 32-node maze resulting maze may not span the full grid but remains a valid traversal problem. Figure 5 illustrates such partial mazes with 16, 32, and 64 nodes.
8 x grid, we run the algorithm until 32 nodes are added. The
Figure 8 mazes with number of nodes equal 16, 32, and 64. We define length generalization as the ability to generalize to mazes with higher number of nodes.
We consider two maze tasks: (1) shortest path where the model outputs the shortest path from
start to end node, and (2) DFS trace, where the model simulates depth-first search traversal (including backtracking). Shortest path is harder learn perfectly, as requires "lookahead" at branch points, while DFS trace allows exploration and backtracking. Figure 6 shows that in the

---

multi-task setting, the addition of shortest path helps DFS
trace generalize to higher lengths.
The opposite true as well: DFS trace helps shortest path generalize to higher lengths, which is shown in Figure
1.00
1.00
Accuracy
0.75
Accuracy
0.75
0.50
0.50
0.25
DFS Trace (main)
0.25
Shortest Path (aux)
Trace (main)
0.00
0.00
20
40
50
60
20
30
40
50
60
Length
Length
(a) Main: DFS trace.
Aux: shortest path
(b) DFS trace (No Aux Tasks)
Figure 6: Performance plots for maze tasks. Co-training DFS trace with shortest path (a)
enables generalization to longer lengths compared to training on DFS trace alone (b).
4.3.1 Transfer with Swapped Main and Auxiliary Tasks
We consider another maze task group where we the main and auxiliary tasks are reversed relative to Section 4.3. In this case, the main task shortest path, and the auxiliary task is DFS trace. As shown in Figure 7, co-training with the auxiliary task again improves length generalization performance. While shortest path is more difficult than DPS trace, the model benefits from learning related traversal strategy.
1.00
1.00
Shortest Path (main)
Accuracy
0.75
Accuracy
0.75
0.50
0.50
0.25
DFS Trace (main)
0.25
Shortest Path (aux)
0.00
0.00
20
30
50
60
20
30
40
50
60
Length
Length
(a) Main: shortest path Aux: DFS trace
(b) shortest path (No Aux Tasks)
Figure/ 7: Length generalization results for maze task group with reversed task roles.
Co-
training shortest path with DFS trace (a) leads improved generalization training shortest path alone (b).
4.4 Control Tasks
To verify that length generalization transfer does arise from merely seeing longer inputs, further test arithmetic tasks and string operations with control auxiliary tasks. or arithmetic, we use copy-first-op. which follows the addition format but simply copies the first operand. For string operations, pair string copy with reverse. As expected, length generalization transfer is not observed with unrelated task (Figure 8).
Length Generalization Transfer from Pretraining
Remarkably, we find that natural language pretraining can serve as an effective form of implicit auxiliary task that enhances length generalization in synthetic tasks. To explore this, we finetune various checkpoints of SmolLM-360M [Allal et al., 2024] on reverse add and shortest path

---

1.00
1.00
Accuracy
0.75
Accuracy
0.75
Copy (main)
Copy String (main)
0.50
0.50
Reverse Add (aux)
Reverse String (aux)
0.25
0.25
0.00
0.00
10
20
20
40
50
Length
Length
(a) reverse add & copy- first-op
(b) string copy & reverse
Figure 8: Control tasks for (a) addition and (b) string operations. These unrelated task pairs fail to produce length generalization transfer, confirming that task relatedness crucial.
tasks. SmolLM is released by Huggingface and pretrained on a diverse corpus containing natural language and programming data, which includes long-range structures and dependencies.
Before finetuning, we verify that the model does not already solve these tasks. For reverse add zero-shot evaluation using prompt-based input results in near-zero accuracy, confirming that the model has not learned this task during pretraining, For the maze task, all node tokens are newly introduced during finetuning, meaning the entire input format unseen by the pretrained model.
then finetune models from multiple publicly available checkpoints, taken throughout the pretrain- ing process (from step 160K to 2.56M), and evaluate their length generalization performance out-of- dittt-iiiittitttiion inputs. As shown in Figure 9, we observe clear trend: generalization to longer inputs improves steadily with pretraining progress, for both arithmetic and maze-solving tasks. This suggests that natural language pretraining instills reusable inductive biases that transfer to novel tasks-even when those tasks have little structural resemblance natural language. We speculate the extent of generalization transfer from pretrained models may not be limited to length generalization, but could extend other forms of out-of-distribution generalization such as compositional reasoning. distributional shifts, and task complexity. Future work could explore whether similar transfer effects exist for other generalization challenges.
Additionally, we confirm that length generalization transfer not limited to small models trained from scratch, but also emerges in finetuned pretrained models. Additional results across other task groups are provided in Appendix A.3.
1.0
1.0
Accuracy
Accuracy
0.6
0.6
step-0
step-0
step-160800
step-160800
0.4
step-320000
step-320000
step-640000
step-640000
0.2
step-1260000
0.2
step-1280000
step-2560000
step-2560000
0.0
10
25
Length
Length
(a) reverse add task.
(b) shortest path task.
Figure Finetuning at different SmolLM-360 M-30M checkpoints reveals that length generalization transfer improves with more natural language pretraining.
6
Ablations
In this section, we present several complementary analyses to better understand the conditions under which transfer occurs. We examine the effect of varying the length configurations of the main and auxiliary tasks and also provide an initial mechanistic explanation of the transfer phenomenon based

---

on circuit sharing between tasks. Additional analyses, including the instability of training dynamics (Section 6.2) and the effect of positional encodings (Section 6.3) are included in the Appendix.
6.1 Varying Main and Auxiliary Task Lengths
our previous experiments, we fixed the main task length to 16 and the auxiliary task length to 32. A natural question is: does length generalization transfer persist across other main main-auxiliary length configurations? To investigate this, we define the generalization gap (Figure 10), scalar between and that quantifies the discrepancy in performance between the main and auxiliary tasks across range of evaluation lengths. A smaller generalization gap indicates stronger transfer, with value of O implying perfect alignment between the main and auxiliary generalization curves.
First we fix the task group zeverse add no carry and carry only Then, we systemati- cally vary the training lengths of both main and auxiliary tasks across the range {4, 8, 16...., 256} and compute the average generalization gap over three random seeds. As shown in Figure 10, find that the transfer effect is most effective when the ratio between the auxiliary and main lengths is between 0.5 and The intuitive explanation is that, when the difference between task length is too high, the model will overfit to the task length difference and therefore do not exhibit length transfer.
Gener liizaiion ion Gap
Sum the difference between length vs. accuracy curves
0.91
0.94
0.93
0.87
0.63
0.20
0.00
128
0.94
0.95
0.91
0.36
0.04
0.00
0.34
0.8
Generalization Gap
main length
0.6
Main
0.90
0.72
0.15
0.00
0.16
0.34
0.87
0..4
Accuracy
15
0.60
0.03
0.00
0.23
0.67
0.90
0.95
0.28
0.00
0.27
0.74
0.91
0.91
0.89
0.2
0.00
0.41
0.65
0.78
0.78
0.78
0.75
0.0
16
32
64
128
256
Length
auxiliary length
Figure 10: (a) The generalization gap is defined as the average difference in accuracy between the main and auxiliary tasks across evaluation lengths, normalized the range [0, 1]. lower value indi- cates better transfer. (b) Generalization gap across different combinations of main ( reverse add) ) and auxiliary no carry & carry only) training lengths. The transfer effect is strongest when the ratio between auxiliary and main lengths is between 0.5 and 2, as shown by the dark diagonal band.
6.2 Unstable Training Dynamies in Length Generalization Transfer
seed
1.00
1.0
1.0
0.2
0.2
0.2
0.0
0.0
0.0
15
1.5
Training Staps [X10P)
Training Staps [X10P)
Training Steps [XLOP]
In-iistribtt Bution Ienn z 166
Buttio then 16]
In-Divit on (wen 16]
Unseen lan main 116 lan
Unseen lan by main (16 len 320
Unseen lon by main (16 a 32)
Unseentor
dan
by main
Figure 11: Training curves for the reverse add when co-trained with no carry and
carry only. Accuracy in the transfer region (length 17-32) fluctuates significantly, illustrat- ing unstable training dynamics in length generalization transfer.

---

As shown in Figures 3. 4, and 6, not all random seeds exhibit successful length generalization transfer. In our experiments with 5 different seeds per task group, we observe considerable variabilityy length generalization transfer performance. The variability is entirely due to different model initializations, since we keep the dataset the same between runs. To better illustrate this instability, we visualize training dynamics in Figure 11.
The plots show training curves for the reverse add main task when co-trained with no carry and carry only auxiliary tasks. During evaluation, we sweep input lengths from 36, which is classified into three regimes:
In-distribution (length 1-16): These inputs fall within the training range for the main task. Accuracy this regime improves quickly and remains stable.
Expected transfer range (length 17-32): These inputs are unseen by the main task but seen by the auxiliary tasks. Performance in this range is highly variable and sensitive training dynamics.
Fully OOD (length >32): These inputs are unseen by both the main and auxiliary tasks. As expected, accuracy in this regime remains low.
6.3 Rotary Position Encoding Encourages Length Generalization Transfer
In length generalization literature, NoPE (no positional encoding) often favored for its strong extrapolation on individual tasks. However, many modern transformer models use Rotary Positional Encoding (RoPE) due its empirical robustness long-context and real-world settings [Peng et al., 2023, Ding et al., 2024, Barbero et al., 2024].
We re-evaluate our multitask transfer setup under both encoding schemes. Across task families, RoPE consistently yields stronger length- generalization transfer from auxiliary to main tasks. Detailed per-task curves are provided in Appendix 1. Figure 12 summarizes the overall trend. This finding is orthogonal to the previous understanding that NoPE is better suited for length generalization and potentially explains the superior performance of RoPE in real-world models and tasks.
RoPE Model
NoPE Model
0.30
Generalization
0.25
0.20
0.15
0.10
0.00
Reverse Add, Reverse
Capitalize- Reverse
Copy,
Carry.
Only
Task
Figure 12: Comparison of generalization gap across task groups. Smaller gap means stronger transfer. RoPE consistently outperforms NoPE, indicating that rotary embeddings better support cross-task extrapolation.
7 Limitations
While our work demonstrates length generalization transfer across range of synthetic tasks, several important limitations remain. First, our study does not provide a formal theoretical framework for understanding when and why transfer occurs. Without a principled understanding of the underlying mechanisms, predicting or optimizing transfer remains challenging. Second, our experiments are limited relatively simple algorithmic domains with well-defined length parameters and deterministic solution paths. While this setup allows for controlled comparisons, it is unclear whether similar transfer effects would hold in settings that involve hierarchical reasoning, abstract problem-solving. or tasks requiring integration of multiple skills simultaneously. Addressing these limitations is promising direction for future work and could further illuminate the generalization capabilities of transformer models in more realistic settings.
10

---

References
Amirhesam Abedsoltan, Huaqing Zhang, Kaiyue Wen, Hongzhou Lin, Jingzhao Zhang, and Mikhail Belkin. Task generalization with autoregressivee compositional structure: Can learning from tasks generalize to d' tasks? arXiv preprint arxiv:2502.08991, 2025.
Kartik Ahuja and Amin Mansouri. On provable length and compositional generalization, 2024. URL https://arxiv. org/ abs/2402 04875.
AI@ Meta. Llama model card. 2024. URL htttt:::///gthbbb htt.tttaa..cmmmmtaaeeeee www.///mttaaammaaaaaeeeaaea 1atea--aamamaaaaaaa / wwwp/main/Moni. CARD mdd
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024.
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In Alice H. Oh, Alekh Agarwal, Daniclle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: wpw.revieeeeeeeeeeeeeeeeeeeeeewe net forum?id= 2KKVex=bCCK
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models, 2023.
URL htt.h:tt/s/ssttet http::s//axxiv.rrrr/. org/ abs/2312, 04927.
Pranjal Awasthi and Anupam Gupta. Improving wength-gneerllizttinn in transformers via task hinting, 2023. URL https: wtt.aaxivivv....../.aaa....... owg./ org/ www.aaasaa.. /2310. 00726.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Velivckovi's c. Round and round we go! what makes rotary positional encodings useful? ArXiv, abs/2410.06205, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel MerberttVosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https: // //arxiivorrgg./..o...aaoo.....oa abs 2005 14165.
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah, Curve http::/diisillllub///0//eiririie
circuits. Distill, 2021. doi: 10.23915/dissee circuits.
Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, and Chulhee Yun. Position coupling: Improving length generalization of arithmetic transformers using task structure. 2024. URL https: /api. sewanticschoneitioo.e wttpaaoooo//oost///22300 :D:
org
Yiran Ding, L. Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. ArXiv, abs/2402.13753, 2024.
Shaoxiong Duan, Yining Shi, and Wei Xu. From interpolation to extrapolation: Complete length generalization for arithmetic transformers. arXiv preprint arXiv:2310.11984 2023.
Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Location attention for extrapolation to longer sequences. arXiv preprint arXiv:1911.03877 2019.
Ying Fan, Yilun Du, Kannan Ramchandran, and Kangwook Lee. Looped transformers for length generalization. arXiv preprint 15644 2024.
Arian Hosseini, Alessandro Sordoni, Daniel Toyama, Aaron Courville, and Rishabh Agarwal. Not all Il reasoners created equal. arXiv preprint arXiv:2410.01748, 2024.

---

Dieuwke Hupkes, Verma Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020..
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. arXiv preprint arXiv:2307.03388 2023.
Nayoung Lee, Ziyang Cai, Avi Schwarzschild, Kangwook Lee, and Dimitris Papailiopoulos. Self- improving transformers overcome easy-to-hard and length generalization challenges, 2025. URL https: / /wwpvvaaaivvv//aa///aa//eeeaawaa org/ alss awwp/////22222 01612.
Mingchen Li, Xuechen Zhang. Yixiao Huang, and Samet Oymak. On the power of convolution- augmented transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 39, pages 18393-18402, 2025.
Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang. Sanjiv Kumar, and Srinadh Bhojanapalli. Functional interpolation for relative positions improves long context transformers. arXiv preprint arXiv:2310004418, 2023.
Sean McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, et al. Transformers can do arithmetic with the right embeddings. arXiv preprint vrx.v:22405.17399 2024.
Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning. The eos decision and length extrapolation. arXiv preprint arXiv:20/0.07174, 2020.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hladfeed-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL //tpxii.woo..r/////............. abs/ 2209. 11895.
OpenAl. Gpt-4 technical report, 2024. URL
08774.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. ArXiv, abs/2309.00071,, 2023.
Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108. /2409, 2021.
Philip Quirke and Fazl Barez. Understanding addition in transformers. arXiv preprint arXiv:2310.13/22, 2023.
Philip Quirke, Clement Neo, and Fazl Barez. Arithmetic in transformers explained, 2025. URL abs// 2402.02619.
Rahul Ramesh, Ekdeep Singh Lubana, Mikail Khona, Robert Dick, and Hidenori Tanaka. Compo- sitional capabilities of autoregressive transformers: A study on synthetic, interpretable tasks. In Forty-first International Conference on Machine Learning.
Anian Ruoss, Grégoire Delétang, Tim Genewein. Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, and Joel Veness. Randomized positional encodings boost length generalization of transformers. arXiv preprini arXiv:2305. 10.305/.6443... 2023.
Mahdi Sabbaghi, George Pappas, Hamed Hassani, and Surbhi Goel. Explicitly encoding structural symmetry is key length generalization arithmetic tasks, arXiv preprint arXiv:2406 0185, 2024.
12

---

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang. Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URL https: wwwp//wwee.eeeeeeep/pppe. net/
Ruoqi Shen, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, and Yi Zhang. Positional description matters for transformers arithmetic. arXiv preprint arXiv:2311.1477 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL httph: peattss:/arxivve wwtp:/iwwwrgga/////e/ooooo 09864.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL httpps://ae /aapii///rvvvvv//i org/ http/abs/22200 2211. 005993.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, O. Vinyals, P. Liang. J. Dean, and W. Fedus. Emergent abilities of large language models. ArXiv, (bss/200.07682. 2022.
David Bruce Wilson. Generating random spanning trees more quickly than the cover time. In Proceedings of the Ewenty-Eightt Annual ACM Symposium on Theory of Computing. STOC '96. page 296-303, New York, NY, USA, 1996. Association for Computing Machinery. ISBN 0897917855. doi: 10.1145/237814.237888 URL httpt:////oee 237880.
Zhuoyan Xu, Zhenmei Shi, and Yingyu Liang. Do large language models have compositional ability? an investigation into limitations and scalability, 2024. URL http:: htttps://arxvv.org abs/2407, 15720.
Haoran Yang, Hongyuan Lu, Wai Lam, and Deng Cai. Exploring compositional generalization of large language models. In Yang (Trista) Cao, Isabel Papadimitriou,, Anaelia Ovalle, Marcos Zampieri. Francis Ferraro, and Swabha Swayamdipta, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 4: Student Research Workshop), pages 16-24, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 11853/v1/2024 1/6653//1/2224mmaaclss/// naacl-srw.3. URL https: / aaa/cllaattttt htt///caannthhoggog www.orrg//00. 2024. http:/2222..2211aaaaaaaa haacl-ssw.3. 3/.
Gilad Ychudai, Ethan Fetaya, Eli Meirom, Gal Chechik, and Haggai Maron. From local structures to size generalization in graph neural networks. In International Conference on Machine Learning, pages 11975-11986. PMLR, 2021.
Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models, 2023. URL https: //arxiv..rr//.eeee........o...lt org/ 2310. 17567.
Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the out-of-distribution generalization of multimodal large language models, 2024. URL htts::/ arxiv. org/ http://abb/222200055 06599.
Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can models learn skill composition from examples?, 2025. URL htttttts:://axxiitt www. abs////// 40949990 19808.
Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? study in length generalization. arXiv preprint ar arXiv:2310.1600 2023.
13

---

Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou, Trans- formers can achieve length generalization but not robustly. arXiv preprint arXiv:2402.09371, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's contributions,
including investigating length generalization transfer across related tasks, demonstrating this phenomenon across algorithmic tasks, observing similar effects in pretrained language models, and providing mechanistic evidence related to attention head reuse.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper explicitly discusses limitations in Section 7, acknowledging the lack of comprehensive theoretical framework, the focus relatively simple algorithmic tasks, and open questions about how length generalization transfer might manifest in more complex scenarios.
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper is primarily empirical rather than theoretical in nature, and doesn't tt present formal theorems requiring proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Section and Appendix B provide comprehensive details about experimental settings, including model architectures, task definitions, data sampling procedures, training parameters, and evaluation methods necessary reproduce the results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our code is included in the supplementary materials section of the submission.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/ /ttp:s//nisp.aeee cc/
publica guides CodeSubmiii cesionPolicoe for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
cttpscccccccbccccccceeeeeeeelele public/ ceabbiccguiiee gu/dee//////////// CodeS GoodeSubme Submis ssi ceiionoolcen for more details. The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. cc/
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all relevant training and test details in Section 3 and Appendix B, including model architectures, task specifications, data generation procedures, and hyperparameter configurations as shown in Tables 1-3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper reports results across different random seeds for model initial- ization (mentioned in Section 3), and the figures show multiple runs with different random seeds, providing visual indication of result variability.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides type of compute, amount of compute used for each experiment run well estimate of total compute in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'ttnt make it into the paper).
paper (e.g., preliminary or failed experiments that http.////wc..ooccccccce.eeeeeee
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: Yes
Justification: The authors have reviewed and verified that the paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [NA]
Justification: The fundamental nature of the research on transformer models' length general- ization capabilities suggests limited direct societal implications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper works with synthetic algorithmic tasks rather than models datasets that pose risks of misuse, so safeguards aren't applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors
should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly cites previous work and models used, including SmolLM and other transformer architectures, attributing them to their creators with appropriate citations.
Guidelines:
19

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers swi thoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper doesn't introduce new datasets or models intended assets for the broader community.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This research doesn't 't involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
20

---

Justification:
No human subjects are involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [No]
Justification: While the paper studies transformer models, it doesn't use LLMs as part of its research methodology.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: :ppps//neeule /nnerinisss.. ce Cconferennece/eeeeceeeceeeeaeeee / 2025/ LLM) for what should should not be described.
21

---

A Additional Results
A.1 Detailed Plots for Rotary vs. NoPE Models
To complement Section 6.3, we show full length-generraizatton curves for the No Positional Encoding (NoPE) and Rotary Positional Encoding (RoPE) variants under the same task settings. Each subplot reports exact-match accuracy versus input length. The dashed vertical line indicates the maximum training length for each task. In all domains, RoPE models exhibit smoother extrapolation and better alignment between main and auxiliary tasks.
1.0
1.0
Accuracy
0.6
Reverse Add (main)
Reverse
0.4
0.4
Reverse Add (main)
No Carry (aux)
Only Carry
0.0
0.0
10
15
20
25
30
35
40
5
10
15
20
25
30
35
40
Length
Length
(a) Main: reverse add Aux: no carry & (b)
Main:
reverse add
Aux:
ca rry only
reverse subtract
0.8
1.0
0.6
0.8
Accuracy
0.6
0.2
Capitalize (aux)
Capitalize
0.4
Reverse (aux)
Copy String (main)
(aux)
0.0
Length
10
20
30
40
50
(c)
Main:
Sapitalize-revee
Aux:
Length
capi talize & reverse
(d) Main:
copy
Aux:
MQAR
Figure 13: Detailed performance curves comparing RoPE and NoPE variants across arithmetic and string tasks. RoPE models maintain strong transfer longer lengths, while NoPE variants degrade rapidly beyond the training range.
A.2 Additional Results on Arithmetic and String Tasks
For task groups with two auxiliary tasks- reverse add with carry and carry only and eapitalize-revett with capitalize and reverse -we additionally evaluate the effect of training with only one of the auxiliary tasks. As shown in Figure 14, length generalization transfer performance consistently declines when only a single auxiliary task is used, compared to co-training with both. Notably, the choice of auxiliary task matters: models trained with the more relevant auxiliary (no carry or reverse) exhibit stronger generalization than those trained
with less relevant ones ( (carry only or capitalize) ). These results reinforce the importance of task alignment for successful transfer. As shown in Figure 14, length generalization transfer performance consistently declines when only single auxiliary task used, compared to co-training with both. Notably, the choice of auxiliary task matters: models trained with the more relevant auxiliary no carry reverse exhibit stronger generalization than those trained with less relevant ones ( (carry only or capital ize). These results reinforce the importance of task alignment for successful transfer.
22

---

Accuracy
0.50
Revense Add (malin)
Reverse Add iman)
No
taux
0.50
Only (aux)
Reverse And (main)
0.25
(sox)
0.25
0.25
only Carry
0.00
0.00
0.00
20
30
20
30
10
Length
Length
Length
(a)
(b)
(c)
Top row: reverse add B: no carry carry only
1.00
Accuracy
Capitallo
Capitallos ize Reverse imain)
Cap Reverse (mar) Capitaize
CaptairzeRRveen Reverse Imaini
laur)
0.25
0.00
Longth
Length
Length
(d)
(e)
Bottom row: A: capitalleo-roverac B: capitalize, C: reverse
Figure 14: Additional results for arithmetic and string (copy) task groups. Each row shows perfor- mance on the main task (A) when co-trained with: both auxiliary tasks (left), only one of the auxiliary task (middle & right). Performance degrades when training with only one auxiliary task, especially when the auxiliary is less structurally aligned with the main task.
23

---

A.3 Finetuning from Pretrained Models
We replicate our length generalization transfer experiments using pretrained language model, MMO0LMM where we observe similar patterns of length generalization transfer as in the from- scratch setting. Figure 15 presents results across three arithmetic task groups and one string ma- nipulation group. As with our earlier experiments, co-training with structurally related auxiliary tasks facilitates generalization beyond the training length. Notably, we also confirm that control task pairs-such surs--uchhan as reverse add with copy-first-on do not lead successful transfer. Orthogo- nal to the length generalization transfer, results show that SmolLM-36000 exhibits strong inherent generalization in copying tasks (15c, 15d).
1.0
A seeds)
0.8
(3 seeds)
0.8
Accuracy
0.6
0.4
A seeds)
0.2
0.2
seeds)
(3 seads)
20
10
30
50
Length
Length
(a) reverse add B: reverse subtract
(b)
reverse add
B: no carry
C: carry only
1.0
0.8
0.8
Accuracy
(3 seeds) (3 seeds)
B (3 seeds)
0.4
(3 seeds)
0.2
0.0
20
40
10
20
30
40
50
60
Length
Length
(c) A: reverse add cory-first
(d)
A: capptallie-
B: cacpptallie reverse
Figure 15: Length generalization transfer with the pretrained model SmolLM-360M. (a-c): Arith- metic task groups. (a) and (b), we observe successful transfer from auxiliary to main tasks, mirroring results from from-scrater training. In (c), no transfer occurs when using the control task copy-tizst-op. confirming the importance of task relevance.
(d): String manipulation task,
showing transfer from capitalize and zeverse to capitalize Seettalieerreeeseeee reverse. Overall, the transfer effect persists in the pretrained model.
24

---

B Mechanistic Evidence of Circuit Sharing
In this section we consolidate the mechanistic analysis of length generalization transfer, showing that successful transfer coincides with reuse of internal attention circuits across related tasks.
Metries and protocol. We study whether transformer models reuse similar attention mechanisms across tasks when length generalization transfer occurs. We use complementary metrics:
Attention matrix difference: sum of entry-wise absolute differences between attention matrices (per head) for two tasks. Lower values indicate more similar attention patterns.
Attention-head mean-ablation map difference: for each head (6 layers heads), replace its output with the batch mean and measure the accuracy drop (activation patching). This produces head-impont -na-rrtpee map per task; then take the average absolute difference between the two maps. Lower values indicate more similar head usage.
This follows standard activation-patchiin methodology used in mechanistie-interpreaal
stud- ies [Wang et al., 2022, Cammarata et al., 2021, Olsson et al., 2022]. Across checkpoints, reductions in the generalization gap (defined in Fig. 10) generally coincide with smaller differences in both metrics i.c., tighter alignment of attention mechanisms across tasks when transfer strengthens.
1.00
0.0028
Attention difereecc
0.20
Head ablation map difference
Generalization Gap
0.15
0.75
0.0026
0.10
0.50
0.0024
0.0022
0.05
0.25
0.00
2000
4080
6050
5000
10000
12:000
14000
16000
18000
20000
2000
4000
6m00
6000
10000
12000
14080
160000
18850
20080
2000
41000
6030
6000
10me
12000
148800
18n00
10000
20100
Checkpoint
reverse subtract no carry ). Evolution of generalization gap, attention matrixx difference, and head mean-ablation map difference across checkpoints. When transfer improves (smaller gap), attention mechanisms align (smaller differences).
Figure 16:
Arithmetic task group reeeerse add with auxiliaries
B.0.1 Example Attention-Heal Ablation Maps
We visualize the attention-head mean-ablation maps for pair of related tasks reverse add and reverse subtract across four training checkpoints (Figure 17). Each 6 x 6 matrix represents the importance of each attention head: the value at position (i. indicates the drop in accuracy when head layer j is replaced with the mean activation across the batch. These matrices reveal which heads are functionally critical for each task. If two tasks reuse the same circuitry, their ablation maps will appear similar; our scalar similarity metric is the average absolute difference between the two matrices.
B.0.2 Extended Results Across Tasks
We next compare how the two circuit-similarity metrics track with the generalization gap across training checkpoints for string, arithmetic. and control tasks.
String tasks. Figure 18 shows that the raw attention-matrix difference does not correlate well with generalization for string tasks, whereas the ablation-man -app difference does. Shared head usage, rather than raw attention weights, better captures functional similarity.
25

---

2
0:25
0.27
227
0:21
1.14
0.03
2.44
0
3.22
6.22
-00
0000
0.95
0.00
0.25
4.46
2.87
4.00
0.33
3:31
4077
6:22
0.2
860
0.00
2.02
-855
40.05
0.00
4032
000
0.28
0.
2.04
604
6:37
0.29
3.00
60:14
6.21
0.20
3.03
4,79
0.01
0.00
400
-3.8
4.30
0.66
0.02
0.50
4.73
0:31
Task A, 2k steps
(b) Task
8k steps
(c) Task
16k steps
(d) Task A, 20k steps
0.02
40
20
3:20
0:
6:30
0.50
6:12
44
6.1
3.00
4:20
0.90
600
3:29
6.13
0.01
0.94
100
0.35
0.01
002
3.33
-C4
2.08 3.34
0.7
4:11
603
-03
6:41
000
4.22
8:00
2.0L 2.08
0.00
250
0.22
0:00
0.33
6.22
12
0.51
4.55
3.34
4.30
4.62
0:22
455
-3.0
4.32
2.2
0.02
6013
3.00
4.73
05
(e) Task B, steps
(1) Task B, steps
(g) Task B. 16k steps
(h) Task B, 20k steps
Figure 17: Mean-ablation maps for reverse agd and reverse subtract across checkpoints.
Each ji entry shows the accuracy drop after mean-ablating head layer j. Similar maps indicate overlapping computational circuits.
1.00
0.018
Altention difference
ablation map differncce
Generalizatinn
0.016
0.050
0.50
0.014
0.25
0.000
1000
1500
2030
2500
2500
4000
4500
50000
500
1000
1500
2000
2500
3000
3500
d0op
4500
50o0
500
1000
1500
2000
2500
2000
3500
4000
4500
Soon
Chockpoint
(a) A: capital i ze reverse B:
capital ize C: reverse
1.00
Attention difference
Head ablation map difference
Generalization Gap
0.0175
0.03
0.0150
0.25
0.02
0.0125
0.00
500
1000
1500
2000
2300
3000
2500
4000
2500
5000
500
1000
1500
2000
2500
3000
3500
4300
5000
500
1000
1500
2000
2500
3000
3500
400g
4500
5000
Checkpoint
(b) copy B: MQAR
dfference
0.15
Head dfferienc
Ganvralization Gap
0.016
0.75
0.10
0.50
0.014
0.05
500
1000
1500
2010
2500
2000
3500
4000
4500
5000
500
1000
1500
2000
2500
3000
2000
4000
4500
5000
500
1000
1500
2000
2500
3000
3500
1000
4500
5000
Checkpoint
(c) A: copy B: reverse
Figure 18: Circuit-sharing results for string task pairs. The attention-matri difference shows weak correlation with generalization, whereas the head-ablation map difference tracks it closely, highlighting shared head usage.
26

---

Arithmetic tasks. For arithmetic task pairs (Figure 19), both metrics strongly correlate with the generalization gap, suggesting that these tasks share not only head usage but also detailed attention- pattern structure.
Control tasks. For unrelated task pairs such as reverse add with copy- copp--frrttoop neither metric correlates with performance, confirming that the observed correlations are not incidental.
difference
Head ablation map difference
Generalization Cap
0.011
0.50
0.010
0.25
0.02
0.00
2000
4000
6000
8000
10000
12080
14h00
16050
1amr
20000
2080
4000
6mm0
8080
1000
12000
14000
10000
18080
20/00
2000
4000
6000
8000
10080
12000
16000
lamo
20100
Checkpoine
(a) A: reverse add B: copy-Eirse
1.00
0.0020
Altention difference
Head ablation difference
Generaization Cap
0.15
0.0026
0.0024
0.10
0.50
0.0022
0.05
0.00
2000
4000
6000
6000
10030
12000
14030
16000
16000
20000
2000
6000
6030
20000
12000
14000
16000
16000
20000
2000
4000
6050
10000
12000
14030
16at0
lama
20000
Checkpoint
(b) A: reverse add B: no ca rry
1.00
Attantion difference
Head ablation map difference
Generalizaton Cap
0.0045
0.15
0.75
0.0000
0.10
0.30
0
0.0035
0.00
2010
4000
6000
8008
100000
12000
14050
1d000
18n00
20000
4000
8050
10000
120ne
14000
16000
16000
20000
4000
60n0
0000
10000
13000
14000
165n0
18000
20000
Checkpoint
(c) A:
add, B:
subtract
Figure 19: Circuit-sharing results for arithmetic tasks. Both attention-matrix and head-ablation map differences correlate with generalization gap related task pairs (b,c) but not in the unrelated control pair (a).
Takeaway. Across arithmetic, string, and maze domains (not shown), strong transfer coincides with shared attention-head usage between tasks. This supports the hypothesis that transformers reuse compositional "scaffold" of attention circuits when transferring extrapolation behavior across related problems.
27

---

C
Experiment Details
C.1
Model
For all experiments, use decoder-only transformer models following the Llama architecture. Unless otherwise specified, we use Rotary Positional Embeddings (RoPE) for positional encoding: exceptions are noted in the ablation studies in Section 6.3.
For pretrained model experiments, we Smo0LM-360M [Allal al., 2024], compact transformer trained on natural language and code. Table summarizes the model configurations used in our experiments.
Table I: Model configurations used in our experiments.
Model
Self-Attn Layers
Num Heads
Embedding Dim
Sorm-Scratch SmolLM
6
6 384
32
15
2560
C.2 Data Formats and Data Sampling
We provide examples of each task in Table 2. For all arithmetic tasks, both the inputs and outputs written in reverse digit order. For the x CoT multiply task, the output includes intermediate steps where the first operand is multiplied by each digit of the second operand.
For maze- based tasks, we serialize graphs using adjacency list format with unique node tokens, followed by query specifying the start and end node. detailed example is shown in Figure 20.
Table 2: Examples of algorithmic tasks used in experiments.
Task Name
Input
Output
only carry
82050465+237829555-
010010111
no carry
$2050465+237822555=
057323100
reverse add
$2050465+23782955=
067333211
subtract
$2050465+23782955=
692674000
3 CoT multiply
60844671+502=
03042283--00000000
02042288-0021169255300
0325817163
copy string
NVOBAIfR=
fVOBA1fR
Multi-Query
[VOBAIIR=
IVOB;OBAI;
Associative Recall
string reverse
NVBBAIR==
REIABOVE
capitalize
NVBBIIR=
FvobalFr
capitalize-rent reveree
IVOBAIR==
rFlabovF
Shortest Path
[0][10], [15]44]55, [11][[][[][5],
[1][8][13] [11]ll[o1)] (011111411111111111111
[3111 [411111551
[14]:19[55.
[101111111111.
[2]:14]
[1]:[1]] [7]:5]. [13]:18][10.
511111111111555 [12][8][6].
[І1[1]144 [81(111133) [6]:[12]
122)>12
DFS trace
[0]100] [1514115], [1111113355)
[12]6]]
[311111) 1411111511.
[14]:1115].
[1]] 12118101111111111111111111111111
[10][(111111]
[2]:14]
[1]][]] [5][15][4]1]
[1]:[11] [7][S]. [13]:[8][10.
[12]:[8][6]
19]1111 [811111113) [6][[12]
112]>[217
28

---

Maze with Solution Peth
20
Figure 20: Detailed example of maze data format. Each node is a random number selected from n X n nodes in the grid.
C.3 Experimental Settings
C.3.1 Hyperparameter Configurations
Table 3 lists the hyperparameters used for training across different task domains and model types. From-scratch models are trained with a higher learning rate and larger batch sizes, while pretrained models (Sm00LM)360M) use lower learning rates and shorter training schedules. All models optimized using AdamW with a learning rate schedule that includes warm-up phase, a constant phase, and cosine decay phase.
Table 3: Hyperparameters for training
Task
Batch Size
LR
Iterations
Warmup Iter
Decay Iter
Arithmetic Tasks
1024
le-3
20000
2000
5000
String Tasks
1024
le-3
5000
500
1000
Maze Tasks
256
le-3
20000
2000
5000
Arithmetic Tasks (SmolLM)
128
5e-5
2500
250
500
String Tasks (SmolLM)
128
5e-5
1000
100
500
Maze Tasks (SmolLM)
256
Se-5
2500
250
500
C.3.2 Computational Resources
For all experiments in the paper, we run on single machine with two NVIDIA GeForce RTX 3090 graphics cards. For all experiment settings, each individual training run is at most 2 hours. The total estimate of compute used, in terms of hours on the 2-GPU machine, is around 300 hours.
29


Reviews:
[{"review_id": "u4hcnzbqhK", "reviewer": "Reviewer_8R8h", "summary": "This paper investigates a novel mechanism for improving length generalization in Transformers, termed \"extrapolation by association.\" The central hypothesis is that a model can learn to generalize to longer, unseen input sequences for a \"main task\" if it is jointly trained on a related \"auxiliary task\" that includes examples of those longer sequences. The authors provide empirical evidence for this phenomenon, which they call \"length generalization transfer,\" across a diverse set of algorithmic domains: arithmetic (e.g., reverse add), string manipulation (e.g., capitalize-reverse), and maze navigation (e.g., DFS trace).\n\nThe key findings are:\n1. Co-training with a longer, related auxiliary task consistently enables the main task to extrapolate beyond its training length, a capability it lacks when trained in isolation.\n2. This transfer is contingent on the structural similarity between tasks, as control experiments with unrelated tasks show no such benefit.\n3. The effect is also observed when fine-tuning pretrained language models (SmolLM), and the capacity for generalization improves with the extent of pretraining, suggesting that pretraining builds a reusable \"computational scaffolding.\"\n4. Initial mechanistic analysis suggests that successful transfer correlates with the reuse of the same attention heads for both tasks, indicating a sharing of internal computational circuits.", "questions": "1. The instability of transfer due to random initialization (Sec A.1) is a significant concern, as it suggests the phenomenon might occur by chance rather than being a reliable property. Could you analyze the properties of the model initializations that lead to successful transfer versus those that fail? For instance, are there discernible statistical differences in initial weight norms or other properties? A positive result here would greatly increase my confidence in the robustness of your findings and could change my score.\n2. The concept of \"task relatedness\" is central to your claims but is defined intuitively. Your post-hoc analysis showing a correlation with circuit sharing is insightful, but it doesn't help in predicting which tasks will work together beforehand. Have you considered any a priori metrics for quantifying task relatedness (e.g., based on algorithmic complexity, information theory, or embedding similarity of task descriptions) that might predict the likelihood or effectiveness of length generalization transfer?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength: \n- The paper introduces a novel and significant research direction for length generalization, shifting focus from purely architectural solutions to the training paradigm itself.\n- The experimental validation is thorough and well-designed, using three distinct task domains which strongly supports the generality of the core claim.\n- The inclusion of control experiments with unrelated tasks provides compelling evidence that the observed transfer is due to task relatedness and not merely exposure to longer sequences.\n- The investigation extends to pretrained models, bridging the gap between findings on synthetic tasks and their potential relevance to large language models.\n- The mechanistic analysis using attention head ablation maps provides valuable, albeit initial, insight into how the transfer might be occurring at the circuit level.\n\nWeakness: \n- The primary weakness is the reported instability of the training dynamics; the success of the transfer is highly sensitive to random model initialization, which may limit the reliability and practical applicability of the phenomenon.\n- The crucial concept of \"task relatedness\" is defined intuitively and only analyzed post-hoc; the work lacks a predictive, a priori metric to determine which tasks will facilitate transfer.\n- The experiments are confined to relatively simple, deterministic algorithmic tasks, and it remains unclear how these findings would translate to more complex, ambiguous, or knowledge-intensive domains like natural language.", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you for your constructive review pointing out the novelty and thoroughness of our research!**\n\nLet us address the three main concerns below\n\n## 1. Instability of the training dynamics\n\nTo further investigate the source of training instability, we conducted two experiments that either 1) varied 5 seeds for model initialization keep dataset the same, or 2) varied 5 dataset seeds and keep the model initialization the same. However, in both cases, the transfer phenomenon is unreliable. This suggest that both model initialization and dataset composition contributes to the unstable generalization transfer. A previous work [5] similarly observed that \"length generalization is not robust to neither weight initialization nor training data order\". \n\nThe third source of instability is from the training progress itself. Section A.1 shows that the generalization performance in fact oscillates in the same training run. Another work [1] also makes similar observations for hierarchical generalization, describing it as \"competition\" between two solutions (one generalizes, one does not). We will update these reference and discussions in the paper.\n\nImportantly though, the unstable training dynamics is only limited to the train-from-scratch case. Section A.5 repeat the same experiments for pretrained models and we observe successful length generalization across multiple seeds, suggesting more robustness for pretrained models.\n\nThe transfer is also robust for highly correlated tasks. For example, training the following tasks together always results in robust transfer to longer addition. \n* task 1: short reverse addition with random padding on either side, e.g. 00012300+00045600\n* task 2: short reverse addition, e.g. 123+456\n\nOverall, We found that finetuning from pretrained model or having highly aligned tasks can result in robustness of such transfer.  However, it is unclear in the general case whether the instability is initialization-driven or data-driven, and it is an interesting future direction to find conditions for robust generalization transfer. \n\n$$ $$\n## 2. A predictive, a priori metric for \"task relatedness\" \n\nThis is an important direction, and we are working on one promising hypothesis, that \"pairs of tasks with high mutual information will result in length generalization transfer\". Concretely, assume we have two functions over random variables, $f$ and $g$, which takes the same random variable $Z$ to $X=f(Z)$ and $Y=g(Z)$, then define the estimated mutual information $$I(f;g)=I(X; Y) = \\sum_{x \\in \\mathcal{X}} \\sum_{y \\in \\mathcal{Y}} p(x, y) \\log \\left( \\frac{p(x, y)}{p(x)\\,p(y)} \\right)$$\nWe can then measure the mutual information between the task pairs that exhibit transfer and tasks that don't. For example, to estimate the mutual information for the addition task, \n\n* The input $Z$ are two integers sampled uniformly sampled over length $l$. \n* The functions of interest are\n    * reverse addition ($f$),\n    * carry only ($g$),\n    * no carry ($d$)\n    * reverse subtraction ($e$)\n    * Repeat first (control task) ($h$)\n* Suppose we want to estimate $I(f;g)$. We can sample a dataset $D_Z \\sim p(Z)$, and apply our functions to the dataset $D_X=f(D_Z)$ and $D_Y=g(D_Z)$. \n* Finally we estimate the discrete mutual information using $D_X$ and $D_Y$ according to the formula. \n\nEstimating using task length 8 and 1 million examples, the results below shows that the control task has much less mutual information than the other tasks. \n- $I(f; g) = 1.7294$\n- $I(f; d) = 5.5450$\n- $I(f; e) = 0.7094$\n- $I(f; h) = 0.0331$ - Control task\n\nWe believe this is a promising direction for formalizing the \"task relatedness\" concept, and we will include this framework in the paper if it passes more careful validation. \n\nWe have also included the script for the example mutual information estimation at the end. \n\n$$ $$\n## 3. More complex domains like natural language\n\nIt is indeed a very interesting future direction to find a natural language setting to investigate length generalization transfer. For example, length generalization of math proof, or more complex synthetic settings like GSM-Infinite [4]. The challenge is that natural language tasks are almost always an amalgamation of many sub tasks, which makes it hard to isolate the transfer effect. We will update the limitations & future works section to include these specific examples. \n\nIn this work, we mainly follow and expand upon the experimental settings in the line of algorithmic length generalization literature [2] [3], focusing on smaller-scale, controlled experiments. \n\nWe hope these clarifications address your concerns and will gladly provide further details in the camera‑ready if accepted.\n\n$$ $$\n$$ $$\n$$ $$\n$$ $$\n$$ $$\n\n_References_\n\n[1] T. Qin, N. Saphra, and D. Alvarez-Melis, Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalization, arXiv:2412.04619, 2024\n\n[2] H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. Susskind, S. Bengio, and P. Nakkiran,\n“What algorithms can transformers learn? A study in length generalization,”\narXiv preprint arXiv:2310.16028, 2023.\n\n[3] E. McLeish, A. Bansal, A. Stein, N. Jain, J. Kirchenbauer, B. R. Bartoldson, B. Kailkhura, A. Bhatele, J. Geiping, A. Schwarzschild, et al., “Transformers can do arithmetic with the right embeddings,”\narXiv preprint arXiv:2405.17399, 2024.\n\n[4] Y. Zhou, H. Liu, Z. Chen, Y. Tian, and B. Chen, “GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?,” arXiv preprint arXiv:2502.05252, 2025.\n\n[5] Y. Zhou, U. Alon, X. Chen, X. Wang, R. Agarwal, and D. Zhou, “Transformers Can Achieve Length Generalization But Not Robustly,” arXiv preprint arXiv:2402.09371, Feb. 2024.\n\n_MI Estimation Script_\n```python\nimport numpy as np\nfrom sklearn.metrics import mutual_info_score\nfrom collections import Counter\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\ndef int_to_bits(n, length):\n    return [int(b) for b in np.binary_repr(n, width=length)][::-1]\n\ndef reverse_add(a_bits, b_bits):\n    carry = 0\n    result = []\n    for a, b in zip(a_bits, b_bits):\n        total = a + b + carry\n        result.append(total % 2)\n        carry = total // 2\n    return result\n\ndef carry_bits(a_bits, b_bits):\n    carry = 0\n    result = []\n    for a, b in zip(a_bits, b_bits):\n        total = a + b + carry\n        result.append(int(total >= 2))\n        carry = total // 2\n    return result\n\ndef repeat_first(a_bits, b_bits):\n    return a_bits + a_bits\n\ndef digitwise_sum_mod_10(a_bits, b_bits):\n    return [(a + b) % 10 for a, b in zip(a_bits, b_bits)]\n\ndef reverse_subtract(a_bits, b_bits):\n    borrow = 0\n    result = []\n    for a, b in zip(a_bits, b_bits):\n        diff = a - b - borrow\n        if diff < 0:\n            diff += 2\n            borrow = 1\n        else:\n            borrow = 0\n        result.append(diff)\n    return result\n\ndef bits_to_tuple(bits):\n    return tuple(bits)\n\ndef generate_samples(n_samples, bit_length):\n    max_val = 2 ** bit_length\n    a_vals = np.random.randint(0, max_val, size=n_samples)\n    b_vals = np.random.randint(0, max_val, size=n_samples)\n    return a_vals, b_vals\n\ndef compute_mutual_info(f_outs, g_outs):\n    unique_f = list(set(f_outs))\n    unique_g = list(set(g_outs))\n    f_map = {x: i for i, x in enumerate(unique_f)}\n    g_map = {x: i for i, x in enumerate(unique_g)}\n    f_encoded = [f_map[x] for x in f_outs]\n    g_encoded = [g_map[y] for y in g_outs]\n    return mutual_info_score(f_encoded, g_encoded)\n\ndef joint_distribution(x_outs, y_outs):\n    joint_counts = Counter(zip(x_outs, y_outs))\n    df = pd.DataFrame([\n        {'x': x, 'y': y, 'count': count}\n        for (x, y), count in joint_counts.items()\n    ])\n    return df\n\ndef plot_joint_distribution(df, title):\n    pivot = df.pivot_table(index='x', columns='y', values='count', fill_value=0)\n    plt.figure(figsize=(10, 8))\n    sns.heatmap(pivot, cmap=\"Blues\", cbar_kws={'label': 'Frequency'})\n    plt.title(title)\n    plt.xlabel('Output of second function')\n    plt.ylabel('Output of f')\n    plt.tight_layout()\n    plt.show()\n\ndef main(l=8, n_samples=1000000):\n    a_vals, b_vals = generate_samples(n_samples, l)\n\n    f_outs, g_outs, h_outs, d_outs, e_outs = [], [], [], [], []\n\n    for a, b in zip(a_vals, b_vals):\n        a_bits = int_to_bits(a, l)\n        b_bits = int_to_bits(b, l)\n        f_outs.append(bits_to_tuple(reverse_add(a_bits, b_bits)))\n        g_outs.append(bits_to_tuple(carry_bits(a_bits, b_bits)))\n        h_outs.append(bits_to_tuple(repeat_first(a_bits, b_bits)))\n        d_outs.append(bits_to_tuple(digitwise_sum_mod_10(a_bits, b_bits)))\n        e_outs.append(bits_to_tuple(reverse_subtract(a_bits, b_bits)))\n\n    mi_results = {\n        \"I(f; g)\": compute_mutual_info(f_outs, g_outs),\n        \"I(f; h)\": compute_mutual_info(f_outs, h_outs),\n        \"I(f; d)\": compute_mutual_info(f_outs, d_outs),\n        \"I(f; e)\": compute_mutual_info(f_outs, e_outs),\n    }\n\n    print(\"Mutual Information Results:\")\n    for k, v in mi_results.items():\n        print(f\"{k} = {v:.4f}\")\n\n    # Plotting\n    # plot_joint_distribution(joint_distribution(f_outs, g_outs), \"Joint Distribution of f and g\")\n    # plot_joint_distribution(joint_distribution(f_outs, h_outs), \"Joint Distribution of f and h\")\n    # plot_joint_distribution(joint_distribution(f_outs, d_outs), \"Joint Distribution of f and d\")\n    # plot_joint_distribution(joint_distribution(f_outs, e_outs), \"Joint Distribution of f and e\")\n\nif __name__ == \"__main__\":\n    main()\n```"}, {"review_id": "P839s70CKj", "reviewer": "Reviewer_hBvs", "summary": "This paper uncovers a novel finding adding to our understanding of transformer models: length generalisation can transfer across similar tasks in a variety of data, task, and model settings. They validate this on several task families, show instances of lack of generalisation due to task dissimilarity, show the finding holds on pretrained models, and provide interpretability analysis showing cross-task transfer is related to shared underlying mechanisms.", "questions": "- How were these tasks chosen? For the synthetic tasks, is it possible to formalise why task transfer is happening? (E.g. is the [C-RASP](https://arxiv.org/abs/2506.16055) program for these tasks sharing some key operations?) This isn't necessary for the paper as it stands, but I'm curious if there is some intuition on the authors part as to when generalisation does or does not occur across tasks.\n- Can you explain figure 9 more? Particularly the leftmost subplot doesn't seem to be showing a strong trend. Maybe some error bars could help guage the significance of the correlation in the three metrics.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n- This is a really interesting and novel finding that adds to our understanding of language models. The overall idea is well-motivated and the paper is very nicely written, and the analyses (generalisation gap, different checkpoints, control task transfer) is comprehensive. Overall, I am very convinced by the narrative presented by the paper.\n- I appreciate the inclusion of details of mechanistic results in sec. 6.2 and the relevant appendices, which elucidate why this happens.\n- Further experiments comparing NoPE and RoPE are great to see; while these are in the appendix, this is important (along with the control task failures) to show that the finding is not merely about better learning of how to use RoPE but something more fundamental about cross-task transfer.\n\nWeaknesses\n- None apparent.", "was_revised": true, "final_justification": "I am satisfied by the author responses and hope to see this paper accepted; it is a valuable contribution to the field.", "rebuttal": "**Thank you for your encouraging review pointing out the novel findings and detailed experiments!**\n\nLet us address your questions below\n\n> How were these tasks chosen? \n\nWe picked the string and arithmetic tasks to be common tasks used in the length generalization literature [1] [2], and to increase the task complexity, we picked the maze solving task.\n\n> For the synthetic tasks, is it possible to formalise why task transfer is happening? (E.g. is the C-RASP program for these tasks sharing some key operations?)\n\nFormalizing the transfer condition is indeed an interesting future direction. Our current hypothesis is that task pairs that has higher mutual information is more likely transferrable, as measured by the mutual information of the task function applied to the input random variable. We provide more details in the rebuttal for reviewer 8R8h. \nWe also believe that the tasks needs to share some internal mechanism in order for the transfer to happen, and the C-RASP framework as you have mentioned would definitely help us. \n\n> Can you explain figure 9 more? Particularly the leftmost subplot doesn't seem to be showing a strong trend. \n\nYou are right to point out weak correlation of the \"attention difference\" metric comparing to other two metrics. Since its function is the same as the \"head ablation map difference\" metric, we will consider removing this metric.\n\nLooking at Fig 9 and Fig 19 together, the observation is that\n* For task pairs with successful transfer, the \"head ablation map difference\" metric correlates with generalization gap. \n* For task pairs without transfer, the same metric does not correlate with generalization gap. \n\n> Maybe some error bars could help guage the significance of the correlation in the three metrics.\n\nWe display the metrics for individual seeds as opposed to aggregate over multiple seeds (with error bars) because we want to show that the local maxima or \"bumps\" in the generalization gap is mirrored on the other two metrics. Fig 19 shows more examples of this correlation in different tasks. \n\n\n$$ $$\n$$ $$\n$$ $$\n_References_\n\n[2] H. Zhou, A. Bradley, E. Littwin, N. Razin, O. Saremi, J. Susskind, S. Bengio, and P. Nakkiran,\n“What algorithms can transformers learn? A study in length generalization,”\narXiv preprint arXiv:2310.16028, 2023.\n\n[3] E. McLeish, A. Bansal, A. Stein, N. Jain, J. Kirchenbauer, B. R. Bartoldson, B. Kailkhura, A. Bhatele, J. Geiping, A. Schwarzschild, et al., “Transformers can do arithmetic with the right embeddings,”\narXiv preprint arXiv:2405.17399, 2024."}, {"review_id": "4JZyJHHMIU", "reviewer": "Reviewer_ySe2", "summary": "This paper investigates whether being trained on longer instances of an auxiliary task helps with length generalization on a main task. The authors find that it generally does, but only when the tasks are related and when the length difference is not too stark. They also find evidence that the improvement in generalization performance is associated with an increased sharing of mechanisms (as indicated by attention maps and ablation maps) between tasks.", "questions": "As noted above, I really liked this paper. My primary concern is the evidence in Section 6.2. I think a more extensive evaluation of the hypothesized association between mechanism sharing and length generalization would make the paper considerably stronger (whether or not this hypothesized association turns out to be true). In my comments above, I have provided one example for what such an analysis could look like. If this is not possible in the rebuttal period, I think it would be good to more strongly acknowledge the limitations of the current approach.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\nI really liked this paper. The idea is clear and elegant, the results are convincing, and I appreciated that the authors studied (relatively small) pretrained language models as well. I also thought the paper was well written and I thought the schematic figures indicating the authors' hypotheses were very nice. The limitations section helpfully acknowledges questions that this paper does not address and highlights potential for future work. Finally, I thought it was good (and important) that the authors studied tasks from several domains.\n\n**Weaknesses**\n\nPrimarily, I think Section 6.2 is currently lacking. It is both a little difficult to assess the extent to which ablation and attention difference are similar over time (other than them both going down) and there are many confounding factors, as many quantities vary which the training epoch (e.g. training set performance, in-distribution generalization, convergence to more consistent solutions from potentially more random initializations). I therefore think the evidence provided for more length-generalizing networks having more mechanisms in common is pretty weak. I think it would be interesting to compare the ablation and attention difference between different random seeds that exhibit different length generalization or between the networks exhibiting different length generalization in Figure 8. Finding an association between ablation/attention difference and length generalization across different networks would be much stronger evidence in my opinion.\n\nAs a more minor comment, I think it would be nice to connect the limitations section (which, as noted above, was generally good) to prior work that could be useful in addressing these questions.", "was_revised": true, "final_justification": "I continue to think this is a good paper that should be accepted to Neurips.", "rebuttal": "**Thank you for your encouraging review highlighting the clarity of our idea, results and presentation!**\n\nLet us address your concerns below:\n\n> It is both a little difficult to assess the extent to which ablation and attention difference are similar over time (other than them both going down) and there are many confounding factors, as many quantities vary which the training epoch (e.g. training set performance, in-distribution generalization, convergence to more consistent solutions from potentially more random initializations). \n\nWe agree the analysis in sec 6.2 has much room to improve. In particular we will consider removing the \"attention difference\" plot since the correlation with generalization gap is weak. We'd also like to point out a few clarifying points: \n\n* In Fig 9 and 19, the training set performance and in distribution generalization saturates to 100% at step 2000, the first point on the x-axis. Therefore the plot shows entirely the progress of out-of-distribution performance. \n* The \"bumps\" of the OOD performance (generalization gap) is the result of unstable training dynamics (Sec A.1), this means throughout training, the model flip-flops between a generalizable solution and one that is not. \n* We show that whenever the generalization gap is high, so is the head ablation map difference (and vice versa), therefore linking the macroscopic metric with the mechanistic metric. \n\n> I think it would be interesting to compare the ablation and attention difference between different random seeds that exhibit different length generalization or between the networks exhibiting different length generalization in Figure 8.\n\nYes, we agree. Figure 9 and 19 only shows the comparison for different tasks at different checkpoints, and we are working on incorporating further variations between different seeds. \n\nWe will update the manuscript to acknowledge the limitations of the current mechanistic analysis."}, {"review_id": "XVnlxvyabo", "reviewer": "Reviewer_BHPD", "summary": "This paper addresses the length generalization problem, a challenge facing transformers. The paper finds that length generalization can transfer between tasks: when training two tasks at different input lengths, the model may end up generalizing to the longer length on both tasks.", "questions": "Questions:\n* see Weakness #3\n* In Figure 3c, it’s presupposed that when training Reverse Add up to length 8 on its own, it won’t length-generalize (which is not shown in Figure 3d). Is this the case?\n* The paper shows transfer for related tasks, but not lack of transfer in the case of unrelated tasks. Have the authors tried to pair up unrelated or less-related tasks to show that indeed “related” tasks (line 6) are needed for transfer to happen?\n\nMinor:\n* line 245: Figure 10 → Figure 9\n* line 180, “or” → “For”", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n* 1 Addresses an important challenge facing transformers\n* 2 Provides clear experimental promise of the proposed approach on algorithmic tasks\n* 3 Provides diverse experimental results: algorithmic tasks, benefit from pretraining, mechanistic analysis\n* 4 The paper is easy to read\n\nWeaknesses:\n* 1 The paper argues that length generalization transfer happens for “related” tasks (e.g., line 6), but it remains unclear what kind of relatedness here is needed. There is no theoretical (or even principled conceptual) understanding of how one would select a useful auxiliary task. The selection of auxiliary tasks is heuristic and exploratory. Hence, the paper provides no deeper insights for guiding the future design of auxiliary tasks. [UPDATE: the authors have some initial ideas based on MI, though no deeper principled insights are promised]\n* 2 I found Figure 9 unconvincing. The caption claims that “All three metrics closely align” but other than maxima at 2000 and 16000 steps, I found little obvious alignment between the metrics. Also, (even if the alignment between the metrics were more obvious), the conclusion that “successful length generalization transfer accompanies shared attention mechanisms between tasks” is just based on correlational evidence [there is no causal evidence that attention map/ablation map similarity is causal to the generalization], and should be qualified accordingly. [UPDATE: the authors promise to address this]\n* 3 In Figures 3 and 4, the training lengths vary (16+32 in Figures 3a,3b,4a,4c; 8+16 in Figure 3c, and similarly variable in Figures 5, 6). What is the motivation for this? Why not report unified results with the same lengths?  [UPDATE: the authors have explained the motivation]", "was_revised": true, "final_justification": "The first weakness is only partly addressed. Theoretical understanding is declared out of scope by the authors. I believe this to limit the overall value of the work.\nThe other two weaknesses have been addressed to my satisfaction.\nOverall, I maintain the \"4\" score.", "rebuttal": "### Thank you for your constructive review highlighting the clarity and importance of our work!\n\nLet us address each of the concerns below:\n\n$$ $$\n### 1. Principled understanding of task \"relatedness\"\n\nIn order to create a prescriptive formalism over the meaning of \"relatedness\", we are exploring the hypothesis that \"tasks with high mutual information is more likely to have generalization transfer\". Please see the details in reply to reviewer **8R8h**, as we dive deeper into this with some new experiment results. \nTo thoroughly formalize and validate this hypothesis may be outside the scope of this work, in which we focus on elucidating the transfer phenomenon with tasks of different difficulty. \n\n$$ $$\n### 2. Correlational evidence in figure 9\n\nWe agree that the evidence in section 6.2 is only correlational and we will update the wording in the paper to reflect this. \n\nFurthermore, we will consider removing the \"attention difference\" metric since it serves the same purpose as \"attention ablation map\" and shows weaker correlation.\n\nFinally, we would like to point to figure 19, which supplements Fig 9 by showing the same analysis for more tasks in the addition family. Across these tasks, the local maxima, or \"bumps\" in the generalization gap (orange), mostly match with the maxima in the \"head ablation map difference\" (green).\n\n$$ $$\n### 3. Motivation for different task lengths\nWe choose different max lengths because the tasks have different character lengths for the same length parameter. For example, the multiplication task is much longer than the addition task, because it is solved using a chain of thought format. Thus we need to adjust the lengths we sample to make training feasible on our hardware. In Table 2, we provide examples of the task input/output format. \nAdditionally in Sec 6.1, we show that the transfer phenomenon depends on the _relative_ lengths between the main and auxiliary tasks, rather than the absolute lengths we choose. \n\n$$ $$\n> In Figure 3c, it’s presupposed that when training Reverse Add up to length 8 on its own, it won’t length-generalize (which is not shown in Figure 3d). Is this the case?\n\nYes, the lack of length generalization when training on single task is true for all lengths. \n\n>The paper shows transfer for related tasks, but not lack of transfer in the case of unrelated tasks. Have the authors tried to pair up unrelated or less-related tasks to show that indeed “related” tasks (line 6) are needed for transfer to happen?\n\nSection 4.4 shows that pairing reverse addition with the unrelated \"copy-first\" task yields no transfer, even though the auxiliary task provides longer inputs. This reinforces that structural relatedness, not mere exposure to long sequences, is necessary for transfer."}]

Author rebuttal:

Meta-review: This paper proposes a mechanism for length generalization in transformers. It works by co-training for length generalization using a sufficiently similar task, such that extrapolation on the original task can take place via association.

Reviewers are (very) positive about this work, highlighting the novelty of the approach relative to prior works that focus on architectures for addressing the length generalization problem, and the overall quality of the paper. Some weaknesses, such as the mechanistic analysis and quantifying the notion of task-relatedness, were only partially addressed in the rebuttal, but are not in the way of publication as is also evident from the reviewer scores. One reviewer, reviewer 8R8h who is recommending weak reject, did not take part in the author-reviewer discussion even after repeated attempts from the AC to contact them. Overall, I agree with the other reviewers that this is a valuable contribution, potentially even worth highlighting at the conference.
Decision: Accept (spotlight)