Learned Meta-Tokens for Language Modeling
Authors: Alok Shah
Khush Gupta
Keshav Ramji
Pratik Chaudhari
Venue: ICLR 2026 Poster (2026)
Keywords: meta-tokens
language models
pre-training
positional encoding
Primary area: foundation or frontier models, including LLMs
TL;DR: We pre-train language models with inserted meta-tokens, demonstrating strong performance and length generalization on synthetic tasks for long-context modeling. We explain these results based on positional encoding and implicit context compression.

Abstract: Transformer-based language models (LMs) notably struggle to reliably capture distant contextual information. This work introduces a novel approach using meta-tokens -- special tokens injected during pre-training -- paired with a dedicated meta-attention mechanism to guide LMs to use these tokens. We pre-train a language model equipped with meta-attention in addition to causal multi-head attention on <100B tokens, achieving strong performance on a suite of synthetic tasks. Our method facilitates length generalization up to 2$\times$ the context window after extension with YaRN. We provide an information-theoretic analysis which reveals that meta-tokens \textit{sharpen} the positional encoding, allowing them to operate as content-based anchors that compress preceding context and “cache” it within the meta-token. We empirically confirm this by visualizing model internals to study the residual stream. Together, our findings demonstrate that meta-tokens and meta-attention provide a simple, data-efficient pre-training method, grounded by new mechanistic insights into their role in enabling length generalization behavior.

Published as a conference paper at ICLR 2026
LEARNED META-TOKENS FOR LANGUAGE MODELING
Alok N. Shah
Khush Gupta'
Keshav Ramji
Pratik Chaudhari'
1 University of Pennsylvania
IBB Research Al
Denotes equal contribution
(a (alokshah, khushg, pratike] @upenn edu, keshav. ramj j @ibm. com
ABSTRACT
Transformer- language models (LMs) notably struggle to reliably capture distant contextual information. This work introduces novel approach using meta- tokens special tokens injected during pre-training paired with a dedicated meta-attention mechanism to guide LMs to these tokens. We pre-train a language model equipped with meta-attention addition to causal multi-head attention <100B tokens, achieving strong performance tasks. Our method facilitates length generalization up to 2x the context window after extension with YaRN. We provide an information-theeretii analysis which reveals that meta-tokens sharpen the positional encoding, allowing them to operate as content-based anchors that compress preceding context and "cache" it within the meta-token. We empirically confirm this by visualizing model internals to study the residual stream. Together, our findings demonstrate that meta-tokens and meta-attention provide a simple, data-efficient pre-training method, grounded by mechanistic insights into their role in enabling length generalization behavior.
suite of synthetic
INTRODUCTION
Transformer-base based language models (LMs) have showcased remarkable capabilities across diverse language tasks (Brown al., 2020b; Chowdhery et al., 2022; OpenAI, 2023). Nevertheless, such models suffer from an inability to capture dependencies spanning over their entire context window. With growing adoption and ever-expanding demands on the context over which the model can process and reason, it is vital to develop methods that facilitate long-context adaptation and length generalization. Despite numerous architectural remedies, including sparse attention (Beltagy et al., 2020; Zaheer et al., 2020), recurrent blocks (Hutchins et al., 2022), and modified positional encoding (Press et al., 2021; Su et al., 2021; ;; Chen et al., 2023), the fundamental challenge still remains: how can models reliably access and summarize distant context in concise, cheap, yet expressive manner?
We propose a simple solution, by way of meta-tokens, learned tokens periodically injected into the input sequence during pretraining, and cleverly placed during fine-tuming Unlike conventional dummy tokens (Goyal et al., 2024), meta-tokens are explicitly trained via a dedicated sparse attention layer, guiding the model to condense and "cache" contextual information mechanism. As a result, these tokens act as adaptive content-based anchors, summarizing preceding context segments into compact representations. pathways to distant information, enabling models to generalize effectively across sequences longer than those encountered during training.
in-line storage
inference time, meta-tokens provide implicit
We demonstrate the empirical efficacy of this approach by pre-training a 152M parameter modified GPT-2 model with meta-tokens and sparsely activated meta-attention mechanism. Our approach not only excels on recall-oriented synthetic tasks but also generalizes up to 2x the context window (including after extension via YaRN) - a rare feat for decoder-only architectures trained on 100B tokens or less. We trace these gains to subtle mechanism: meta-tokens provably induce a sharpening effect on positional encoding, enabling the meta-token to locate its position based on the content stores and reducing the entropy of the attention distribution. We present theoretical and empirical evidence that this sharpening is responsible for the ability to retrieve information from relevant distant tokens, facilitating robust length generalization. Furthermore, by analyzing internal model activations and studying the rate- rate-distortion tradeoff, we validate that meta-tokens function as compressed representations of context.

---

Published as a conference paper at ICLR 2026
Our contributions can be summarized as follows:
1. We introduce simple language model pre-training scheme using meta-tokens and a meta- attention mechanism to improve performance on a wide range of synthetic tasks.
2. We show that meta-tokens sharpen the positional encoding, enabling precise long-range attention; we further show that length generalization improves withoul positional encoding.
3. The sharpening hypothesis and implicit compression behavior are supported by visualiza- tions of model internals and cnermatiionthhoretee analysis into the rate-distortion tradeoff.
PRELIMINARIES
Causal Multi-Head Attention. Let {II, (2...2...... denote an input sequence of tokens of length T, V denote the vocabulary size of V, and E' function mapping each token to d-dimensional vector. Each xy is embedded into some continuous representation where = Ex+,) + Pt. such that p: is the positional encoding for t.
IR represent the the token embedding
In decoder-only architecture, we utilize causal self eeff-attention ensure that predictions for given token are only based on preceding tokens. The causal self-attention mechanism modifies the attention computation by masking future positions in the attention weights. Formally:
[QK
Causal Attention Q1 K, V) = -softmax
+ M
dd
where M masks future tokens, ensuring that the model can only attend current and past tokens.
This masking zeros attention scores for future tokens, allowing only the relevant past tokens to influence the current token's 's representation.
Positional Encoding. Positional encoding was introduced in Transformer pre-training provide models with information about the ordering of tokens. With absolute positional embeddings (APE; Vaswani et al. (2017)), each position t in the sequence receives vector Pt, independent of its content, so tokens are distinguished in an mndex-by-inded manner. Given learned token-embbe embedddin lookup table E V Rdd for vocabulary V and hidden dimension d, and positional embedding Emb (t) for E [o. T-1 1] and Emb E RTxxd Each token embedding is then defined as e, F(.,)) + P&; this method was used in GPT-2 and GPT-3 (Radford al., 2019; Brown al., 2020a).
By contrast, Rotary Position Embedding (RoPE; Su et al. (2023)) rotates each pair of embedding dimensions by an angle proportional to position, rather than adding separate vector per position. This makes the difference in attention scores directly encode relative distance between embeddings. The hidden vector h is split into 5 contiguous 2-D slices, and the angle for a position t is defined as 0,,-
The 2-D rotation matrix is taken R(0)
cos 0 sin 0
cos Then, RoPB(h), (24:2+11)
(2
10000477
sin 0
R(6, (2222+4)) This has proven successful in the Llama models (Grattafiori et al., 2024).
TRAINING LANGUAGE MODELS WITH
We introduce set of M meta-tokens (denoted m); given context length block size of the model, n, we take = kn for some constant fraction E [0, 1].. The aim of introducing these meta- tokens is to capture or store contextual information to enhance the model's 's retrieval and reasoning capabilities; attending to a meta-token should enable implicit retrieval of the context that it stores, guiding shortcut paths over the context window.
The M tokens are injected into the input sequences during pre- treitrannng uniformly at random, which was informed by two key premises. While we desire interpretability and control in applying these tokens, and as result, prefer distinguishability at the task level, this is challenging to do without
We take k = 0.1 in practice; halancing next-token prediction over the standard vocabulary while injecting non-trivial numher of meta-tokens.
2

---

Published as a conference paper at ICLR 2026
explicitly fixing downstream task, impeding generality. The second consideration was in how they specifically they should be injected. While Zelikman et al. (2024) introduced <lstartofthought> and <lendofthought>> tokens interleaved between reasoning steps near punctuation (serving as natural break), the introduction of a rough periodicity between tokens during pre-training could result in being trapped into local minima in the optimization landscape. We instead chose follow the random injection scheme, supported by the pre-training approach outlined in Goyal et al. (2024).
We ensure that the trained model incurs no loss for predicting meta-tokens, unlike a standard token in the vocabulary - the meta-tokens' indices are simply shifted and removed when computing the binary cross-entropy (BCE) loss.
Meta-A Attention Mechanism. We augment our transformer to take which contains the posi- tions of the meta-tokens. We introduce sparse attention mechanism, called meta-attention, which selectively modifies attention scores for the specially marked "meta-tokens" within a sequence. This allows the model to simulate selective attention, influencing the final behavior by focusing these meta-tokens. The underlying principles of the desired behavior is influenced by dual cross-attention (Jiang al., 2024), such that operations are performed higher the abstraction hierarchy than the feature space alone. This induces a veeta-earning-liie -ike setup over which attention on the meta-tokens is learned.
Let the indices of meta-tokens be positions E NBxT", where T is the number of meta-tokens per batch. We then build meta mask P E T BxTxT to shape attention. For each batch b and token pair i, j:
P = 0
if both i and j are meta tokens (i.e., i.j E positions [b,
otherwise
The meta-attention operation defined as:
(( [QKT VTL
MetaAtttention(( K, V) softmax
+ M
+ P V
Where M is the same causal mask before. Here, the mask P allows attention flow only among the meta tokens in the sequence, introducing a distinct interaction compared to regular attention. This meta-attention layer selectively modifies the attention by influencing the flow of information to and from these meta tokens, distinguishing itself from the standard causal attention.
To assemble the architecture used for our model, we insert the meta-attention mechanism after the causal masked self-attention computation, to specifically attend to the injected meta tokens, as defined above. We provide a complete breakdown of the architecture in Appendix A.
4 RESULTS
4.1 MODEL TRAINING AND ARCHITECTURE
All experiments were performed with 4 NVIDIA A100 GPUs, training the meta attention transformer on 98B tokens using Distributed Data Parallel (DDP) on the Colossal Cleaned Crawl Corpus (C4) (Raffel et al., 2020). The configuration and hyperparameters used in our pre-training are included in Appendix and B. baseline, we also pre-train GPT-2 (124M) on C4, with identical hyperparam- eters. The primary change make from standard GPT-2 architecture is the addition of RoPE to enable better generalization to longer contexts and improve stability in next-token prediction tasks.
We extend our transformer model's context window from 1024 tokens to longer sequences by training two distinct models with context lengths of 4096 and 8192 tokens, respectively. This extension is implemented using the YaRN method (Peng et al., 2024), which dynamically scales Rotary Positional Embeddings (RoPE) to effectively process significantly longer sequences without compromising performance or computational efficiency. The key parameters detailed in Appendix C.
3

---

Published as a conference paper at ICLR 2026
Anmumay at Langth
Ancunncc at Lengtti
Sugment Counting
Frots Lacag
TIME Lacglt
Aocunny (50 Phat Langth 13.
202
68
25.
26.5
12.3
MAA ATE
GPE- 1253
3ina AZB
Neb Bath
Accurscy Longth Partty
Accuracy Leagth Copytigg
Leaçia
Inc-
Lecuet.
Aonnrary (44. thont eergh -115
X22
302
64
20.5
74
OPP2MC
14l
Mosa 16Z2
o9thler ECOM
Mele
MHA Fee
OPT
Figure 1: We study the performance of the pre-trained GPT-2 w/ APE, Meta-attention w/ APE, and Meta-attention w/ RoPE, as well as GPT-Nco- 125M, all fine-tuned on synthetic data for their respective tasks at the maximum train lengths indicated in the legends. All experiments performed on a test set of prompt lengths up to 512 tokens.
4.2 EXPERIMENTAL SETUP AND TASKS
We designed four synthetic tasks (List Recall, Segment Counting, Parity, and Copying) to probe the recall abilities of models trained with meta-tokens. Each task targets distinct aspect of sequence memory and offered three difficulty levels determined by maximum sequence length. To guide meta-attention, insert designated PAUSE_ meta-token task-specific positions. We fine- tune models on synthetic data generated for these tasks (binned by instance length) and evaluate performance held-out set. Examples of task instances provided in Appendix H.
List Recall: Given lists of length k, the model recalls a specified item. A PAUSE token follows the target list and precedes the question. Difficulty varies with N and k.
Segment Counting: Lists contain segment bounded by PAUSE_ tokens. The prompt asks how often target item appears within that segment. Difficulty scales with list number and size.
Parity: bit sequence contains a PAUSEASSE_EE token marking a position. The model computes the XOR of preceding bits. Difficulty depends on sequence length.
text span bracketed by _AUSS_ tokens. The model must reproduce the span exactly. Difficulty grows with span length and complexity. Performance is measured by sequence accuracy.
Copying:
Within these tasks, we study length generalization by lnn-ttnnig in multiple phases. At each phase, we evaluate performance on sequences longer than those seen training. Appendix reports results at 2048 tokens, twice the pretraining length of 1024.
Baselines. For controlled comparison, also pre-train GPT-2 model (NanoGPT, 124M; Karpathy (2023)) on C4, with identical hyperparameters as the meta-tokens model. Additionally, we Eleuther Al's GPT--Ne- 125M (Black et al., 2021) another baseline.
4

---

Published as a conference paper at ICLR 2026
Table 1: Token Accuracy (%) on List Recall and Segment Counting across long contexts.
Task
Train/Finetune
2k
3k
4k
5k
6k
7k
8k
10k
12k
14k
16k
4k/ 2k
19.5
16.0
13.7
0.9
0.0
0.0
0.9
1.1
0.0
2.1
1.1
4k/ 4k
85.0
88.2
90.2
20.5
1.8
1.0
3.5
4.4
1.1
2.1
2.1
List
8k 4k
85.0
95.8
91.2
97.4
98.2
96.2
93.9
31.9
0.0
2.1
2.1
8k/ / 8k
92.9
98.3
97.1
100.0
98.2
100.0
100.0
89.0
26.1
10.4
9.6
4k 2k
19.1
23.8
19.2
14.6
25.2
14.1
14.0
12.0
16.0
8.0
6.0
4k/ 4k
17.5
23.8
31.8
20.3
30.4
19.3
19.1
14.0
26.0
12.0
16.0
Count 8k 4k
19.1
23.8
14.3
11.1
20.6
12.7
12.7
14.0
16.0
14.0
12.0
8k/ 8k
27.0
33.3
15.9
19.1
27.0
19.1
23.8
22.0
18.0
18.0
18.0
Table 2:
The configurations where zeroing the positional encoding at inference time results in accuracy improvements the List Pointer task, denoted by the ((p) percentage points column.
Model (Split, Train Len)
Full
No Pos
App)
Meta + APE (medium, 128)
77.8%
88. 88.9%
+11.1
Meta APE (hard, 128)
11.1%
22.2%
+11.1
Meta + APE (extra-hard, 512)
11.1% 1%
50.0%
+38.9
Meta RoPE (medium, 128)
44.4%
55.6%
+11.1
Meta + RoPE (hard, 256)
33.3%
66.7%
+33.3
Meta RoPE (extra-hard, 256)
0.0%
22.2%
+22.2
Meta + RoPE (extra-hard, 512)
44.4%
55.6%
+11.1
4.3 META-TOKENS IMPROVE PERFORMANCE ON SYNTHETIC RECALL-ORIENTED TASKS.
As seen in Figure 1, we find that the models trained on meta- tokens substantially outperform our pre-trained GPT-2 and GPT-Neo- 125M baselines, across all tasks and all train lengths. The complete tables for these results are included in Appendix F.
Our models outperform the GPT-Neo- 12-NNo0-15M and GPT-2 with APE models by substantial margin; given that GPT-Neo was pre-trained on 300B tokens, nearly three times the volume of data on which our meta-attention models were trained (albeit from a different corpus), highlighting the data-efficiency of our meta-tokens models. The models also gain in performance much more quickly with fine-tuning when increasing the train length a phenomenon not observed with the GPT-2 models.
In addition synthetic recall tasks, we evaluate our approach on the PG19 long-fore language modeling benchmark. Training meta-attention RoPE model on 6B tokens from PG19 yields lower perplexity than both matched GPT-2 baseline and an in-line retrieval long-context method, Landmark Attention (Mohtashami and Jaggi, 2023a) demonstrating that these gains transfer beyond synthetic settings. (Table 3)
To study the effect of positional encoding our results, ablate by zeroing out the positional encoding, zeroing out the text embedding, and performing both operations all solely at the meta- token indices. Curiously, nearly matches or exceeds the accuracy of the model with the positional encoding as is. The lone exception is the segment counting task, where there is gap for all settings except the model trained with APE at length of 256, which achieves +4.8% improvement over the "Full" model. By contrast, zeroing out the token embedding hurts performance substantially in nearly every setting on List Recall, Segment Counting, and Copying; on Parity, this generally matches the performance of zeroing out the positional encoding. Thus, we find that pre-traine trainininng with teta-tokens and meta-atee aecaa-ttention boosts performance, and 2. zeroing out the positional encoding at just the meta-tokens can match or improve performance inference time.
observe in Tables 11-14 that the score without positional encoding
ToctaTTokes Aid in Length Generalization. In Figure and Appendix F, we find that the model trained meta-tokens length generalizes well the parity and copying tasks with APE, and
5

---

Published as a conference paper at ICLR 2026
Table Perplexity on the PG19 dataset. Meta-Attention and GPT-2 are trained on 6B tokens from PG19 and Landmark Attention was trained on -15B tokens.
Model
PG19 PPL
GPT-2 (124M)
16.13
Landmark Attention
16.23
Meta-Attention RoPE (Ours)
14.79
performs somewhat well (much better than the baselines) on list recall and segment counting at train length of 256. For instance, despite relatively similar performance the 128 train length on the segment counting task, the performance on the test set of up to length of 512 dramatically increases when training the 256 length, by +28.6% with APE and +10.7% with RoPE, compared to +3.5% for GPT-2 with APE. Table exhibits similar trend for the YaRN models, achieving strong performance across its respective context windows, and even achieves non-trivial accuracy beyond the window. Fine-tuning the 8k YaRN model on examples of up length of 4k can generalize very well up to 8k. These findings underscore the substantial advantages of training with meta-tokens and the nuanced role positional encoding plays in task specific and length generalization contexts.
Moreover, when looking at the results on Meta + RoPE on test set lengths of prompts up to 1024 tokens (denoted extra-hard in Table 2), we find that zeroing out the positional encoding also plays sizable role in improving length generalization, especially in the List Recall task. While the model originally achieves performances of 11.1%. 0% and 44.4% when fine-tuned on train lengths of 512 (APE), 256 and 512 (RoPE), respectively, the scores improve by +38.9%, +22.2% and +11.2%, by simply zeroing out the positional encoding the meta- teke-ttokens
If meta-token retains its PE. a portion of its representational capacity is spent encoding position rather than semantic content. This mndex-deendeent signal introduces unnecessary variance, increasing the distortion of the compressed summary. By contrast, zeroing out the PE forces the full embedding capacity to encode task task-relevant information. As a result, we observe lower distortion (higher retrieval accuracy) at given rate both theoretically and ampirically-cceet across all four synthetic tasks.
WHAT MAKES
USEFUL?
The results in Table 2 suggest that the positional encoding of the meta-token can potentially be holding back the downstream performance of the meta-attention models. We posit that the model is instead relying on its content - cached context stored within the meta-token to sharpen its sense of its position in the sequence.
Next, we aim to formally define this notion of sharpness in the context of positional encoding, and its relationship the model's logits. Let a=>b = softmax; (Q.KT + be-)) be the attention distribution for query over keys j, with relative bias term b,-j. We define the sharpness of the positional encoding by the entropy:
H(ss) = - E a logas
Intuitively, when meta-token is present at position f. the model's attention becomes peaked around small set of keys; this "honing in" behavior reduces (a)) compared to APE or RoPE without meta-tokens. In this manner, meta-tokens behave as content-driven landmarks they serve as a low-entropy channel that serves as a pointer to relevant context. As noted prior, the data efficiency observation suggests that the meta-token helps to accelerate next-token prediction behavior while introducing stabilizing effect in the midst of noisy positional encoding.
N. Let ag (j) Theorem 5.1. Consider Transformer head at query position i over keys I,...,N.
exp(Q,K?)) be the attention under absolute positional encoding and let aqiete x exp(Q.K, 5.. A) when a mela-token at position j" introduces additive logit boost of A > O. Then, for some function ((A)= > 0:
H H(o metu
H(a k(A)
(1)

---

Published as a conference paper at ICLR 2026
(Hets Non-Hetts)
(Nen-Hets- Mara]
Neta Taken #2 Past Cosine Sim (Padded))
0.-
07
06
Layyer
JUTTM ago
Taken Nets Iolen
let distenc))
Figure 2: (Left) We analyze the change in logits at the meta-token position, and observe that the meta-tokens do indeed induce a sizable boost logits compare to zeroing token embedding. (Middle) We find the boost in logits to correspond with meaningful reduction in Shannon entropy over the softmax of the logits between the zeroed meta-token sequence and the sequence with the meta-token is. This corroborates with assumptions and claims in Theorem 5.1. (Right) We study the implicit "caching" ability of the meta-token by studying the cosine similarity over the token embeddings. We observe high spikes (the yellow column), diminishing This substantiates our claims of the presence of an implicit compression and "caching" mechanism.
move further away.
Sketch. Sketch. Parametrize the path by [0, AJ. and define logits eg) and their softmax (!) respectively. Since boosting the true index tightens the margin, the derivative of H(a strictly negative. Therefore, over the path, H(aA)) < H(a()),, so H mamel) < H (aath), where their difference must be function of A. The full proof is included in Appendix G.
We note that this theorem also applies RoPE, using a (j) exp (ROPE(K)))) A natural consequence of Theorem 5.1 is that the meta-token operates as an "anchor" from logits standpoint by creating a margin that concentrates the softmax. Concretely, we can specify that for meta-token at position and embedding E Rd. and query at position i with vector Q,, has contribution the (i.j) logit of A - Qi We, Lj=r for learned linear head W. Summing over yields the bias matrix B E Bmota the set of all realizable bias matrices under the meta-token embeddings. Thus, any learned meta-toker embedding- provided that adds to the logits at the summary position j guarantees sharper attention by reducing that attention head's entropy.
In Figure 2, analyze the logits, comparing two settings: (1.) the current meta-token and (2.) the meta-token with its token embedding zeroed out. We find that the former gains sizable amount over the latter, reinforcing the assumption made Theorem 5.1 that the meta-token introduces an additive logit boost of A > O. Our empirical results show that the entropy over the softmax distribution of the logits decreases (the difference between "non-meta-token " and "meta-token" is positive), thus corroborating our central claim in Theorem 5.1.
5.1 A RATE-DISTORRTON PERSPECTIVE ON CONTEXT COMPRESSION
Given that these results provide evidence that meta-tokens can compress context in their representation, we develop mathematical formalizations to analyze this behavior. In particular, we turn to information- theoretic tools specifically, information bottleneck view.
For meta- token 2. succeeding a sequence tokens Zim-1 from indices i m we consider a compression function <() which transforms the subsequence X into As such, we define X {(X)= (( be the compressed representation stored in Fm-- This can be generalized to the full set of M meta-tokens:
NI.M- (X C2(Xm, my +1an -1m SMr (mas+1 : m.)]
For practicality, consider the variational information bottleneck (Alemi et al., 2017). This introduces an encoder qs x) and decoder ( é), along with a simple prior r(ě) (e.g. N 1)), yielding the following form to solve for these variational distributions:
7

---

Published as a conference paper at ICLR 2026
Diee-iiittttio::: Meta-Token
Last token
Meta token
Distortion (Cross (Cross-Entro) Entropy LosS)
10.6
10.4
10.2
10.0
40
50.55
70
100 200 400
Figure 3: (Left) This plot shows the residual stream each layer, highlighting the meta-token's 's role in causal attention. The colors before the meta-token (colored band across layers) indicate the context attends to/stores. The rightmost line shows the final meta-token, which attends to the previous one at the band. (Right) We analyze the variational information bottleneck (VIB) objective and its decomposition into rate and distortion components. Supporting Theorem 5.1, for given rate R, the distortion D is strictly lower for the meta-tokes tekan compared to the last non-meta- meta-token in the sequence.
min
E
E log (y []] + 3. E [KL(qa(# =)]lr())]]
p(a.3) (8(2))
r(a))
This form admits an equivalent perspective rate-distortion theory. Specifically, the first term measures the quality predicting the downstream target given a lossy compression X ("distortion"). The second term measures the average number of bits required to encode X, relative to some simple reference code r(z) ("rate"). As such, analyzing rate-distortion curves sweeping values of can provide valuable insights into the quality of the "compression" behavior and its informativeness when the meta-token is attended
Theorem 5.2. Let Dobs (R) be the minimum distortion achievable at rate R under the VIB objective only using absolute positional encoding (no meta-tokee tokkns), and Dmeaal R)) be the minimum distortion achievable at rate R with meta-tokens. Then, for every R 2 O,
Dineaa R) Date (R)
Intuitively, meta-tokens expand the feasible of encoders and decoders, which will either match lower distortion for given rate. Thus, the quality of compression with respect to its informativeness in predicting the target can only improve. In Appendix G.4 we provide further intuition how meta-tokens facilitate context caching by showing that they act as inducing points in a Nyström approximation of the softmax kernel. This connects directly to our rate-distortion analysis, as bound the prediction error (distortion) by the quality of kernel approximation.
5.2 DITE-DIITORTION INFORMS THE QUALITY OF CONTEXT CACHING
To obtain empirical rate-distortion curves for our meta-token bottleneck Figure 3, we freeze the pre-trained meta-token model and fix small variational bottleneck head to the last meta-token hidden state. Concretely, let hm E R be the output of the final Transformer layer at the last meta-token position. We introduce
g0(2 hm) (Hos)))) dimm())(hm))), go(y) 2) softmax(W:: b),
with Pg. Tb RD R and W E [ We then optimize the ELBO:
min E. (y =)] + Es... [KL(go(2 hm) (0, T))]
8

---

Published as a conference paper at ICLR 2026
Training is performed on the small List-Pointer D.1.1 split (50 examples, batch size 1), for 5 epochs at each 3 E {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0]. After each run, we record the average cross- entropy loss ("distortion") and KL ("rate"") on the same 50 examples. Finally, we plot the resulting rate-distortion curves on s sym og x-axis (linear below 20 nats, logarithmic above) so that both the low-rate "knee" and the long tail are visible (see Figure 3).
6 RELATED WORK
Pause and Memory Tokens As detailed in our work, recent studies on Transformer-base models have explored the introduction of special tokens, beyond ordinary vocabulary symbols. Pause or dummy tokens as introduced in Goyal et al. (2024) enhance computational width, allowing models to perform additional internal computation by effectively delaying their outputs, yielding empirical maasoning-intensvve intensive tasks. Similarly, Pfau et al. (2024) explore using filler tokens- - sequences of seemingly meaningless symbols - as a stand-in for chain-of-thoupe Recent works such as Merrill and Sabharwal (2025) and London and Kanade (2025) demonstrate that the inclusion of polynomial number of pause tokens to constant-depth Transformers increases expressivity to represent the TC class. In contrast, our work introduces tokens that not merely delays or placeholders but are explicitly designed to direct attention and structure computation, allowing the model to recall and generalize over the context length in ways these prior approaches do not address. Works such as Memory Transformer (Burtsev et al., 2021) and Landmark Attention (Mohtashami and Jaggi, 2023b) introduce memory tokens; the former prepends them, while the latter uses them as learnable keys for retrieval over blocks of context. Unlike Memory Transformer's prepended slots or Landmark Attention's 's block-level landmarks, our meta-tokens are dynamically positioned and operate through dedicated meta-attention layer rather than fixed structural scaffolds.
gains question answering and reasoning-
Positional Encoding We have already described absolute positional embeddings (APE), rotary positional embeddings (RoPH) and relative bias in Section 2. In addition to these methods, ALiBi (Press et al., 2022) adds a fixed linear penalty to attention scores based on the distance between query and key positions, favoring nearer tokens and generalizing to longer contexts with minimal loss in perplexity. Recent work has suggested that Transformers without any added position embeddings can still learn order information and, in some cases, generalize to longer sequences better than models with standard positional encoding. NoPE (Kazemnejad et al., 2023) showed that models trained without positional embeddings can achieve strong length extrapolation in comparison to models trained with positional encoding. They can internally represent both absolute and relative PEs without any explicit positional signal, suggesting these may emerge implicitly via training dynamics over the data distribution. NoPos (Haviv et al., 2022) also found a similar result, suggesting that models trained without PE can infer their absolute position due to causal attention masks. These findings highly relevant to our work, given our evidence on length generalization behavior whiling zeroing the positional encoding at the meta-tokens.
7 DISCUSSION AND LIMITATIONS
Our findings suggest that decoder-only language models trained with meta-tokens and meta-attention achieve strong performance on recall-oriented tasks. Furthermore, they are able to length generalize, with performance improvements when removing the effect of positional encoding at the meta-tokens Given the prior findings of NoPos, we believe the introduction of the meta-attention mechanism and second causal mask (the "meta mask") could be responsible for this behavior, provided that this behavior is specific to the meta-tokens. We suggest that hybrid attention methods such as RNoPE (Yang et al., 2025) could be suitable for facilitating long-context modeling with meta-tokens.
Given the findings that the meta-tokens operate like anchors within the context, would be valuable to explore the impact of our proposed mechanism in pre-training larger models over longer context windows, under greater computational resources. We employ synthetic tasks that are well-aligned to recall abilities, and design experiments to test length generalization, with the aim of strong synergy with long-context modeling capabilities. Nonetheless, training larger models would indicate the viability of approach for real-world deployment.

---

Published as a conference paper at ICLR 2026
Notably, method requires little overhead - the addition of meta-tokens is simple data augmen- tation strategy, and the meta-attention layer is added after standard causal masked self-attention, described in Appendix A. It would also be informative study larger-scale corpora given the data-efficient nature of the meta-tokens approach in vastly outperforming the vanilla GPT-2 model at the 100B tokens scale, how rapidly does each model saturate designed synthetic tasks?
CONCLUSION
We introduce meta-toker in language model pre-training, addition to a dedicated mechanism which learns the relationship between standard tokens and meta-tokens, We find that this improves performance on suite of synthetic recall tasks, and enables length generalization behavior when removing the positional encoding at each meta-token. We provide evidence suggest that the meta-tokens sharpen the positional encoding, enabling them to operate as content- baaee landmarks in the context; we further show that they implicitly compress preceding context, demonstrated by similar token embeddings. These interesting phenomena demonstrate the promise of long-contex. language modeling enabled via data-efficient pre-training using meta-tokens.
ACKNOWLEDGEMENTS
We thank Surbhi Goel for valuable discussions that shaped the direction of this study and for support with computational resources. We also thank Ben Keigwin for helpful suggestions in designing our analysis. We are grateful to Simran Arora for feedback that led to substantial revisions of earlier drafts. This work was supported in part by research compute grant from Lambda Labs.
REFERENCES
A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2017. URL https:: net/ fortmforu? Hww.mmmmddd-Hxxyooxoemmoo
I. Beltagy, M. E. Peters, and A. Cohan. Longformer: The Long-document transformer. arXiv preprint arXiv:2004.055 051040 2020.
S. Black, G. Leo, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, Mar. 2021. URL https: /ooii org/10...000 zenodo.. 5297715. If you use this software, please cite it using these metadata.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Sterbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, Hesse, M. Chen, F. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. tors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020a. URL https: htttttss://ppooeeeeeeee (10000///1ee11111111 httpe//000..aeeaaooooooo6610011.
D. Ziegler, Wu, C. Winter,
H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, edi-
http:////occc/eeceeeeeeee . nwwrips wtt.urrpppscccppprrrreeeeeee.e pdf.
T. B. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. NeurlPS, 2020b.
M. Burtsev, Y. Kuratov, A. Peganov, and G. V. Sapunov. Memory transformer, 2021. URL https: /arxivvv org/ orggabbse abbs//0ss 11527.
D. Chen et al. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0889)) 2023.
Y. Chen, Q. Zeng, H. Ji, and Y. Yang. Skyformer: remodel self-attention with gaussian kernel and nyström method. In Proceedings of the 35th International Conference Neural Informa- tion Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
10

---

Published as a conference paper at ICLR 2026
A. Chowdhery, S. Narang. J. Devlin, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
T. M. Cover and J. A. Thomas. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley-Interscience, USA, 2006. ISBN 0471241954.
T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://ppereviiiiee net / Eorum 2id= 2d-1ddd3n2nni
S. Goyal, Z. Ji, A. Rawat, A. K. Menon, S. Kumar, and V. Nagarajan. Think before you speak: Training language models with pause tokens. In The Twelfth International Conference Learning Representations, 2024. URL https: hppp/eeeeeeeeeeeeeeeeeeeeeee wwt.:eeooooomiippeoeeeeeeeeeeeee
A. Grattafiori et al. The Ilama 3 herd of models, 2024. URL URL.ntpsss/arrxw...rr/////. 21783.
A. Haviv, O. Ram, O. Press, P. Izsak, and O. Levy. Transformer language models without posi- tional encodings still learn positional information. Y. Goldberg. Z. Kozareva, and Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1382- 1390, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguis- tics. doi: 10.18653/11/002...0 findings-emna 120p:/0000..dnnnnnmem..999999. URL URl.rttpee//aelaaanaooo.oo..oo
99/.
D. Hutchins, I. Schlag, Y. Wu, E. Dyer, and B. Neyshabur. Block-recurrent transformers, 2022. URL https: //axiiv///iiii/////a wwr.vii.oorg..ooooooooooooo org/ abs 2203. 07852.
W. Jiang, J. Zhang, D. Wang, Q. Zhang, Z. Wang, and B. Du Lemevit: Efficient vision transformer with learnable meta tokens for remote sensing image interpretation. In K. Larson, editor, Proceed- ings of the Thirty-Thrr International Joint Conference Artificial Intelligence, IJCAI-24, pages 929-937. International Joint Conferences on Artificial Intelligence Organization, 8 2024. doi: (M.24963/jjaai.2224//00 URL htt.ttts://oi..r//DDD00 Track.
http:/aa...2././// 2024/0222222 /10.. Main
A. Karpathy. nanoGPT. https:///gtthub..cmm/krpttttttt 2025-05-16.
2023. Accessed:
A. Kazemnejad, I. Padhi, K. Natesan, P. Das, and S. Reddy. The impact of positional encoding on length generalization in transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:/ // wttp//peeeeee.eeeee/ee net/ forum? wwt.:oordommmrmaaaaaa id=Drr12gejzl. /
E. Kim, S. Kim, and J. Thome. Learning to insert [pause] tokens for better reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, page 23760-23777. Association for Computational Linguistics, 2025. doi: 10. 2.116533v1/2022 //dx. doi. org.orgg////ooo 186.3865333/
www.:o2s...ininn-ac....... 1217. URL http:
1ww.11/61////000 wtlvv1/2222/ www.iiiiiiiinnnnnnnneeeie acl 1217.
C. London and V. Kanade. Pause tokens strictly increase the expressivity of constant-depth trans- formers, 2025. URL wtt.tttts///ariv..oooo org/ absaabss/550 21024.
A. W. Marshall, I. Olkin, and B. C. Amold. Inequalities: Theory of Majorization and Its Applications. Springer Series in Statistics. Springer New York, New York, NY, edition, 2011. ISBN 978-0- 387-68276-1. doi: 12.10007978-0--87--62777
W. Merrill and A. Sabharwal. Exact expressive power of transformers with padding, 2025. URL org/ alss//b///// 2200..255.111 18948.
A. Mohtashami and M. Jaggi. Landmark attention: Random-access infinite context length for transformers, 2023a. URL
wttp.//g//.........a/// 6300.
A. Mohtashami and M. Jaggi. Random-acces infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL https: www.wwww.wwtwtteeeeeeee // /poperevii net formm d 64w0Vy.

---

Published as a conference paper at ICLR 2026
N. Muennighoff. Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and Hashimoto. sl: Simple test-time scaling, 2025. URL U hos/2501.19393,
OpenAl. Gpt-4 technical report. https://openal.com http://aaa...o///eeerreettoo
2023.
B. Peng, J. Quesnelle, H. Fan, and E. Shippole. YaRN:
Efficient context window extension of large language models. In The Twelfth International Conference Learning Representations, 2024. URL https: http/pppeevviieeeeee net wttt:/ooummiic www.:rmddww:B
Pfau, W. Merrill, and S. R. Bowman. Let's think dot by dot: Hidden computation in trans- former language models. In First Conference Language Modeling, 2024. URL https:
// //openreview net forummid-NikbrdtYYo
O. Press, N. A. Smith, and O. Levy. Alibi: Trainable linear biases for transformers, arXiv preprint Sttt:2208.124009 2021.
O. Press, N. A. Smith, and Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. URL
org. hts//bbs/21000 12099.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and Sutskever. Language mod- els are unsupervised multitask learners. OpenAI Technical Report, 2019. URL fttps://eet //cc openai hawp:/eettrr--nnugeeeee mooeee--ooella0 laslsssllnnggeee models are_unsupervi sed__mp titasktskkkk.
httpr/edeemmttttteeeeeeeeeeeeeee wew.eers..rre...ee.e.eee.ee ffd. Accessed: 2025-05-15.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), Jan. 2020. ISSN 1532-4435.
J. Su, Y. Lu, S. Pan, et al. Roformer: Enhanced transformer with rotary position embedding, In ACL, 2021.
J. Su, Y. Lu, S. Pan, A. Murtadha, B. Wen, and Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL www.aaavvv..oo. org. wtgp/as////aaaaa//a///////// 09864.
A. Vaswani. N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http::://prroceednnggs
ctp.aperccpppp...ppp httpppaaeleeppppppppppeeeeeeepee /ttpe//ppp/pp11////////// Paperr..-Pppr.pdda .df.
Y. Xiong, Z. Zeng, R. Chakraborty, M. Tan, G. Fung. Y. Li, and V. Singh. Nyströmformer: A the AAAI Conference on
nyström- based algorithm for approximating self-attention. Proceedings the Artificial Intelligence, 35(16):14138-141) May 2021. doi: 10.1609/aaai http://aai.v3516........ v35i16.176644 URL petps://oj3. aaai org/ index.nin.. php/ ahpp/AAI///aaaaaaaaaaaaaaa////aa AAAAAAAAaaaataeaaa http///aticeeeee//////666/6
B. Yang, B. Venkitesh, D. Talupuru, H. Lin, D. Cairuz, P. Blunsom, and A. Locatelli. Rope to nope and back again: A new hybrid attention strategy, 2025. URL https http:////wwwww/////eaaa/e
25 18795.
M. Zaheer, G. Guruganesh, A. Dubey, et al. Big bird: Transformers for longer sequences. In NewrIPS. 2020.
E. Zelikman, G. R. Harik, Y. Shao, V. Jayasiri, N. Haber, and N. Goodman. Quiet-STar: Language models can teach themselves to think before speaking. First Conference on Language Modeling. 2024. URL https: openrevieww net forum rum? ? d RRXP 1SOGH9.
12

---

Published as a conference paper at ICLR 2026
A FULL ARCHITECTURE DETAILS
We provide a full outline of the architecture design out method uses. Our architecture is equivalent to the NanoGPT (GPT-2) architecture, while introducing the meta-attention block after the initial causal masked attention and layer normalization computation.
1. Input Layer: Given an input sequence of tokens x {I1, 22, 2T ). we first embed each token into continuous representation. Instead of absolute positional encodings, we apply Rotary Position Embeddings (RoPE) Su et al. (2023) to inject positional information. For each token, the embedded representation is:
e (x)))
where RoPE(., (, f) denotes the rotary positional embedding applied to the tt position, with base 10000.0.
2. Causal Masked Self-Attentio:: The first layer consists of the causal masked self-attention mechanism. For each head h, the attention operation is computed as:
QK dde
CausalAttention, (Q, V) softmax
M) Vn.
where Q, K, V are the query, key, and value matrices derived from the input embeddings E, and M is the mask matrix.
3. Meta Attention Layer: After the causal masked self-attention, we integrate the meta- attention mechanism to specifically attend to the injected meta tokens. This operation is defined as:
QK Vdi
MetaAttentioon(,,KK =sofmma
+ Mcausal + P) V,
where P is the meta mask constructed from the indices of the meta tokens.
4. Feedforward Layer: Following the attention layers, we pass the output through a feedfor- ward neural network defined by:
FTN ReLU(>W, + b,) b2,
where W1, W2 are weight matrices, and b1. by are bias vectors.
5. Layer Normalization: After both the causal self-ateention and meta-attention operations, we apply layer normalization:
1
LayerNorm(s) = σ +
where and a are the mean and standard deviation of the features, and is a small constant for numerical stability.
6. Final Output Layer: The final layer projects the output of the last feedforward layer back to the vocabulary size to produce the for the next token prediction:
$ softmax(:W..
baut);
where Wont and bout are the output weight matrix and bias vector, respectively.
13

---

Published as a conference paper at ICLR 2026
B PRE-TRAINING HYPERPARAMETERS AND MODEL DETAILS
Our decoder-only modified GPT-2 model was pre-trained on the C4 dataset with the following configuration and hyperparameters:
Table 4: Pretraining Configuration Parameters
Parameter
Value
Batch Size
12
Gradient Accumulation Steps
40
Block Size
1024
Number of Layers
12
Number of Heads
12
Embedding Size
768
Learning Rate
6c-4
Weight Decay
le-1
Max Iterations
600,000
Warmup Iterations
2,000
Minimum Learning Rate
6c-5
Dropout Rate
RoPE Theta
10000.0
Initial Model
Resume
Optimizer
AdamW
AdamW Betal
0.90
AdamW Beta2
0.95
Gradient Clipping
1.0
Tokenizer
tktokken
C YARN HYPERPARAMETERS
Parameter
4096-token model
8192-token model
yarn_scall scale
4.0
8.0
cain_original_meee
mnx_seq__een
1024
yarn_ dean_extrapolation.n factor
yarn_alh aarn_att___a_ factor
1.0
yarn_beta_fee
32.0
yarn_ setn__et___slo__
1.0
Table YaRN parameter configurations for extended context models.
D ADDITIONAL EXPERIMENTAL DETAILS
D. SYNTHETIC DATA GENERATION
We generate 90,000 train examples and held-out test set of 10,000 examples for each task.
D.1.1 LIST RECALL
We generate suite of "list-pointer" examples by sampling random categories and list items, inserting a special meta token as a marker, and asking the model to recover the item immediately following the meta-token. Each example consists of:
1. categories drawn without replacement from fixed set of 20.
2. items per category, sampled with replacement from the category's 10-item inventory
14

---

Published as a conference paper at ICLR 2026
3. One "target" category in which we inject single meta token after the jth item (j [rz]) and then append the remaining items
4. A question line "Q: What is item of <target>? _ETA_
This pipeline yields to and copy items in long, multi-list list contexts.
data that systematically probes the model's 's ability to attend
Table Curriculum schedule for synthetic data.
Phase
m (Num. categories)
(List length)
Approx. prompt-token range
Uniform 3-8
Uniform 3-10
Short (== 100-200 tokens)
Uniform 8-12
Uniform 3-16 (bimodal)
Mid-range (+= 200-300 tokens)
Uniform 12-19
Mixture (3-8, 9-16, 17-25)
Full-range (=s 500-700 tokens)
Uniform 15-20
Uniform 40-60
"Extra-hard" 1024 tokens
Uniform 15-20
Uniform 90-110
"Long" - 2048 tokens
D.1.2 SEGMENT COUNTING
Similar to List-pointer, except the model must count occurrences of target token within a meta-token bracketed list segment. Uses the schedule dictated by Table 6. Asks the question: "Q: How many times does <token> appear between the pauses around <Category>? MMEAA"
D.1.3 PARITY
Generates examples where the model computes the XOR (parity) of a bit-string segment to the first L. characters where L drawn phase-dependently. the same scheduling dictated by Table 6. Asks the question: "Q: What is the XOR of all bits before this pause? _META_
D.1.4 COPYING
Generates examples where the model must copy bracketed span from text. Uses schedule dictated by Table 6 and samples an additional copy length C' and distance length / depending on the phase
D.1.5 MULTI-HOP
Generates examples where the model must resolve chain of references across list indices. Each example begins with an input list containing items at numbered positions. The task specifies starting index, and the model must repeatedly follow the indicated pointers. Uses schedule dictated by Table
E LICENSES
NANOGPT
Our implementation of the vanilla GPT-2 is based on the nanoGPT repository (attps://github com/kaarpathy/nancGee which licensed under the MIT License.
ELBUTHERAA 1O9--NE--12MM
We directly use the EleutherAI GPT-Neo 125M model checkpoint and weights, available Hugging Face Model Hub at https: / htt//hhggggggeee http//eetteerr/gggeeeee http:/gptteeo1115551 This model is released under the MIT License.
C4 DATASET
Our model was trained on the C4 dataset (httpps ct..aasets al Allneai /c4), which is provided under the Open Data Commons Attribution License (ODC-BY).
15

---

Published as a conference paper at ICLR 2026
TIKTOKEN
We use the t tiktoken library from OpenAl for tokenization htttt//s//gtthbb..om//opeeeeee tiktoken), which is released under the MIT License.
16

---

Published as a conference paper at ICLR 2026
F COMPLETE EXPERIMENTAL RESULTS
F.1 SYNTHETIC TASK ACCURACIES ACROSS TEST LENGTHS
We RoPE models sequence length of 2048- the pretraining block size of 1024 as relative position embeddings naturally support extrapolation beyond the training context window. In contrast, absolute positional encodings (APE) cannot generalize sequences longer than those seen during pretraining.
Table 7: Accuracy (%) across evaluation lengths for each model train on List Recall
Model (Train Length)
128
256
512
1024
2048
GPT-2 APE (128)
4.2
1.2
0.0
0.0
GPT-2 APE (256)
6.8
2.4
0.0
0.0
GPT-2 APE (512)
19.8
9.5
3.6
0.0
Meta + APE (128)
100.0
86.4
12.0
4.1
Meta APE (256)
100.0
98.6
42.6
3.9
Meta + APE (512)
100.0
100.0
98.7
11.1
Meta RoPE (128)
100.0
60.7
5.9
0.0
0.0
Meta RoPE (256)
100.0
100.0
48.6
23.5
0.0
Meta RoPE (512)
100.0
100.0
99.3
58.9
5.6
GPT-Neo- 25M
85.6
86.0
81.2
Table 8: Accuracy (%) across evaluation lengths for each model trained on Segment Counting, Each model evaluated on longer contexts than seen during training.
Model (Train Length)
128
256
512
1024
2048
GPT-2 APE (128)
32.1
27.4
20.2
0.0
GPT-2 APE (256)
40.3
56.2
23.7
0.0
GPT-2 APE (512)
30.1
32.1
25.0
0.0
Meta + APE (128)
77.4
55.9
25.0
11.1
Meta + APE (256)
83.3
77.4
53.6
22.4
Meta APE (512)
91.7
79.8
80.9
33.3
Meta + RoPE (128)
77.4
64.3
25.0
22.7
0.0
Meta RoPE (256)
64.3
64.3
35.7
33.3
0.0
Meta + RoPE (512)
90.9
91.4
95.3
66.7
11.1
GPT-Neo-112M 125M
31.4
25.9
24.9
Table Accuracy (%) across evaluation lengths for each model train on Parity
Model (Train Length)
128
256
512
1024
2048
GPT-2 APE (128)
75.0
56.0
53.4
45.2
GPT-2 APE (256)
75.0
67.0
60.7
46.2
GPT-2 APE (512)
75.0
54.8
60.0
40.5
Meta + APE (128)
100.0
75.0
67.9
52.4
Meta + APE (256)
100.0
97.6
96.4
69.1
Meta + APE (512)
100.0
100.0
100.0
86.7
Meta + RoPE (128)
100.0
66.7
76.2
59.5
44.1
Meta + RoPE (256)
97.6
100.0
96.4
61.9
52.4
Meta + RoPE (512)
100.0
100.0
100.0
69.1
63.1
GPT-Neo- 125M
80.4
59.1
54.8
17

---

Published as a conference paper at ICLR 2026
Table 10: Accuracy (%) across evaluation lengths for each model trained on Copying
Model (Train Length)
128
256
512
1024
2048
GPT-2 APE (128)
6.0
5.3
3.0
0.0
GPT-2 APE (256)
6,8
6.0
5.7
0.0
GPT-2 APE (512)
3.8
4.8
7.8
0.0
Meta APE (128)
100.0
66.7
76.2
2.6
Meta APE (256)
100.0
100.0
96.4
7.9
Meta APE (512)
100.0
100.0
98.5
87.4
Meta RoPE (128)
96.6
73.0
5.2
0.0
0.0
Meta RoPE (256)
98.2
100.0
23.6
9.3
3.2
Meta RoPE (512)
99.0
98.9
98.9
89.4
11.8
GPT-Neo-1220 125M
31.5
22.7
16.9
F.2 ABLATIONS ON POSITIONAL ENCODING AND TOKEN EMBEDDING
Table 11: Accuracy (%) on the List-Recall task under different ablations: zeroing the positional
encoding (No Pos), zeroing the text embeddings (No Embed), zeroing both of the meta-tokens.
Model (PE)
Full
No Pos
No Embed
Neither
Meta APE (128)
100.0
99.3
17.4
59.7
Meta RoPE (128)
100.0
100.0
32.4
24.0
Meta APE (256)
86.4
86.9
12.2
16.2
Meta RoPE (256)
100.0
100.0
4.0
Meta APE (512)
100.0
100.0
52.1
84.3
Meta RoPE (512)
100.0
100.0
59.6
25.2
Table 12: Accuracy (%) on the Segment Counting task under different ablations: zeroing the positional encoding (No Pos), embeddings (No Embed), or both, only on the meta-token.
Model (Train Length)
Full
No Pos
No Embed
Neither
Meta + APE (128)
77.4
63.1
31.0
47.6
Meta + APE (256)
83.3
88.1
32.1
40.5
Meta + APE (512)
91.7
82.1
34.5
51.2
Meta + RoPE (128)
77.4
70.2
59.5
36.9
Meta + RoPE (256)
64.3
53.6
30.9
30.9
Meta + RoPE (512)
80.9
72.6
36.9
25.0
Table 13: Accuracy (%) the Parity task under different ablations: zeroing the positional encoding (No Pos), text embeddings (No Embed), or both, only on the meta-token.
Model (Train Length)
Full
No Pos
No Embed
Neither
Meta + APE (128)
100.0
100.0
100.0
100.0
Meta APE (256)
75.0
77.4
77.4
79.8
Meta + APE (512)
67.9
71.4
72.6
66.7
Meta RoPE (128)
100.0
97.6
100.0
100.0
Meta + RoPE (256)
66.7
66.7
73.8
66.7
Meta + RoPE (512)
76.2
75.0
75.0
64.3
18

---

Published as a conference paper at ICLR 2026
Table 14: Accuracy (%) on the Copying task under different ablations: zeroing the positional encoding
(No Pos), text embeddings (No Embed), or both, only on the meta-token.
Model (Train Length)
Full
No Pos
No Embed
Neither
Meta + APE (128)
96.6
93.2
7.2
4.8
Meta APE (256)
98.2
99.6
5.0
3.6
Meta + APE (512)
99.0
96.6
5.7
5,4
Meta + RoPE (128)
100.0
99.6
6.9
4.9
Meta RoPE (256)
100.0
100.0
4.5
5.1
Meta + RoPE (512)
100.0
95.6
4.9
F.3 POSITIONAL ENCODING ROBUSTNESS ABLATIONS
Table 15: Accuracy (%) on the List Pointer task with Gaussian noise added to positional encoding.
Model (Train Length)
Noise 0.0
Noise 0.1
Noise
Noise 1.0
Noise 2.0
GPT-2 + APE (128)
4.8
1.2
2.4
2.6
3.5
GPT-2 + APE (256)
17.4
11.9
4.6
3.6
3.2
GPT-2 + APE (512)
14.0
16.3
16.7
17.9
14.3
Meta + APE (128)
98.7
98.6
67.5
55.6
42.8
Meta + APE (256)
81.8
79.7
48.9
43.1
37.9
Meta APE (512)
100.0
100.0
79.5
65.5
57.1
Meta RoPE (128)
98.1
100.0
100.0
96.0
88.9
Meta + RoPE (256)
100.0
100.0
100.0
97.9
82.6
Meta RoPE (512)
100.0
100.0
100.0
98.8
81.0
Table 16: Accuracy (%) the Copying task with Gaussian noise added positional encoding.
Model (Train Length)
Noise 0.0
Noise 0.1
Noise 0.5
Noise 1.0
Noise 2.0
GPT-2 Abs (128)
2.9
1.2
0.0
0.0
0.0
GPT-2 Abs (256)
6.0
7.1
3,6
0.8
0.7
GPT-2 Abs (512)
6.0
5.8
3.6
0.4
0.3
Meta + APE (128)
96.1
98.5
69.8
58.6
54.9
Meta APE (256)
100.0
100.0
76.3
68.8
57.2
Meta + APE (512)
98.9
98.7
74.4
68.9
50.5
Meta RoPE (128)
100.0
100.0
75.9
68.6
49.9
Meta + RoPE (256)
100.0
100.0
82.6
65.6
45.1
Meta + RoPE (512)
100.0
100.0
84.4
67.6
46.3
19

---

Published as a conference paper at ICLR 2026
F.4 LENGTH GENERALIZATION ABILITY UNDER No POSITIONAL ENCODING ABLATION
Table 17: List-Recall: "No Pos" vs. Full accuracy for Meta-attention with APE and Meta-attention with RoPE.
Model (Split, Train Len)
Full
No Pos
! (pp)
Meta APE (128)
small
medium
77.8%
88.9%
+11.1
hard
11.1%
22.2%
+11.1
Meta + APE (256)
small
100.0%
100.0%
0.0
medium
100.0%
100.0%
0.0
hard
44.4%
22.2%
-22.2
Meta + APE (512)
small
medium
hard
100.0%
100.0%
0.0
Meta RoPE (128)
small
medium
44.4%
55.6%
+11.1
hard
11.1%
11.1%
0.0
extra-hard
0.0%
0.0%
0.0
long
0.0%
11.1%
+11.1
Meta RoPE (256)
small
100.0%
100.0%
0.0
medium
100.0%
100.0%
0.0%
hard
33.3%
66.7%
+33.3
extra-hard
0.0%
22.2%
+22.2
long
0.0
0.0
0.0
Meta RoPE (512)
small
medium hard
100.0%
100.0%
0.0
100.0%
100.0%
0.0
extra-hard
44.4%
55.6%
+11.1
long
0.0%
0.0%
0.0
F.5 MULTI-HOP RETRIEVAL RESULTS
Table 18: Accuracy (%) across evaluation lengths for each model trained on Multi-Hop Retrieval
Model (Train Length)
128
256
512
1024
GPT-2 APE (128)
27.5
4.7
2.2
3.6
GPT-2 APE (256)
32.0
31.3
18.3
6,4
GPT-2 APE (512)
38.3
31.7
21.4
2.1
Meta + APE (128)
27.7
3.6
4.5
2.9
Meta APE (256)
29.1
26.4
23.0
15.4
Meta + APE (512)
41.0
33.0
34.0
15.9
Meta RoPE (128)
78.8
66.0
51.1
18.4
Meta + RoPE (256)
74.2
84.1
59.7
39.4
Meta RoPE (512)
85.0
72.0
72.0
62.0
20

---

Published as a conference paper at ICLR 2026
F.6 RANDOM PLACEMENT TOKENS INFERENCE
Since our list-recall results succeed the answer, we also explore variant with random placement to ensure that meta-tokens and meta-attention actually help. Uniform placement is explored in D.1.5
Model (Train Length)
128
256
512
GPT-2 APE (128)
1.2%
0.0%
0.0%
GPT-2 APE (256)
1.7%
1.8%
0.0%
GPT-2 APE (512)
1.7%
1.9%
1.5%
Meta APE (128)
12.4%
9.9%
6.7%
Meta APE (256)
16.8%
18.2%
8.6%
Meta + APE (512)
17.4%
18.6%
15.5%
Table 19: Performance of GPT-2 APE and Meta + APE models across different train lengths.
F.7 INFERENCE EFFICIENCY
Meta-tokens are generated inference. The additional computation is sparse each attention head only considers small number of meta positions rather than the full attention matrix. In our current PyTorch implementation, which materializes the sparse mask as a dense tensor, we observe throughput drop from 130.82 to 117.86 tokens/sec and TTFT increase from 7.44ms to 7.57ms, i.e., a 1.1 slowdown. We expect optimized sparse attention implementations to reduce or eliminate this overhead.
Table 20: Inference speed comparison with and without meta/pause tokens.
Metric
No meta/pause tokens
With meta/pause tokens
TPS (tokens/sec)
130.82
17.86
TTFT (ms)
7.44
7.57
Slowdown factor
1.00
1.11
G THEORETICAL ANALYSIS
G.1 PROOF OF THEOREM 4.1
Lemma G.I. Let l la....,.e. eN be logits and define softmax distribution oj ΣN exp(66)) Suppose that for some "correct" index jr we have E5- L. and for all other indices 4 lj 5 L for some A 0. Then, entropy H H(x) is strictly decreusing in
expl(,) )
Proof. First,
jr, such that S = E Eb, Then, since each l group the other logits (i.e. j H
j41"
carries the property that e S e given Ej- = L. we have that S (N - 1)el- since there are N terms. Revisiting the softmax a. we have that a,- 2 -f- -)e--x We will denote this quantity as henceforth. Next, each other softmax aj for j + j must have the property that a, o - (N 1+(--1e-- 1-g
-
As result, have the following entropy maximization problem:
maximize a
aj log aj
N
subject to
Σ j=1
aj 1,
0tj* p-
0₁ 2 0,
j = I,..,,N.
N.
21

---

Published as a conference paper at ICLR 2026
Observe that the entropy (objective) function is Schur-concave in a, so it maximized when ayr and the remaining softmax mass is split uniformly over the N- 1 elements, i.e. 0j 1- V Plugging this in for H (a) yields:
p
#
(aa) f -plog p (1- p) log(1 - p) + (1 - p) log(N - -1)
(3)
Next, we aim to study the relationship between H and A. By the chain rule, dH
dH
dp
as
dp
dp (1 + p) + log
1-p N-I
log
1-p [N-1}p Substituting
1- (N
1)
we get
p
dH A and since A 0,
dp
0. We then turn to du dH
dp
<
as
[++NN- Dye apz
(N - 1)
0 since both
numerator and denominator must be > 0. Therefore,
All [+ 2
< 0, meaning that
1ye-
((a) is strictly decreasing in the margin A.
We will now use Lemma G.1 to prove Theorem 5.1.
Proof of Theorem 4.1. Consider a parametrized path by variable E [0, A]: define eg = ly + 65.
(0)
and aj
Define ej(h de g(t) and aj = d (t)
di aa)
(
Next, we differentiate the entropy ((a) with respect to t:
d N dt
H(a) Lia + aj af -
N
Σ a + Ina,
aj
j=l
Since a due to Σaj 1, this simply reduces to di H(a)= - - N of In
=
N
From Cover and Thomas (2006), we have that a² = a, - E. (eT), where E., [e] = Σ
anl.
k=
Plugging this into the expression for the derivative of entropy with respect to t:
d ((a)= = - Inaj = - -
Eaje,naa, Eall Σ a, Ina,)
dt
Observe that E, a, Inj- = Ea[Inn ai so this simply reduces as:
d (()) - -F,, ( In E. E,, (n al) -Cova (n
dt
Revisiting the meta-token setup where only the "correct" logit at j is boosted, this suggests that E = 1i(j = j²). Therefore, E. Ea(e"] = ag. and [e Ina] = ay- In ayr. This can be substituted into the covariance term above:
d [1(a) Cov. Cov. (aa- In ag- ag- E., al) -a-- -ay-(laa- E,, al)
di
Due to the Schur-concavity of H (a) (Marshall et al., 2011), an max, In and In a.. > E, [ln a). As such, given a,. > 0 and an. E,, [ln al > 0, this suggests that Cov ("", In > 0 and thus, & (a)) < 0. Therefore, we conclude that adding positive logit boost at the meta-token index ("correct" logit) strictly decreases entropy, supporting the proposed "anchoring" effect notion.
G.2 PROOF OF THEOREM 5.1
Proof of Theorem 5.1. The meta-tokens are simply a new (latent) channel that may be utilized to search for candidate distributions. However, this latent can be ignored, yielding the original search
22

---

Published as a conference paper at ICLR 2026
space; that is, any encoder ( z) that does not meta-tokens can be implemented in the meta-token model by zeroing out all meta-token contributions. Therefore, Cabs C Qmeta, where q (go. qu) over the feasible combinations of encoder and decoder. Naturally, minimizing function over a larger feasible set cannot increase its minimum. Thus, for a fixed rate R.
D meta (R)
D(g) S min qf meta : 1(x:)==R
min
D(g) = Dabe (R).
qeQass: 1(x;))=1
Note that the same result holds for RoPE in place of APE (i.e. DROPE in place of Daba), as well.
G.3 THEOREM G.2
Theorem G.2. Consider functions p (0,...,T-1))>>> and (-(T -- (-.T-)..... I-13 + for absolute postional biases and relative biases, respectively. Let Babs to be the set of all fixed absolute positional bias matrices B = p(j) and Butt to be the set of all fixed relative biases B = -(i j). Let meer be the set of bias matrices implementable by the Transformer augmented with meta-token embeddings {mi} which emit a content content-dependee dependent logit boost at their respective indices. Then,
Bahs U Bres & Bmeta
Proof. We break this argument down into two parts - (i.) the forward direction, where we show that all absolute and relative biases without meta-tokens can be modeled by the meta-token model.
(i) Bass UBtl S Bmeta+ Every B Bmeta is obtained by choosing meta-token embeddings e, Rd at each position t and linear head W, so that the total bias at (i,j) is B, = E. Q W e 1j=t-
Absolute case. Given p(j), set W E Rixd
dd and choose each e, so that Q7 W ej = p()) Vi
All other Etz are zero. Then B,, = p(j).
Relative case. Given b(i - j). place a meta-token every position j. Choose W and embeddings e, so that Q W e, = b(i j) v i.j.
For instance, if we let W = Td and arrange that ej encodes the vector (b( - -),, b(2 b). - j)), then Q, e b(i j) when Qi is the i-th standard basis vector.
Therefore, every absolute or relative bias (in Babs and Bret) lies in Bmeta-
(ii) There exists a bias B* E Bmeta such that B" & Bans U Brot- Define content-dependent bias Bi = ((;) where Cý is the full token context preceding position j and f is any non-constant function. Such a B arises by setting each meta-token embedding ej = f(C;) and W ld, so B" Bueta-
However, if there was B" Babe. then there is p(j) with p(j) = f(C,) for all j and all possible Cj which is impossible since C, varies. Furthermore, B" E Biel. then there is b(i -j) with b(i bi-j) j) f(C,) independent of i; again, this condition is impossible to be satisfied. Therefore
B" & Butss Bral-
As result, we conclude that the biases represented by Bmeta contain the set of both absolute and relative biases without meta-tokens, and represent additional biases that cannot be represented without meta-tokens.
The result of Theorem G.2 is that the introduction of meta-tokens strictly grows the expressivity of biases that may be represented, while still being entirely inclusive of the fixed realizable absolute and relative encoding biases. As a result, we do not "lose" anything representationally by introducing meta-tokens, from positional biases standpoint. This enhanced expressive power also plays role in enabling the model to lear to focus attention on relevant context spans, reinforcing the aforementioned sharpening effect.
23

---

Published as a conference paper at ICLR 2026
G.4 META-TOKENS AS CONTENT-ADAPTIC INDUCING POINTS
We formally show that meta-tokens operate akin to inducing points or landmarks, with respect to the attention kernel. Specifically, we suggest that routing attention through a small set of meta- tokens across the context behaves like a Nyström approximation, a low-rank factorization with provable attention error bound. Consider the softmax kernel K, exp( OK" ) and weights yaz
W = ( 2 0 over masks and biases, so the effective kernel is K' = K @ W. and attention can be defined as = RowNorm( Let the meta-token indices be M (I)...,T)
Theorem G.3. Define the Nyström kernel K K.,MKK Kt M KM,, and À RowNorm( K W). For any row for 8min = min [( o W)rs
2 Smin
I|A Av.
|[(K K);, O Wa,l11
(6)
If the kernel feature map of {ki} lies the span of (km m E M). then K K and therefore Á = A.
Proof. First, we start by showing that K, the gram matrix of c(x,8) exp( (3))))
is
va
a positive semidefinite matrix. By the power series, exp( (m-u))
Σ
1
(anu)) ym
Vd
ml
vd
m=0
Vmldmma Vinldmaa y ())) Thus, we can write <( y) = (6(x), d(v)), so K is a Gram
0
matrix of inner products, and therefore is PSD (and valid kernel).
Lemma G.4. For positive u,y ERT:
2 LTu l 112 s min (1 1Tv) 1/u w/11
Proof of Lemma G.3. Let a 1 u and 1' v; we can write the following:
va (u b)b + v(b a) ab
ab
Next, taking the Ll-norm on both sides and applying the triangle inequality yields:
u a 7 ll
1 = (u - w)b + v(b a)llı = ab (ll(u - w)ll + ||o(b a)l12)
1
Since ||l||| b and al 11 1=uul S ||u v//1, we have:
v |||
b/lu
+ w/1i
bill al]i 2 ||u - w//
ab
By symmetry, the tightest bound is:
2
2112
||u 1//2
a min (a, b)
min (1Tu, (Tv)
Take (K ② W);, and o= (K O W)... and apply Lemma G.3. This yields:
2 (RowNormuu) RowNorm(v)]l S min (1 u, 1To) ||u v/|i
(8)
Recall that A,,- = RowNorm(u) and A. = RowNorm(u). Observe that all rows are positive, so IT and (Ty v are 2 O. Therefore, we can replace min (1Tu, o") by Smin = min [( o W)rj+
and substitute the values of u and v in to yield our desired result:
2 8min
||A A <
[I(K K)2, @ Waalh
(9)
24

---

Published as a conference paper at ICLR 2026
We currently have that the softmax kernel can be written inner product in feature space, as K = ((Q)&(K)) Suppose that the feature vectors of all keys {Φ(k;) } lie in the span of meta- token features (#(k))
E M). This means there exists some coefficient matrix C' such that for Substituting this
((K), M which only collects the meta-token feature columns, ((K) = into the kernel expression yields:
K = =(Q) (KO)) = (#())
(10)
We can observe that ( is equivalent to the sub-matrix of K with columns at the meta-token indices, which is denoted by K.MM. so
k= K
and since every column of K lies in the span of K.M under this assumption, there must exist some matrix C' such that this holds.
Consider single column j of K, over the rows indexed by M, yielding:
KMJ KM.MC
for coefficient vector Cj (the j-th column of CT)) This is a linear system of the form Az = b, and stacking the columns gives the form of KM.:- = KM,MC*, which a least-squares problem for which the general solution is given by using the Moore-Penrose pseudoinverse, so C K K M,
Now, recall that the Nyström approximation is defined as K = K.MMKM.M. This is exactly equivalent our form for K K.M -K KMM Since K K, the normazzmd attentions must be equivalent as well, so À A.
This suggests that if meta-tokent span the feature map of a segment of context, then they capture that segment's structure, and thus the meta-tokens perfectly represent the attention kernel.
Proposition G.5. Take f [0, 1|T to denote a test statistic for a downstream task, and ÿ, (Ar, f) to be the predicted score at query position i; the distortion is measured by D; = KA,,,, (A... S' 1i- Del. the change induced by the low-rank approximation of A. Then, for 8min min Σ [K @ W)ry:
1 -----11 II(K R);, @ Wa,lh
1i Til S I1. Acolli
(11)
Smin
Proof. We can first show that 1ú: 5.l I(A., A... 8I S 11 A.,.l.. Take A., and q = Ar.. and let p-q
q be a probability distribution; notably, Σ, 2, = 0. Split the indices of z into
positive and negative sets: 5 = j 23 2 and S_ = 23 < 0, s0 (z, Σ 2z5ة + Σ 2,
jESt
JES_
Becuase 0 H fy I 1, we have
Σ 24f1 Σ JES
2j and Σ JES_
2fj 0, so
jES+
f) < Σ 2j
JES_
Similarly, using fg 2 over S4 and fj < 1 on S_, we have
(2,f) 2 Σ 2j
JES
Thus,
Σ
24 (2, < Σ 2j
JES_
JES+
25

---

Published as a conference paper at ICLR 2026
However, note that for zero-sum vector, the L1 norm would split evenly between positive and negative indices, so Σ =
Σ 2j ||[2|11. Therefore.
JES+
JES_
1 ||=1|/ (2,
Substituting Ar.: As., yields:
(A A..: f) S 2
1 A1 - Ar,012
(12)
Next, applying the result in Theorem 5.2 that ||Ar, Ar,ll1 S 2_ [I(K K);, @ Wall provides the complete result:
smin
1 |[(K KK,, @ Wa.lla
1 Jel = IAA Al.ff1l blIA A S
(13)
8min
From rate-distortion perspective, the number of meta-tokens controls the rate, and the approximation error in Yi an upper bound on the distortion. The above analysis provides a total variation bound for the distortion in application to downstream tasks, which kernel residual. This perspective guides our analysis on meta-tokens caching preceding context.
frame in terms of the Nyström
H EXAMPLE PROMPTS FOR SYNTHETIC TASKS
We provide example prompts and target completions for each synthetic task used our experiments. Each example illustrates the input format, the placement of PAUSS_____ meta-tokens, and the expected model output.
List Recall
Prompt:
Fruits: orange peach banana plum
Tools: hammer wrench level pliers
Sports: boxing baseball boxing golf
Spices: turmeric cardamom cumin cinnamon
Animals: cat tiger wolf fox
Profeasions: teacher nurse lawyer architect
Vegetables: onion cucumber broccoli cucumber
Instruments: piano clarinet violin piano
Professions: teacher nurse lawyer PAUSE_ architect Q: What is item of Professions? PAUSE_
Completion:
lawyer
Segment Counting
Prompt:
Fruits: orange peach banana plum
Tools: _PAUSE_ hammer wrench level pliers _PAUSE_
Sports: boxing baseball boxing golf
Spices: turmeric cardamom cumin cinnamon
Q: How many times does wrench appear between the pauses around Tools? PAUSE_
Completion:
1
26

---

Published as a conference paper at ICLR 2026
Parity
Prompt:
Bits: _PAUSE_
Q: What is the XOR of all bits before this pause? PAUSSE
Completion:
0
Copying
Prompt:
...as ever Mr _AUSE_ Darey had been, was much relieved by discerning such different feelings,
They had not been long together before Darcy told her that Bingley was also coming to wait on her; and she had barely time to express her satisfaction, and prepare for such visitor, when Bingley's quick step was heard on the stairs, and in a moment he entered the room. All Elizabeth's anger against him had been long done away; but had she still felt any, it could hardly have stood its ground against the unaffected cordiality with which he expressed himself on seeing her again. He inquired in a friendly, though general, way, after her family, and looked and spoke with the same PAUSE_ good-humoured ease that had done. Q: Copy the bracketed text. _PUSS__
Completion:
Darcy had been, was much relieved by discerning such different feelings.
They had not been long together before Darcy told her that Bingley was also coming to wait her; and she had barely time express her satisfaction, and prepare for such visitor, when Bingley's quick step was heard on the stairs, and in moment he entered the room. All Elizabeth's s anger against him had been long done away; but had she still felt any, it could hardly have stood its ground against the unaffected cordiality with which he expressed himself on seeing her again. He inquired a friendly, though general, way, after her family, and looked and spoke with the same.
Multi-Hop Retrieval
Prompt:
Flowers: sunflower daisy sunflower tulip rese marigold rose jasmine daisy 1ily
BodyParts: nose back ear foot hand foot mouth mouth ear foot PAUSE
Minerals: calcite dolomite halite gypsum magnetite halite gypsum magnetite fele
Animals: tiger elephant giraffe elephant elephant dog wolf tiger tiger zebra Pi
Hobbies:
reading hiking cycling painting gaming painting reading painting writ:
Clothes: scarf scarf gloves shirt dress coat dress dress dress shirt PAUSE
Planets: Pluto Mercury Uranus Ceres Uranus Mercury Jupiter Jupiter Jupiter Pluf
Vehicles: scooter plane bus train train plane plane tram van car PAUSE
Fruits: orange kiwi mango kiwi orange lemon pear plum grape banana PAUSE
Spices: turmeric paprika clove saffron pepper turmeric pepper turmeric nutmeg
Jumps:
nose -> Vehicles [3]
bus Vehicles [7]
27

---

Published as a conference paper at ICLR 2026
plane Animals [1]
tiger -> Flowers [8] PAUSE
Start: [1]
Hops:
Q: After following 4 jumps, what is the final item? PAUSE
Completion:
jasmine
I ADDITIONAL RELATED WORK
1.1 LANGUAGE MODELING WORKS WITH SPECIAL TOKENS
Kim al. (2025) inserts pause tokens when the token log likelihood is low, allowing the model to think for longer, aiding reasoning. These special tokens may also delineate phases of reasoning, as in Quiet-STaR (Zelikman et al., 2024). Quiet-STaR uses begin-of-though token and an end-of-thought token, generating a silent rationale sequence for each step before emitting the next word, showing that this helps zero-shot reasoning. Such demarcations have begun to used widely in the development of large reasoning models (Muennighoff et al., 2025).
1.2
IN VISION TRANSFORMERS
For vision transformers (ViTs), Vit (Jiang et al., 2024) introduces a similar meta-tokens notion our work by adding learnable sparse tokens and an attention mechanism between standard tokens and their meta tokens, improving performance and reducing spatial redundancy. Darcet et al. (2024) uses specialized "register" tokens applies patches denoise images by extracting the high-norm, outlier tokens, smoothening the feature and attention maps. These works suggest that special tokens, even devoid of semantic content, can influence model's internal reasoning and memory mechanisms.
1.3 NYSTROM APPROXIMATION OF ATTENTION KERNEL
There exist few works related to Nyström approximation in Appendix G.4; Nyströmformer (Xiong et al., 2021) approximates self-attention, yielding an O(n) method that outperforms standard self-attention at longer lengths. It selects inducing points or landmarks using Skyformer (Chen et al., 2021) uses a Gaussian kernel (kernelized attention) in place of softmax in self-attention, and applies the Nyström method for efficiency. We instead the Nyström method analysis tool study the role of meta-tokens - inducing points that we introduced during training in approximating the softmax kernel by caching preceding context.
BROADER IMPACTS STATEMENT
Our work on learned meta-tokens and meta-attention offers lightweight, data-efficient way pre- train language models while demonstrating strong performance when fine-tuned for recall tasks. This suggests a path toward more capable, leaner language models that could be used to handle contexts such as like long legal or medical documents, extended multi-turn dialogues, or large codebases without resorting to prohibitively large architectures or expensive fine-tuning runs. Such models could bring real benefits to areas such as conversational agents for education or healthcare. Building off of prior literature that performs a more explicit learned retrieval from the context (Mohtashami and Jaggi, 2023b), this could induce improved and efficient in-line retrieval over vast corpora.
Our work relates strongly to the recent debates in the language modeling community on the impact of positional encoding, particularly around works such NoPE (Kazemnejad et al., 2023). We provide strong evidence that zeroing the positional encoding can improve performance, providing motivation for hybrid attention mechanisms such as RNoPE (Yang et al.. 2025), and other, more efficient ways to pre-train language models with long-conter contectntext modeling settings in mind. We note that advances in long-context modeling could introduce risks around misuse and unintended harm. More powerful
28

---

Published as a conference paper at ICLR 2026
context understanding over long ranges can fuel phishing text and distracted models, especially in the phase noisy context. However, models trained on corpora without data pre- pre-pesessing a priori may be subject to harmful behavior such profane generations. In the context of our work, which uses standard, pre-filtered corpora, this issue is avoided; we encourage users to audit the data used for pre-training first.
K LARGE LANGUAGE MODEL USAGE STATEMENT
Large language models (LLMs) used in proofreading and editing the writing in this paper, for the purpose of ensuring the contributions and findings in this work are made clear to readers. They also used confirm consistency of the mathematical notation used in the paper.
29


Reviews:
[{"review_id": "ZjcSI1J4a0", "reviewer": "Reviewer_ViJ9", "summary": "This paper proposes meta-tokens and a per-layer lightweight meta-attention step that enable decoder-only LMs to cache compressed summaries in-sequence, improving long-range recall and length generalization with minimal architectural changes and modest compute.", "questions": "- Because a meta-attention layer is added after every self-attention block, the parameter count rises from 124M to 152M; could the observed gains be primarily due to the increased parameter count rather than the proposed mechanism? If the parameter budget were held fixed, would the method still yield performance improvements?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The method appends a meta-attention layer after each standard self-attention block, creating a lightweight in-sequence caching pathway that is minimally intrusive and easy to integrate.\n- The paper provides mutually consistent interpretability evidence—entropy visualizations and a rate–distortion perspective—that corroborates the empirical findings.\n- The approach is compatible with length-extension techniques such as YaRN, facilitating deployment within existing long-context pipelines.", "weaknesses": "- The evaluation focuses primarily on synthetic recall tasks and small to mid-sized models, with limited validation on real long-document tasks and larger-scale settings.\n- There is no direct comparison against recent strong long-context baselines, such as Infini-attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Q1hCXQUqE", "reviewer": "Reviewer_4UF2", "summary": "The paper proposes to insert \"meta tokens\" during pretraining and apply an additional \"meta attention\" layer to sharpen the positional encoding, allowing for better long context capabilities.", "questions": "- have you compare with a strong baseline that already pretrained for longer context? Although not related to claiming SoTA, this is useful to see how much regret there still is in the post-training context extension.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The idea is novel.\n- The method is well-motivated\n- Very good presentation. \n- It went further by presenting evidence that meta-tokens can compress context in their representation and formalizing into theories.", "weaknesses": "- The meta token method, though theoretically simply, requires a change in the architecture which is sometimes hard to adopt.\n- When performing length extension at post-training, one small nitpick is that I wonder if it is suitable to use YaRN as a baseline to compare with. YaRN is a technique that applies to a pretrained model without any additional effort in pretraining, whereas the current paper in a sense already put int a lot of additional preparation in pretraining dataset and architecture changes. But it is relatively small because I don't have a good alternative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ml6CYZgkr8", "reviewer": "Reviewer_8ygB", "summary": "This work proposes a novel method to improve long-range context modeling in Transformer-based language models through the introduction of meta-tokens and a corresponding meta-attention mechanism.", "questions": "1. All experiments are conducted on a 152M parameter model, leaving scalability to contemporary 1B+ models unevaluated. Furthermore, the analysis of computational overhead in Appendix F.7 is minimal. Could the authors provide a more detailed analysis of the added latency (ms) and TFLOPs, and critically, provide any evidence (empirical or theoretical) that the method's benefits will scale to much larger models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Provides information-theoretic and empirical analyses that explain how meta-tokens function as content-based anchors and sharpen the positional encoding.", "weaknesses": "1. While the abstract claims that “meta-tokens and meta-attention provide a simple, data-efficient pre-training method,” the paper does not clearly demonstrate this aspect.\n\n2. The empirical validation is conducted solely on four self-designed synthetic benchmarks. Although these tasks are useful for controlled analysis, they may not sufficiently reflect the behavior of the proposed method on real-world or natural language datasets. This limits the generalizability of the findings.\n\n3. The paper does not appear to provide a theoretical or empirical analysis of the upper bound of long-range generalization enabled by the proposed method.\n\n4.  All experiments are confined to a 152M parameter model, which is insufficient to demonstrate that the proposed method can scale to larger, contemporary architectures. This lack of scalability analysis is a critical missing piece of experimental evidence. Furthermore, the paper fails to provide a detailed analysis of the computational overhead and inference latency introduced by the meta-attention mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZREHa953K", "reviewer": "Reviewer_tDK3", "summary": "Contributions:  \n1. Novel meta-attention mechanism that uses meta-tokens to improve performance on synthetic recall tasks\n2. Theoretical analysis on context compression\n3. Empirical experiments showing length generalization in relation to positional encodings. Zeroing out the positional encodings on meta-tokens long context improves recall.", "questions": "- Does meta-tokens also improve the pretraining perplexity? Language modeling is also a very important metric that evaluates the average \"quality\" of the text generation, which is not always related to the recall accuracy.\n-  Have you tried fixed periodic placement strategies for the meta-tokens? \n- What do you think about adaptive placement strategies based on perplexity? Randomly placing during training currently doesn't match the inference time's deliberate per-task placement.\n- Why was YaRN necessary here? There can be counfounding factors which YaRN by itself improves length generalization on the model. Have you tried using alternatives like sliding window attention combined with meta-attention?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Well motivated, long context modeling is a very pressing issue in LLM research.\n2. Strong theoretical grounding, Theorem 5.1 provides a clear explanation for the effectiveness meta-tokens.\n3. Many synthetic evaluations, tasks, ablations.\n4. Meta-attention is original and novel. The algorithm is fairly simple and can be used during pre-training.", "weaknesses": "1. There are only synthetic evaluations, no evaluations on standard NLP evaluations or proxies to realistic long-context applications.\n2. Degradation of inference speed, no benchmarks for training speed.\n3. Meta-token placement strategy is not explored, no ablations on the actual placement strategy of meta-tokens.\n4. Inference-time deliberate placement of the meta-tokens requires prior knowledge of the task the LLM is trying to solve, hindering generalization.\n5. Limited scale, 150M parameters is very small by modern standards. Tested context length is also very short (512), which does not necessarily translate to longer contexes.\n6. No comparison against other popular long context attention algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihbwBVsy3j", "reviewer": "Reviewer_2ANK", "summary": "This paper introduces a method to improve the long-context reasoning and length generalization capabilities of Transformers. The authors propose two key components: meta-tokens and a meta-attention mechanism.\nMeta-tokens are special, learned tokens that are injected randomly into the input sequence during pre-training (on <100B tokens) and then placed deliberately at key points during fine-tuning.\nMeta-attention is a dedicated, sparse attention layer added to the Transformer architecture. This layer's mask ensures that meta-tokens can only attend to other meta-tokens, creating a separate information pathway.\n\nThe central idea is that these meta-tokens learn to function as \"content-based anchors\" that compress and \"cache\" information from the preceding context. The meta-attention layer allows the model to build and access a \"summary stream\" of this cached information.\n\nThe authors train a 152M parameter model and show it outperforms baselines (including a GPT-Neo model trained on 3x more data) on synthetic tasks. The model also demonstrates strong length generalization, performing well on sequences up to 2x its training context length, even after extension with YaRN.\n\nThe paper provides a good theoretical and empirical analysis, arguing that meta-tokens work by \"sharpening\" the attention mechanism (reducing its entropy) and serving as a more efficient (in a rate-distortion sense) method of context compression. A key finding is that model performance improves when the positional encoding for these meta-tokens is zeroed out, suggesting the model learns to rely on their compressed content for localization, not their absolute position.", "questions": "- Your work shows that meta-tokens, trained with a dedicated meta-attention layer, function as effective content-based anchors for caching context. This quite similar to the attention sink phenomenon, where a single, fixed token (like the BOS token) implicitly learns to aggregate information from the entire context via the standard attention mechanism.\nWhat do you think are the main differences? Do you believe the explicit, sparse meta-attention layer is the critical component that distinguishes your method's success and enables robust length generalization, or could a similar caching behavior be explicitly trained into standard tokens (or a single 'sink' token) without requiring a separate attention pathway?\n\n- Plus, not a question but this paper actually gets to very similar conclusions about how having a high attention on specific tokens can create a sort of reference frame that is necessary for proper processing. https://arxiv.org/abs/2508.02546\nThis paper is more specifically about positional encodings and attention sink, but I think it's strongly related and it could strengthen your claims, especially because they show that the sink phenomenon is related to the positional encoding scheme (and of course, the softmax pressure) and in particular, RoPE rotations.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- I really like the idea of exploiting something like attention sink to actually build a coordinate system for the model using the meta tokens. And the paper doesn't just present empirical results, it also provides a solid mechanistic explanation for why it works. The information-theoretic analysis (theorem 5.1 on entropy reduction, theorem 5.2 on rate-distortion) provides a formal basis for the \"sharpening\" and \"caching\" hypotheses.\n- The idea of a dual-pathway system (standard attention for content, meta-attention for a \"summary stream\") is very interesting. It's not just another modification to positional encodings but a new architectural component with a clear, explainable purpose.\n- The fact that zeroing out the positional encoding of meta-tokens boosts performance is a core result. I think it supports the claim that these tokens are not just positional markers but have become true content-based anchors.", "weaknesses": "- The main limitation is that the model is only evaluated on synthetic tasks designed to probe recall (list recall, copying, parity, etc). While this is excellent for analyzing the specific mechanism of memory, it doesn't really provides evidences that these gains will translate to real-world, general-purpose language tasks. Plus, the experiments are performed on a 150M parameters model. So, it's unknown if this meta-attention mechanism will scale effectively or provide the same benefits in billion-parameters models.\n- The paper doesn't fully explains why the meta-tokens are injected uniformly at random during pre-training but are placed deliberately (as _PAUSE_ tokens) at task-specific locations during fine-tuning and it still works. \n- The paper claims \"little overhead\", but it does introduce an entire new attention layer (meta-attention) and increases the sequence length by 10% ($k=0.1$) during pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)