PERK: Long-Context Reasoning as Parameter-Efficient Test-Time Learning
Authors: Zeming Chen
Angelika Romanou
Gail Weiss
Antoine Bosselut
Venue: ICLR 2026 Poster (2026)
Keywords: test-time learning
long-context reasoning
meta-learning
reasoning algorithm
length extrapolation
Primary area: foundation or frontier models, including LLMs
TL;DR: A scalable meta-learning method that learns to encode long contexts through gradient updates to a model adapter at test time, achieving long-context reasoning robust to complexity and length extrapolation.

Abstract: Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts.

Published as a conference paper at ICLR 2026
PERK: LONG-CONTEX" REASONING AS PARAMETER-EFFICIEN TEST-TIME LEARNING
Zeming Chen
Angelika Romanou
Gail Weiss
Antoine Bosselut
EPFL
email (zeming. chen,
@epfl. .offf.cch ch
ABSTRACT
Long-context reasoning requires accurately identifying relevant information in ex- tensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long con- texts using gradient updates test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes con- texts into low-rank adapter (LoRA) that serves parameter-effiiien memory module for the base model. Concurrently, the outer loop learns to use the updated adapter accurately recall and reason long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of to 20% for Qwen-2.5 (0.5B & 7B) on syn- thetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of rel- evant information in contexts. https: /perk-- long-
relevant information from the encoded
Long-contex: context. web app
INTRODUCTION
Large language models (LL.Ms) struggle with identifying and reasoning over relevant information in long, potentially noisy, input contexts (i.e., long-context reasoning; Anil et al., 2022; Lee et al., 2024; Li et al., 2024; Liu et al., 2023; Xu et al., 2024), including modern models with substantial context windows (Kuratov et al., 2024; Levy al., 2024; Hsich et al., 2024). These challenges arise because longer contexts contain more irrelevant distractor information, and may also be attached to more complex reasoning problems (i.e., multiple hops), both of which degrade LLM performance (Chen et al., 2023b). Furthermore, these challenges are exacerbated by positional biases in long- context models, which focus attention on the beginning end of contexts, neglecting intermediate information (Liu et al., 2024).
In this work, we address these challenges of long-context reasoning by reframing it test-time learning, enabling models to encode context at inference time using gradient-based parameter adaptation. Our approach, Parameter-Efficient Reasoning over Knowledge (PERK), processes and compresses lengthy sequences as batch (or multiple batches) of shorter segments of the long context, rather than processing the full sequence token-by token-by-token by-token. The updated model can then use the newly stored parametric information (and its initial parametric knowledge) to answer relevant questions. The original context then discarded in this answering step as is now encoded parametrically by the model. To train PERK, operate two nested optimization loops: the inner loop, which is also used at test time, encodes the long context segment batches into model parameters, while the outer loop learns to respond to queries using the updated parameters representing the context.
However, bi-level optimization algorithms for training test-time learning methods entail prohibitive memory overhead (Finn et al., 2017; Chen et al., 2023b; Hu et al., 2023), particularly when updating all model parameters, because they require backpropagating through a complete multi-step optimization trajectory. To reduce memory overhead, PERK (1) internalizes the context into Low-Rank Adapter

---

Published as a conference paper at ICLR 2026
(LoRA; Hu et al., 2022) rather than the initial model parameters, and (2) backpropagates the outer loop over only a truncated trajectory from the final few inner-loop adaptation steps (Shaban et al., 2019). Consequently, PERK scales efficiently training, supporting larger models and longer contexts.
We compare PERK to models finetuned on long contexts for classical in-context reasoning (FT-ICR) on multiple Needle-in-a-Hayyee Hay-stack benchmarks, demonstrating PERK's s advantages. For Qwen-2.5- 0.5B with a native 32K context window, PERK surpasses FT-ICR baselines by a 20% absolute improvement. PERK also outperforms specialized long-context LLMs (Gao et al., 2025a; Yang et al., 2025) with over 7B parameters that have undergone extensive long-context training, achiev- ing the highest accuracy when generalizing to contexts up to 128k tokens. As relevant facts in Needle-ine -i--iin----aa --aa--aayt problems are often easily distinguishable from distractor text, we also propose more challenging evaluation settings where relevant facts are surrounded by distributionally-smmiaa HotPorQA, TriviaQA). In these settings, PERK models
distractor facts (e.g., Drops-in-the-Ocee again outperform the FT-ICR baselines, and even surpass commercial LLMs (e.g., GPT, Gemini) on benchmarks for long-context symbolic reasoning. PERK also works effectively across many model scales (0. IB, 0.5B, 8B) and families (GPT-2, Qwen-2.5, LLaMA-3.1/3.2).
Finally, our analysis also shows PERK'S advantages yield strong test-time length extrapolation. PERK trained on 8K-token contexts maintains decent performance on 64K and 128K windows, consistently surpassing FT-ICR length generalization, and even outperforming models with native 512K and IM context windows. Finally, because PERK encodes sequences as a inruuuaationiivariin batch of context window segments (rather than maintaining the full contiguous context), it generalizes across contexts where relevant information is variably distributed along context positions, unlike FT-ICR's dramatic drops in performance (up to 90%) when relevant information shifts positions at test time.
2 BACKGROUND AND PRELIMINARIES
Causal Language Models. Causal language models (CLM) fo (Radford & Narasimhan, 2018) with parameters are typically trained with an autoregressive learning objective such as the token-level negative log- --likeihhodd (NLL). For dataset D = {a) with ND sequences, this loss is defined as:
T
LNII(0,,7 = ND
Σ RED
log (xt w<)).
The initial parameters are optimized iteratively via an optimization algorithm Alg (C, + 0', where h are hyperparameters, such as optimizer choice and training length.
Reasoning with CLMs. We denote reasoning problems as tuples r (K, q, y) of context K,, question q. and target response (e.g., question-answerigg dataset like HotpoiQA; Yang et al., 2018). With the pretrained CLMs's ability to reason over facts in input contexts with reasonable success (Clark et al., 2020), prompt-based in-context reasoning (ICR) has become popular for solving reasoning tasks. In this setting, IC and q are combined into a single prompt a(r). The model's parameters are optimized to predict the tokens of the correct answer:
Ty
Creason (O,D)
1
Σ
Σ
log [<().
(2)
ND, (K...V)E
where D is the training corpus, Ty is the length of an example answer, and No, is the total length of all answers in the dataset.
Reasoning as Test-time Learning. In this setting, models simulate reasoning by first encoding the given context I. into CLM's parameters 0 and then using the updated parameters Φ(0, K)) to generate a response to the given question (Chen et al., 2023b). The context is encoded by adapting the model's parameters using a CLM objective over the context: Φ(0, K.) = Alg (CNLL, K. h). We
then use fø to generate answers for any relevant questions q.
Test-time learning methods for reasoning stem from key observation: CLMs can store and retrieve diverse information in their parameters from pre-training (Roberts et al., 2020: Bayazit et al., 2024). Yet these same models struggle with long-contess problems (Hsich et al., 2024; Kuratov et al., 2024; Xu et al.. 2024; Levy al., 2024), despite the contexts containing far less information than what is seen during pre-training. CLMs may reason better with knowledge in their parameters than in context, motivating methods that encode new context parametrically for reasoning.
2

---

Published as a conference paper at ICLR 2026
Encoding Context
Learning Reason OUTER LOOP TRAMING
WNER LOOP TRUNING
Pelamant Oonuments
Pre-trained
Mamory Scratchpad
Pre-trained
Memory Scratchpad
LLM
Question Answer y
LLM
LML
Liuos
V gutor
Meta-learning PERK for long-context reasoning. The training procedure involves nested inner and outer loop. The inner loop optimizes the likelihood of batch of long context segments with respect to the parameters of the LoRA-based memory scratchpad. In the outer loop, the model uses the encoded information the memory scratchpad to answer questions. In both cases, only the memory scratchpad parameters updated while the base LLM's parameters are frozen.
Figure 1:
Training a CLM for Test-time Learning. Given distribution R of possible reasoning problems, the optimal parameters 0 for test-time learning CLM minimize the expected reasoning loss of the CLM's adaptation to each reasoning problem r ~ R:
0* arg min
E
Creason ((0,K))((.,)))
y~R
where o(0, K.) is CLM adaptation of 0 to the context K: 6(0, AAlg(ENLL. IC, h))
Meta-leaming algorithms, such MAML (Finn al., 2017), can be used for training effective test-time learning methods. In MAML, we obtain a meta-model fo via bi-level optimization: the outer loop samples batches of reasoning problems from training dataset of reasoning problems, the inner loop obtains the adaptations of 0 to each problem's 's context K., and the outer loop then optimizes the meta-pen patta--armssters based on the performance each fo(o.)) on the corresponding questions. The outer loop optimization backpropagates through the entire inner loop, an operation that becomes increasingly expensive with more parameters and inner loop update steps.
PERK: META-LEARNING SHAAMETER-EFFICIEE REASONING OVER KNOWLEDGE
We propose PERK, bi-level optimization algorithm for model reasoning via test-time adaptation for any given long context. At inference time, PERK adapts base model to encode a context via CLM optimization that context. Subsequently, the adapted model can be used to answer relevant questions about that context. To scale PERX's adaptation to larger and more capable base LLMs for parameter-efficiee methods to reduce both the size and
effective long-conted reasoning, amount of gradient unrolling required during optimization.
3.1 PARAMETER- EFFFCCIET
WITH LOW-RANK ADAPTATION
During training, PERK optimizes a parameter-efficien adapter, LoRA (Hu al., 2022), instead of the complete model. Specifically, PERK encodes context into the adapter through gradient- based updates, leaving the base model parameters unchanged. We denote by 0 and auupperr the parameters of the base model and the LoRA adapter, respectively. O is never optimized, while Outapter is meta-learned to a good initial state for test-time adaptations.
Test-Time Learning and Inner Loop Training. We set the test time learning objective (thus also the inner loop objective) to causal language modeling of the context: [NLL(KC, (Obase- a ). and set the adaptation algorithm Alg(ENTLL, (Orase, Oatapter), h)) to only update Ondapter. The adaptation algorithm computes the gradient V VLNLL on batch of sebbeequeeces from the full context sequence parallel batch (or multiple batches) allows to process lengthy sequences
K. This compression beyond the model's native context window. The exact manner by which broken into batches is a hyperparameter of the algorithm, and for efficiency reasons, set to batch during training.

---

Published as a conference paper at ICLR 2026
Outer Loop Training. In the outer loop, the model learns reason the encoded information in the adapter (memory scratchpad) given a problem q without access to the long context explicitly in the text space. We optimize the meta parameters Oadupter to learn from R. the corresponding distribution of reasoning problems. The optimal aduptee minimize the expected reasoning loss of the adapter's adaptation each reasoning problem r= (K, 9,999) R:
θ adapter
arg min E noR
Cresson (Besse: Padapter
(( (alleee((() addapie)) ), K.), ((q. ((q.))))
(3)
Qudspter
where @adapeer basse audapter) ) XC) = Alg(ENLL: IC, (foase addpperr) h) is the parameter-efficiee CLM adaptation of addaperr to the context IC obtained via the test-time adaptation. We may use any optimizer for mulapeer and our experiments, we use Adam W (Loshchilov & Hutter, 2017). We describe further hyperparameters, training & inference details, and long-context data preprocessing in Appendix B.
3.2 SCALABLE META-LEARNING WITH TRUNCATED GRADIENT UNROLLING
To optimize Equation (3) with gradient-basee methods, we need to differentiate through Alg, which involves higher-order derivatives and saving the complete trajectory to compute the meta-gradient (i.e., the gradient of (ressa). However, this creates high memory costs, capping the method's 's applicabilliyy To reduce memory costs, we truncate the backpropagation for the meta-gradient computation. We briefly describe this approach here, presenting the case where Alg optimizes Oadaperr via direct gradient descent for simplicity (though we use more complicated optimizers for the inner loop practice).
Gradient Unrolling (GU).
We denote an N-step inner-loop optimization's intermediate states by:
adaatee
axtupters
fn2+ Pnlapter
"daapeer
Oa (1).
Ondapter
adapter?
(N)
where g(n)
DNLL (Obase-
(n) addapter
KC)
is the gradient at step n, and a is the learning rate.
adapter
In vanilla MAMI,, to compute the gradient of the outer-loop loss freuson with respect to the meta- parameters O uadapter. one backpropagates through all N inner steps. By the chain rule, this equates to
d (n+1)
accumulating the Jacobian [(n
Fakeeer
from each step:
i Cresson ((N1)) (0)
do (0) adapter
Cresson (N) adaapter
N-1
V 0 atapts Creason
II
d (N) adupter
30 ndapter
do
==
where ((n) can be computed via: J(n) = I a H(n),
H(n)
V2
LNLL- Each Jacobian ((n
requires storing the computation graph of step in memory, so the cost of the backward pass scales linearly with N, creating a high memory requirement for gradient unrolling.
Truncated Gradient Unrolling (TGU). To reduce memory, we follow (Shaban et al., 2019) and run the inner loop for all N specified update steps, but only store the computational graph for the last T < N steps. This substantially reduces the computational memory requirements of the method, at the cost slightly less accurate meta-gradee peetaggrddeens for the outer-loop optimization of aadapter- To implement this approximation, we do not retain the inner loop optimization's computation graph until the start of the retention window: the last inner-loop optimization steps. Since all Jacobians before step Usstr are treated as constants, the Jacobian product in Equation (4) is effectively cut:
OLrasso a (N) adapter
N-T-1
V Ostoper Creason 2
II (J(N))
(5)
N=N-T
last steps
EXPERIMENTS
We empirically evaluate PERK's long-context reasoning performance in three scenarios: reasoning over (1) Needles-in- Haystack (NIAH, Kuratov et al., 2024), (2) Open-domain, multi-document QA (Multi-Doe), and (3) Drops- (DIO), novel long context evaluation setting we propose.
Heedles-in-aHHaystack
4

---

Published as a conference paper at ICLR 2026
PERK
Onev Oner IB
Owards3 Owar/8
Hamba 14B
cSS fonder
Commandaa horter
Gamin-1.5pm Syrr
NIAH: BABILONG
HOTPOTQA
Single supporting
supporting
Q43 Three supporting
Acconety/
50
25
32K
BK
32K
8K
32K
Contest Length
Context Length
Context Length
Context Length
DIG: STUDEN RECORDS
TRVVAAA
Recall of singlo rocord
Rolation between two records
Aggregation of records
100
75
Accurscy
32K
BK
32K
8K
32K
8K
32X
Context Length
Context Length
Length
Contest Length
Figure 2: Performance on Long-context Reasoning. We show the evaluation results on NIAH with BabiLong, Multi-Doc with HotpotQA & TriviaQA, and DIO with Student Records. All PERK and FT-ICR models (including Mamba) are trained on contexts with 8K tokens. When evaluated on the out-of-distribun contexts with 32K tokens, they must extrapolate to new context length. Note that we use substring exact match as the Accuracy metric for HotpotQA & TriviaQA.
4.1 EXPERIMENTAL SETUP
Datasets. We use the BabiLong framework (Kuratov et al., 2024) for reasoning over Needles-in-a- Haystack (NIAH), with single (QA1), two-hop (QA2), and three-hop (QA3) reasoning over facts scattered across very long documents. We sample documents from FineWeb- Web-Edu (Lozhkov et al., 2024) postdating our base model's 's knowledge cutoff. To address NIAH's fundamental limitation, where target information is stylistically distinct from distractors and thus artifically simpler to identify, we evaluate two additional settings.
First, we evaluate on HotpotQA (Yang et al., 2018) and TriviaQA (Joshi et al., 2017) for real-world long-context QA. Following Yen et al. (2025), we inject retrieved distractors (Zhang et al., 2024a) at random positions until reaching the context window limit. We report substring exact match (SubEM) as accuracy following (Asai et al., 2023).
Second, we propose Drops-ine the Ochenn (DIO), novel evaluation with distributionally similar relevant/irellvantt information. We create Student Records, a synthetic dataset where each context contains multiple student records (ID, name, school, major, grade). Context length scales with record count. We define three tasks: Recall (retrieve attributes by ID), Relation (compare attributes between IDs), and Aggregate (compute max/mmnnaaverage grades). Test entities and attribute combinations are disjoint from training to ensure generalization. Examples are available in Appendix A.
Baselines. We compare PERK against multiple categories of baselines. First, following conventional practice (Fu et al., 2024; Gao et al., 2025b), we finetune open-weight models for prompting-based based in-context reasoning. Our baselines include: (1) GPT-2-127M (Radford & Narasimhan, 2018) with IK context window, extended via positional interpolation (Karypis et al., 2024) for longer contexts; (2) Qwen-2.5 (0.5B, 7B: Qwen al.. 2025) with 32K context; (3) L.aMA-332-11 and LLaMA-3.1 --B (Grattafiori et al., 2024) with 128K context; (4) Mamba-1.48 (Gu & Dao, 2024) with unlimited context length. We also compare to commercial models with long-context abilities, GPT- 4.1 (OpenAI, 2025) and Gemini- (Team et al., 2024), and specialized open-source models: 1M (IM context via long-context pretraining, progressive length extension, and -ing-instruction post-training; Yang et al., 2025) and ProLonn-Llama- 1--amaa---B--111 1----11k--nnnrruue (512K context; Gao et al., 2025a). We use 4-shot in-context learning as the inference method, following the prompt templates of Kuratov al. (2024) and Yen al. (2025)
Swen2.55-B-Inntruue
5

---

Published as a conference paper at ICLR 2026
4.2 LONG-CONTENS REASONING RESULTS
NIAH. In the top row, left section of Figure 2, we show the evaluation results on BabiLong, PERK outperforms all baselines on all three of the QAI, QA2, and QA3 tasks, including much larger models and specialized onn-ccontett models that support much larger context-windows than 32K tokens (512K IM). When comparing similar training settings (i.e., training on 8k contexts), PERK (Qwen-7B) achieves a higher accuracy across all task complexities than FT-ICR (Qwen-7B) with an in-distribution context window of 8K at evaluation time. When extrapolating to 32K context window, PERK much more robust, outperforming FT-ICR by 23% points on average.
molti-de open-domain QA tasks HotpotQA and TriviaQA. When trained on long contexts with 8K tokens, PERK (Qwen) consistently outperforms FT-ICR (Qwen) for both HotpotQA and TriviaQA, achieving 20% absolute gain the 0.5B scale and 15% gain at the 7B scale. When extrapolating to contexts with 32K tokens, PERK shows stronger generalization over FT-ICR by 30% (0.5B) and 14% (7B). Compared to open-source long-context models (Qwen-IM I and ProLong) with extensive training on long-context instruction data, PERK still matches the performance with very narrow performance gap (3% for HotpotQA and 1.5% for Tri,iaQA). While not quite at the level of commercial LLMs, GPT-4. and Gemini- -rop PERK's performance closer to these leading performances than FT-ICR. Our results demonstrate PERK's strong potential for retrieving from and reasoning over multiple long-context documents.
Multi-Doc. Next, we evaluate PERK's performance
Drops-in-the-Ocean.. Results on the Student Records dataset Figure 2 mirror the trends seen in BabiLong. PERK (Qwen) outperforms FT-ICR baselines across task difficulties and context lengths and shows smaller drop than FT-ICR baselines when increasing the evaluation context length to 32k tokens. The performance differential between PERK and FT-ICR baselines also widens monotonically with task difficulty (Aggregate > Relation > Recall), suggesting that PERK enhances complex reasoning capabilities over long contexts rather than merely improving simple retrieval operations. Finally, as before, both PERK (Qwen) 7B and 0.5B outperform specialized long-context models despite their large parameter size and optimized training strategies for long sequences. FERK (Qwen-7B) also achieves performance that matches or exceeds the commercial models, Gemini-1.5-pr and GPT-4.1, even on the Aggregate task.
Model Generalizabilityy We further ana- lyze whether PERK'S performance can gen- eralize across diverse model families and scales Figure While FT-ICR shows highly variable accuracy ranging from 18. 2% on GPT-2 to 89.8% on LLaMA-
100
92.4
96.1
95.8
89.8
95,1
69.4
64.3
8B, PERK maintains consistently high ac- curacy across all tested models. PERK's performance scales positively with model size, achieving 4% increasing on Qwen- 7B compared to Qwen-O. W..BB and reaching 99.1% accuracy on LLaMA-8B. FT-ICR, despite improving with model size, still lags behind PERK by 9.3 percentage points even on the largest model tested. The gen- eralization across model families is also clear. PERK achieves strong performance on models from the GPT-2, Qwen-2.5, and LLaMA-3.1/34. families. Our results demonstrate that PERK's advantages are not limited to specific architectures or model scales, but rather represent a fundamentally effective approach for long-context modeling.
Accuracy (96)
50
25
18.2
GPT-2
Owen-0.58
Owen-78
LLaMA-1B
LLaMA-88
Figure PERK with diverse model scales and fami- lies on BabiLong QA2 with 8K train & test context length. PERK's performance generalizes across models and scales, consistently outperforms FT-ICR baselines.
5 ANALYSIS
In this section, we analyze PERK's robustness in two challenging settings: test-time length robustness, where PERK is evaluated on longer (or shorter) contexts than those seen during long context training (Press et al., 2022), and positional variation, where relevant contextual information is distributed throughout the long input context window (Liu et al., 2024).

---

Published as a conference paper at ICLR 2026
5.1 TEST-TIME LENGTH ROBUSTNESS
Setup. Focusing BabiLong tasks, which allow us to generate contexts of arbitrarily long lengths, we train a Qwen-2.5-0.5B base model using PERK on fixed-length contexts ranging from 1K to 8K tokens (as well as an FT-ICR baseline). Subsequently, we test on contexts ranging from 1K to 32K tokens (Figure 4). For analyzing length generalization on extreme context windows, we also extrapolate on contexts with 64K and 128K tokens (Table 1). We also compare PERK to recent training-free length generalization methods, Yarn (Peng et al., 2023b) and Dual Chunk Attention (DCA; An et al., 2024a) by applying these techniques to the FT-ICR Qwen model inference time.
PERK generalizes to new context lengths at test time. Figure 4 shows that PERK outperforms FT-ICR on the BabiLong QAI (single-hpp) and QA2 (two-hop) tasks for all settings. FT-ICR's performance drops drastically as the inference context length exceeds the training context length, especially for shorter training contexts. While PERK'S accuracy also decreases (from a higher starting point), the degradation is dampened at longer test lengths. Meanwhile, for interpolation, FT-ICR performance drops on shorter contexts when finetuned on longer ones, matching observations from prior work (Gao et al., 2025b). In contrast, PERK maintains (and even increases) its performance.
To stress-test the limit of PERK models' length extrap- olation ability, we test on se- quences with context lengths beyond the context window, 32k, of the Qwen-2.5 model - 64K, and 128K tokens in Table 1, and demonstrate that PERK consistently ex- trapolates to extreme contexts lengths more robustly than long-context finetuning (FT- ICR). With 128K token con- text windows, PERK's s accu- racy experiences noticeable drop to 14% and 44.4% ac- curacy for QAI and QA2, respectively, but these scores remain substantially better than the FT-ICR baseline at this same context length. After applying the training-free length generalization methods, Yarn and DCA, to FT-ICR Qwen models, the baselines show improved generalization on the extreme contexts. However, Yam + DCA's extrapolation performance is still far below PERK for both QAI and QA2 (around 35% lower on average). Compared to the specialized frontier models, PERK outperforms gemini-1.5
(a) Single-hop reasoning
(b) Two-hop reasoning
Model
32K
64K
128K
32K
64K
128K
Commercial Frontier Models
GPT-4.1
87.8
78.7
69.4
80.6
66.4
48.2
Demini-1.5-pp
87.7
82.3
73.1
56.4
48.8
40.2
Trained on contexts with >256 tokens
(wen2.5-7B-Innsin 1M
65.7
30.3
21.4
35.3
21.0
12.2
ProLong-AB-Instruuttss Trained on contexts with 8K tokens
41.0
33.2
24.3
29.0
18.5
17.7
FT-ICR (Qwen2.5-0.33
43.8
34.5
0
39.4
11.7
0
w/ Yam DCA PERK (Qwen2.5-00.BB
59.2
35.5
25.4
42.5
26.3
18.5
95.3
88.9
61.4
80.9
62.5
44.4
Table Test-time length extrapolation beyond 32K on BabiLong QAI (a) and QA2 (b). PERK and FT-ICR are trained on 8K-token sequences. The context length for inference grows from 64K to 128K. PERK extrapolates substantially better than FT-ICR.
and the two open-source models, and achieves a narrow performance gap to GPT-4.1 on both 64K and 128K contexts. We note, as well, that PERK is only trained on contexts with 8K tokens, while the Dwen-IM and ProLong models have been trained on much longer contexts with 256K and 512K tokens. Overall, PERK demonstrates strong length generalization, extrapolating to longer contexts while preserving high performance on shorter ones.
5.2 ROBUSTNESS TO POSITIONAL BIASES
Prior work has shown that the position of relevant information in long contexts affects a language model's 's ability to use that information (Liu et al., 2024). In this experiment, we test PERK's 's robustness to positional biases by evaluating its test-time performance on contexts where the position of relevant information is distributed differently from those seen in training.
Setup. We use API documents from Patil et al. (2024), where each document has an associated APT call, and create long contexts containing multiple must retrieve the correct API call based on user instructions and documents. Using fixed 4k and 8k token contexts, train PERK and FT-ICR models contexts with the target document varying positions and test them on contexts distributed across all positions.
pairs. In this task, the model
PERK generalizes regardless of information position in the context. Figure 5 compares PERK to FT-ICR under different position distributions (at train and test time) and context lengths. When trained on contexts with relevant documents randomly located (Rnd) throughout the context, PERK
7

---

Published as a conference paper at ICLR 2026
Inference Posttion
BK
100
100
99.8
99.0
98.5
95.3
97.0
94.2
93.5
92.4
09.0
80,9
100
PRE
POST
PRE
POST
PERK (Oung)
RND
74.8
73.2
71.9
76.3
08.8
67.1
67
67.2
100
100
100
92.8
86.7
70.8
Training Contout Longth
2K
100
100
92.
75.7
97.2
96.9
84.9
6
50.8
PERK(numi)
POST
75.4
73.2
73.6
75,3
68.1
IK
100
100
99.4
92.5
RQ.
$7
S8. 1
91.6
90.3
73.8
$4.5
40
29) fol.ommzyy
MID
74.5
74.3
74.8
P4.5
2
65.8
67.3
8
PRE
74.5
72.8
P4.1
74,2
691
08.8
67
67
8K
67.1
69.1
70.1
76
68.5
43.8
4
50.8
62.
64.
58.3
39.4
Traiiing
20.6
31.6
47.5
51.5
FI-RR
4K
85.5
87:1
58.5
80.9
73.3
49.4
43.4
56.8
69.1
61.6
53.7
43.3
FTHCR
POST
96.6
1.3
92.1
0.8
47.2
69.8
50.8
43.2
23.5
MID
0.8
68.2
0.4
1.9
0.6
57.3
0.5
1K
91.1
76.8
72.3
64.6
52.7
39.8
70.5
57.5
46.2
32.8
13.0
11.7
PRE
95.5
2.4
93.6
1K
2K
4K
8K
16K
32K
1K
2K
4K
UK
16K
326
Inference Context
Inference Context Length
Figure 5: Positional Bias. Comparison of PERK and FT-ICR on 4K and 8K contexts, on Qwen-2.5 2.5-0.5B. We train on problems where the relevant information appears in the beginning (Pre), middle (Mid), or end (Post) of the context, and evaluate on all three positional settings. We also train mod- cls on contexts where the relevant infor- mation is randomly located (Rnd), testing these on all four positional distributions (Pre, Post, Mid, Rnd). Bordered cells show in-distribution performances. PERK demonstrates strong positional robustness.
(a) linglehop reasoning
(b) Two-hopreasoning
Figure 4: Test-time context length robustness eval- uation on BabiLLong QAI (a) and QA2 (b) between PERK and FT-ICR on the Qwen-2.5-0.5B model. The y-axis represents the training context lengths, while the x-axis indicates various test-time context lengths. We test for both test lengths shorter than the training length, and test lengths longer than the training length. Bordered cells denote evaluation context lengths equal to those in training. PERK shows stronger ro- bustness across both settings.
outperforms FT-ICR by 24%. Furthermore, FT-ICR shows large performance drops when the relevant document consistently appears at different locations (Pre, Mid, and Post) at test time. Meanwhile, shifting relevant positions at test-time shows minimal effect (within 1-2%) on PERK.
To further stress-test positional generalization, we also force relevant documents into specific positions during training and testing, spanning the full context. We find that FT-ICR casily overfits to the position pattern, completely failing to generalize to test-time position changes (performance drops to close to 0% when the position shifts at test time). Although PERK underperforms FT-ICR when the train-test positions are the same and the beginning (Pre) or end (Post) of the context, performance remains consistent regardless of the positional distribution shift between train and test time. We attribute PERK's robustness in this setting to the fact that documents in the long context are encoded as members of permutation-invarian absolute position in the sequence. Meanwhile, FT-ICR directly aligns specific positions with answers during training.
batch into the parameter space, limiting the overall impact of
5.3 INFERENCE LATENCY AND MEMORY USAGE
We evaluate PERK's efficiency on long contexts by measuring its inference memory cost and run-tim run-time, compared to in-context reasoning with finetuned models (FT-ICR).
Setup We complete all runs a single Nvidia H100 GPU with 93 GB of VRAM and with Huggingface's ' Transformers library version 4.51.3. We perform four gradient update steps for the test-time inner loop adaptations for PERK. At inference time, all methods generate 64 tokens through greedy decoding using Huggingface's s text generation pipeline. We measure runtime complexity in seconds (wall-clock time) and memory complexity in gigabytes. Both methods undergo 10 hardware warm-up up iterations.
As explained in Section 3, PERK splits long sequences into batches of shorter segments. In this experiment, each batch has segments with an effective context length of 128 tokens. Since PERK performs bradeent-based encoding over these batches, we can apply gradient accumulation to trade
8

---

Published as a conference paper at ICLR 2026
PERK PERK Accum
Accum
Accum
A
FTACR
Inference Memory Complexity (GB)
Inference Runtime Complexity (s)
100
35
30
75
25
Memory (GB)
Runtime
10
1K
2K
4K
8K 16K 32K 64K 128K
1K
2K
4K
16K 32K Context Length
8K
64K 128K
Context
Figure 6: Inference-time memory footprint and wall-clock runtime context length increases (up to 128K tokens), comparing PERK with FT-ICR. Curves that terminate before 128K indicate the method failed with an OOM error at longer context lengths, preventing further measurement. PERK demonstrates more efficient scaling in both memory and runtime, particularly for longer sequences.
runtime for reduced memory consumption. We thus analyze PERK's efficiency across varying gradient accumulation steps, ranging from 2 to 16.
PERK scales more efficiently in memory and runtime on extremely long contexts. Figure 6 shows that PERK provides more efficient memory and runtime scaling for extremely long contexts compared to FT-ICR. While FT-ICR is initially more efficient, its memory and runtime grow rapidly, leading to OOM errors at context length of 128K. In contrast, PERK can manage the long sequences through gradient accumulation, which, while increasing runtime, reduces the memory footprint. For instance, at 8K tokens, increasing the accumulation steps from 1 to 16 reduces memory usage from 35.2GB to 5. 5.9GB, although the runtime increases from 1.9s to 8.5s. Crucially, even with this runtime trade-off, P PERK runs faster than FT-ICR at very long contexts where both can operate. At 64K tokens, FT-ICR takes 32.6s and 55. 5BB7GB, whereas PERK with 16 steps of accumulation completes in 11 11.4s, using only 19.6GB. With 8 steps, it is even faster at 9.7s, using 35.2 GB. Ultimately, at 128K tokens, where FT-ICR fails, PERK with 16 steps successfully processes the context using 35.2GB in 20.9s, showing that PERK provides practical path to handle extreme context lengths efficiently in both memory and runtime compared to standard approaches.
6 RELATED WORK
Test-Time Learning. MAML (Finn et al., 2017) introduces optimization-baser model-agnostic meta-learning for few-shot adaptation. Chen et al. (2023b) applies MAML to GPT models, integrating knowledge into model parameters through test-time gradient updates for downstream reasoning. Other methods achieve test-time learning in the model architecture. Sun et al. (2025) proposes RNN layers with learnable hidden states updated via self-supervised gradients test inputs. Titans (Behrouz et al., 2024) incorporates a differentiable memory with transformers for rapid inference- time memorization. Building on Titan, ATLAS (Behrouz et al., 2025) proposes new transformer architecture that contains the differentiable memory module as part of the network to support large- scale distributed pretraining. Other approaches adapt through context alone. MetalCL (Min et al., 2022) meta- meta-llerms from in-context examples without gradient updates. Our method, PERK, introduces crrameter-effiienn meta-leannnn using lightweight adapters, improving scalability for long contexts.
Long-Context Language Models. Developing long-context models has become a major research topic of its own. Many works explore extending transformer-he based language models' context windows with minimal training, either by position extrapolation (Chen et al., 2023a; Peng et al., 2023c; Xiao et al., 2024: Yen et al., 2024) or improving the transformer's attention mechanism (Bertsch et al., 2023; Mohtashami & Jaggi, 2023; Shen et al., 2023; Jin et al., 2024). Other works show that using the default attention. applying simple position extrapolation, and fine-tuning the model on long wethheeccallyggenerrteee
documents (Fu et al., 2024; Lu et al., 2024) Xiong et al., 2024) yields much stronger results. The FT-ICR baseline in our work follows this paradigm. To reduce the expansive computation cost of long-context training, Chen al. (2024)'s
long data (An et al., 2024b;

---

Published as a conference paper at ICLR 2026
work combines LoRA with trainable embedding and normalization to achieve efficient finetuning on long contexts. Other LLM works (Lieber et al., 2024; Grattafiori et al., 2024; Gao et al., 2025b) propose achieving longecontext capabilities through a long-contex: continued training stage between standard pre-training and supervised fine-tuning. In contrast to all the approaches above, PERK achieves long-context reasoning through the ability to encode long sequences using test-time gradient updates on a lightweight model adapter.
Long-Contee Architectures. In parallel, there have been many efforts designing more efficient architectures, for example, linear attention with RNNs (Chandrayan et al., 2023; Peng et al., 2023a; Gu & Dao, 2024; Dao & Gu, 2024; Yang et al., 2024; Beck et al., 2024) and alternative attention architectures (Rubin & Berant, 2024; Sun et al., 2024; Han et al., 2023). However, these new architectures often require training from scratch, and many have inherent limitations in terms of long-context recall (Jelassi et al., 2024; Arora et al., 2025). Recent works also explore hybrid models (Lieber al., 2024; De et al., 2024) or distilling existing L.LMs into hybrid models (Wang et al., 2024) and show promising results. In comparison, PERK directly augments off-the-shelf pre-trained LLMs without requiring additional architecture modifications or extensive pre-training, while naturally being parameter-efficient
Long-Context Evaluation. Many benchmarks have been proposed to evaluate language models' long-context abilities (Krishna et al., 2023; Shaham et al., 2023; Bai et al., 2024; Zhang et al., 2024b; Hsieh et al., 2024; Yen et al., 2025; Ye et al., 2025; Xia et al., 2025). Some works focus on the impact of extending input lengths with irrelevant contexts (Kuratov al., 2024; Levy et al., 2024) or changing relevant information positions (Liu et al., 2024) on LLM reasoning. Many works also tasks, such as retrieval (Lee et al., 2024; Qiu et al., 2025), summarization (Kim et al., 2024), and in-context learning (Li et al., 2024: Xu et al., 2024; Bertsch al., 2025), as long-context challenges. In this work, we define Drops-in- toops-in-the tee-thh-Oeeee as new long-context evaluation that makes distractors distributionally similar to relevant information.
reframe natural- -awaaaangnggggessssssssee
7 CONCLUSION
In this work, we introduce PERK, parameter-efficien meta-learning approach that learns to perform long-context reasoning by encoding information into lightweight adapter through test-time gradient updates. Our experiments on multiple long-context benchmarks predle-in-the-Haystee TriviaQA, Sepps-in-the-ecan and Code API retrieval) demonstrate substantial performance gains over models finetuned on long contexts for classical in-context reasoning across reasoning complexity and context lengths. We demonstrate that PERK's performance robustly transfers across model scales (0.5B to 8B) and model families (GPT-2, Qwen2.5, LLaMA-3). Our generalization analysis highlights P PERK's strong test-time length extrapolation ability, where the model generalize to unseen contexts 32x longer than the training data. When relevant information shifts positions at test-time, PERK robustly generalizes across different positions. Overall, PERK excels in performance, generalization, and robustness for long-context reasoning.
HotpotQA,
ACKNOWLEDGMENTS
We gratefully acknowledge the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29) the EPFL Center for Imaging, Sony Group Corporation, and a Meta LLM Evaluation Research Grant.
REFERENCES
Chenxin An, Fei Huang, Jun Zhang, Shansan Gong, Xipeng Qiu, Chang Zhou, and Lingpeng Kong. Training-free long-context scaling of large language models, 2024a. URL https: /tapps:///xxxvv //axxiv. org/abs 2402. 11:42.114500 17463.
Shengnan An, Zexiong Ma. Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. Make your Ilm fully utilize the context. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Newral In- formation Processing Systems, volume 37, pp- 62160-62188. Curran Associates, Inc., 2024b. URL https: httppp/:/pppcceeeinneee neurips, cc/papeecppppeeeeeeeeeeeeee /ttpurppscccpppppee_eeeeee//eeee www.s//pppeeeeeeeeeee file/71c3451 ffdd fadd F
httpoceecaiineeeuurpps.c/ca/aaae Can.rpperrreaaeeeeeeeeee pdd..
10

---

Published as a conference paper at ICLR 2026
Cem Anil, Yuhuai Wu. Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Etham Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546-38556, 2022.
Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughyg tradeoff. arXiv preprint arXiv:2402.18668, 2025. org/
abs axg/abs//
222222222.. 18668.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection, 2023. URL /ttp:p::////vv...///e org/
abs/2310 11511.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong. Jie Tang, and Juanzi Li. Longbench: A bilingual, multitask benchmark for long context understanding, 2024. URL https: /ttppps///axii.../// org abs/ 2308. 14508.
Deniz Bayazit, Negar Foroutan, Zeming Chen, Gail Weiss, and Antoine Bosselut. Discovering knowledge-critical subnetworks in pretrained language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. Association for Computational Linguistics. doi: 10. 12218665//12224ccm/// https: / aacclcl antho ogy orgg 2024 emn 376/
Miami, Florida, USA, November 2024. URL
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. arXiv preprint arXiv:2405.04517..20 2024. URL URLhttts://axxi //tpiv..///xx/v/////// abs / 205.. 04517.
Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time. arXiv preprint arXiv:2501.000 .00663 2024. URL https:://aaatee /ttp:///axxi...r//abbs//s///// htargg/ab//5555 00663.
Ali Behrouz, Zeman Li, Praneeth Kacham, Majid Daliri, Yuan Deng, Peilin Zhong, Meisam Raza- viyayn, and Vahab Mirrokni. Atlas: Learning to optimally memorize the context at test time, 2025.
URL htRps://:/ arxiv. org/ abs 23735.
Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew Gormley. Unlimiformer: Long-range transformers with unlimited length input. In A. Oh, T. Naumann, A. Globerson, Saenko, M. Hardt, and S. Levine (eds.), Advances Neural In- formation Processing Systems, volume 36, pp. 35522-35543. Curran Associates, Inc., 2023. URL htttss:/pproceediiiii wwwpeeeeeeddnnnennenneeeeeeeeeee Fri1:AF8Boaaaa222333333772702020
http:/pppp...ppppp....eee//appee pdf.
Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint arXiv:2405.0022 2025. URL /tt..tttss////rrvvo.rr//////
22000 00200.
Kedar Chandrayan, Lance Martin, Greg Kamradt, Lazaro Hurtado, Arkady Arkhangorodsky, Lazaro Hurtado, and Pavel Kral. Needle in a haystack pressure testing 1lms. 2023.
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. /5595, 2023a. URL https: /arxivv
org/ abs 2306. 15595.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine- tuning of long-context large language models, 2024. URL https: httpttsss//axxv..xttte abs/2300 12307.
Zeming Chen, Gail Weiss, Eric Mitchell, Asli Celikyilmaz, and Antoine Bosselut. Reckoning: reasoning through dynamic knowledge encoding. Advances in Neural Information Processing Systems, 16:622779-66616 2023b.
11

---

Published as a conference paper at ICLR 2026
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers soft reasoners over language. In Christian Bessiere (ed.), Proceedings of the Twenty-Ning Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3882-3892 ijcai.org, 2020. doi: (0.24966/JUCAA2220553. URL https: / /ooi ooo..oog/100 2.4 963 ccai wta.cci...0...... 22020/537.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint 2rXiv:2405.21060, 2024. URL https:
// //arxiv..rr/ org absaabs//5000 2400 21060.
Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Murane Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint arXiv:2402.19427, 2024. URL https: /aaxxvivvvv www.iiivvrrr...ooo abs 2402. 19427.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning. pp. 1126-1135. PMLR, 2017.
Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. Data engineering for scaling language models to 128k context. arXiv preprint 10011:240..0017,, 2024. URL https: : / arxiv. org/ abs 02402.10772
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively), 2025a. httht: /rr/ at/////s//1000 2410.. 02660.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprini arXiv:2410.02660, 2025b. URL https:::: pttttts://axxx...ot/ http//xxxvvv...///// abs/24100 02660.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lec, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli. Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si. Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale,
12

---

Published as a conference paper at ICLR 2026
Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman. Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaclle Goldschlag, Yashesh Gaur, Yasmine Babaci, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ranchandani, Annic Dong. Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer. Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine. Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminaray.anan. Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, -apt-Sae Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatr, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina. Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto. Sharadh Ramaswany, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang. Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao,
13

---

Published as a conference paper at ICLR 2026
Yundi Qian, Yunlu Li, Yuzi He. Zach Rait, Zachary Vito,o, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang. Zhiwei Zhao, and Zhiyu Ma. The llama herd of models. arXiv preprint arXiv:2407. 2024. URL https:://///.. wawpiv.vvvv.. wwwpiiii./..oooooooooooooooooooe owg.oo////ao.o.. aaceg////////// /2407 22777.17......
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00753 2024. URL
wwtpaawwww.....aaaaaaaeaa /ttp://bb//2//// 00752.
Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab S. Mirronni, David P. Woodruff, and Amir Zandieh. Hyperattention: Long-context attention in near-linear time. ArXiv, abs/2310.0586/ 2023. URL
httpss:/aape wew.:essssaaniccchol.eeoe www.aaar.rrrCooocccooeaa 22232uu11::2238124000
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654 2024. URL https: /arxivv.vvvv.///////////////aav/ www.aiiorgg...o org //s///////// 2404. 06654.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Nathan Hu, Eric Mitchell, Christopher D. Manning, and Chelsea Finn. Meta-learning on- line adaptation of language models. arXiv preprint 2rXiv:2403.04317, 2023. URL https:
//arxiv. org/ abs httpg//bb/2350...22 /tt..//3........
Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Trans- formers are better than state space models at copying. arXiv preprint arXiv:2402.01032, 2024.
URL wwt.h:tpp///m/ /tls:///rrae //tpxaaavvv////////a///ee///aavl org/ abs/ 01032.
Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia- Yuan Chang, Huiyuan Chen, and Xia Hu. Llm maybe longlm: Self-extend IIm context window without tuning. arXiv preprint ar 2411v2240..11355, 2024. URL https: /ttttp:://ariiv..o// wwg./bs///aa/....o..o..........s abs///as..1111. 01325.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017, Association for Computational I Linguistics. doi: 10.18655v1//17--1147 URL https: / oaaelaannhhooooyyoooro/771111111
org/p17-1147/.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.033332 2023. URL https: /ttp:/:///xxvvvvo/ org/ /bss2211... 233..
Petros Karypis, Julian McAuley. and George Karypis. Extending input contexts of language models through training on segmented sequences. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACI. 2024, pp. 3040-3052 Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.11853/v1/200 1/202/./1222//// htt.://inn.//...nn--/.../ findings-naacl.199. URL https::/aclantthooogg htw.://aaaatoooloooooo///2/2220 findings naacl... 191//
Yekyung Kim, Yapei Chang. Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit lyyer. Fables: Evaluating faithfulness and content selection in book-length summarization, 2024. URL /
012661
Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit lyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. Longeval: Guidelines for human evaluation of faithfulness long-form summarization, 2023.
URL https: http///axivvv.. org/ a23/ 1 13298.
Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context iaaainngii---ahsstacll Advances in Neural Information Processing Systems, 37:106519-10653 2024.
14

---

Published as a conference paper at ICLR 2026
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411. 15124, 2025. URL httph:th////mmm /twp/aaaivvvv...///v//ea.eaaaavv org/ abs/ 2//.2111..1111.
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, King-Wei Chang, and Kelvin Guu. Can long-context language models subsume retrieval, rag. sql, and more? arXiv preprint arXiv:2406. 13/2/, 2024. URL //tp////////..ttt/////////////// 2447g//bb/244..... 13121.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact input length on the reasoning performance of large language models. arXiv preprini urXiv:2402.14885 2024.
URL https::p///// /taps://aaxiii org/ abs 2402 14848.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yue, and Wenhu Chen. Long-context ilms struggle with long in-context learning. arXiv preprint arXiv:2404.020660, 2024.
abs 0bs/24044 02060.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mambe language model. arXiv preprint arXiv:2403. 19887, 2024. URL https: arxiv. org/ /bbs/ 2403. 1408.79887.
Bingbin Liu, Jordan Ash, Surhhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Exposing attention glitches with flip-flop language modeling. Advances in Neural Information Processing Systems, 28.25549-25583, 2023.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Finewel-edu: the finest collection of educational content. 2024. doi: 10.57967/hf2497.. URL URLhttpss/huuggiigf..s ca/ se/daaaseets
Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, and Alexander M. Rush. A controlled study on long context extension and generalization in 1mm. arXiv preprint arXiv:2409. 12181, 12181.
URL
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. http:///1//222.aallmmoo 201. /aal::///laaaaae URL https: //cl/cllcc/ wowcclnnthooggoo org/ 2022 naaci-ma n.. 201/
Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. In Thirty-sevnnth Conference Neural Information Processing Systems, 2023. URL https: // //openreview..oo www.rooorudddmamm 6wOVy.
OpenAl. Gpt-4.1, https: //ppnaiipp//aai//// com, 2025. Large language model.
Shishir G Patil, Tianjun Zhang. Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544-126515 2024.
15

---

Published as a conference paper at ICLR 2026
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao. Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Jiaju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Bolun Wang, Johan S. Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305. 13048, 2023a. URL https: /ttpes:/aaavvvv.oo org /aa///////a/aa/ / 2 305 13048.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023b. URL htttpp://aaaav..orraabb////
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yam:: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00077 2023c. URL https: (aaxxii..rg/aaaa. haapg//bs/aa /209 00071.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables imput length extrapolation. arXiv preprint arXiv:2108. 12409, 2022. URL https: //arxiv.... wwe.iiv.rg//... abs 2108. 12409.
Yifu Qiu, Varun Embar, Yizhe Zhang, Navdeep Jaitly, Shay B. Cohen, and Benjamin Han. Elic- iting in-context retrieval and reasoning for long-context large language models. arXiv preprint arXiv:2501.08245 2025. URL https: arxiv. org/ abs 2501. 08248.
Qwen, An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint www.ii..../raaaoo.oo.
arXiv:2412.15115. 2025. URL https: / //aa/vivvv..///// org/ abs 11..411111111 15115.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. URL https: /api. weweetiicccooleee
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge you pack into the parameters of a language model? In Bonnie Webher, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 1188866//1/220..ooo /2tp0..000mmmmaii.......m URL https: http///aaaaanttoooo.oaoo. org/2020. emnlp-mainn in. 437/.
Ohad Rubin and Jonathan Berant. Retrieval- transformer: Long-range language modeling with self-retrieval. Transactions of the Association for Computational Linguistics, 12:1197-1213, 2024. doi: 10.1162/tacl a_093... URL ht / aaclantoooantoooo..aaa org/ orgg/022 2024. tacl-1. 66/.
Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back prok-propngttion for bilevel optimization. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Sccond International Conference Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1723-1732, PMLR, 16-18 Apr 2019. URL https:/ httpppoocediingee/e mll. press v 9 1089//haae wwssanaaaannn. 9a .tml.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7977-7989, Sin- gapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/w1/2023. madingg-emmnp.536 URL https: /accaantoooogyyyo.oco.oo hwwphhhoololyoogg////o2//e wrgg findings emnlp. 536/.
Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. arXiv preprint arXiv:2302.06461. 2023. URL https: //ta:////////a//a//aa aarxiv.orgg abs/2 61.
16

---

Published as a conference paper at ICLR 2026
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo. Tatsumori Hashimoto, and Carlos Guestrin. Learning to (learn at test time): Rnns with expressive hidden states. arXiv preprint arXhv:2407.046200 2025. URL https:://///. //tt///raaa www.ii.orrgo....o abs/ 04620.
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang. Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 7339-7361. Curran Asso- ciates, Inc., 2024. URL ht http:::ps//ppcceeediinii n httpeaacpppscepppppeeeeeeeeeeeee files/paper/ 2024 htt//file// (tff http///eeeeE22cc000222277eeeeeee
pdf.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang. Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, Andrea Tacchetti, Colin Gaffney, Samira Daruki, Olcan Sercinoglu, Zach Gleicher, Juliette Love, Paul Voigtlaender, Rohan Jain, Gabriela Surita, Kareem Mohamed, Rory Blevins, Junwhan Ahn, Tao Zhu, Kornraphop Kawintiranon, Orhan Firat, Yiming Gu, Yujing Zhang, Matthew Rahtz, Manaal Faruqui, Natalie Clay, Justin Gilmer, JD Co-Reyes, Ivo Penchev, Rui Zhu, Nobuyuki Morioka, Kevin Hui, Krishna Haridasan, Victor Campos. Mahdis Mahdich, Mandy Guo, Samer Hassan, Kevin Kilgour, Arpi Vezer, Heng-Tze Cheng, Raoul de Liedekerke, Siddharth Goyal, Paul Barham, DJ Strouse, Seb Noury, Jonas Adler, Mukund Sundararajan, Sharad Vikram, Dmitry Lepikhin, Michela Paganini, Xavier Garcia, Fan Yang. Dasha Valter, Maja Trebacz, Kiran Vodrahalli, Chulayuth Asawaroengchai, Roman Ring, Norbert Kalb, Livio Baldini Soares, Siddhartha Brahma, David Steiner, Tianhe Yu, Fabian Mentzer, Antoine He, Lucas Gonzalez, Bibo Xu, Raphael Lopez Kaufman, Laurent EI Shafey, Junhyuk Oh, Tom Hennigan, George van den Driessche, Seth Odoom, Mario Lucic, Becca Roelofs, Sid Lall, Amit Marathe, Betty Chan, Santiago Ontanon, Luheng He, Denis Teplyashin, Jonathan Lai, Phil Crone, Bogdan Damoc, Lewis Ho. Sebastian Riedel, Karel Lenc, Chih-Kuan Yeh, Aakanksha Chowdhery, Yang Xu, Mehran Kazemi, Ehsan Amid, Anastasia Petrushkina, Kevin Swersky, Ali Khodaei, Gowoon Chen, Chris Larkin. Mario Pinto, Geng Yan, Adria Puigdomenech Badia, Piyush Patil, Steven Hansen, Dave Orr, Sebastien M. R. Arnold, Jordan Grimstad, Andrew Dai. Sholto Douglas, Rishika Sinha, Vikas Yadav, Xi Chen, Elena Gribovskaya, Jacob Austin, Jeffrey Zhao, Kaushal Patel, Paul Komarek, Sophia Austin, Sebastian Borgeaud, Linda Friso, Abhimanyu Goyal, Ben Caine, Kris Cao, Da-Woon Chung, Matthew Lamm, Gabe Barth-Maron, Thais Kagohara, Kate Olszewska, Mia Chen, Kaushik Shivakumar, Rishabh Agarwal, Harshal Godhia, Ravi Rajwar, Javier Snaider, Xerxes Dotiwalla, Yuan Liu, Aditya Barua, Victor Ungureanu, Yuan Zhang, Bat-Orgil Batsaikhan, Mateo Wirth, James Qin, Ivo Danihelka, Tulsee Doshi, Martin Chadwick, Jilin Chen, Sanil Jain, Quoe Le, Arjun Kar, Madhu Gurumurthy, Cheng Li, Ruoxin Sang, Fangyu Liu, Lampros Lamprou, Rich Munoz, Nathan Lintz, Harsh Mehta, Heidi Howard, Malcolm Reynolds, Lora Aroyo, Quan Wang, Lorenzo Blanco, Albin Cassirer, Jordan Griffith, Dipanjan Das, Stephan Lee, Jakub Sygnowski, Zach Fisher, James Besley, Richard Powell, Zafarali Ahmed, Dominik Paulus, David Reitter, Zalan Borsos, Rishabh Joshi, Aedan Pope. Steven Hand, Vittorio Selo, Vihan Jain, Nikhil Sethi, Megha Goel, Takaki Makino, Rhys May. Zhen Yang, Johan Schalkwyk, Christina Butterfield, Anja Hauth, Alex Goldin, Will Hawkins, Evan Senter, Sergey Brin, Oliver Woodman, Marvin Ritter, Eric Noland, Minh Giang, Vijay Bolina, Lisa Lee, Tim Blyth, Ian Mackinnon, Machel Reid, Obaid Sarvana, David Silver, Alexander Chen, Lily Wang, Loren Maggiore, Oscar Chang, Nithya Attaluri, Gregory Thorton, Chung-Cheng Chiu, Oskar Bunyan, Nir Levine, Timothy Chung, Evgenii Eltyshev, Xiance Si, Timothy Lillicrap, Demetra Brady, Vaibhav Aggarwal, Boxi Wu, Yuanzhong Xu, Ross Mcllroy, Kartikeya Badola, Paramjit Sandhu, Erica Moreira, Wojciech Stokowiec, Ross Hemsley, Dong Li, Alex Tudor, Pranav Shyam, Elahe Rahimtoroghi, Salem Haykal, Pablo Sprechmann, Xiang Zhou, Diana Mincu, Yujia Li, Ravi Addanki, Kalpesh Krishna, Xiao Wu, Alexandre Frechette, Matan Eyal, Allan Dafoe, Dave Lacey, Jay Whang, Thi Avrahami, Ye Zhang, Emanuel Taropa, Hanzhao Lin, Daniel Toyama, Eliza Rutherford, Motoki Sano, Hyunleong Choe, Alex Tomala, Chalence Sefrranek-Shrader Nora Kassner, Mantas Pajarskas, Matt Harvey, Sean Sechrist, Meire Fortunato, Christina Lyu, Gamaleldin Elsayed, Chenkai Kuang, James Lottes, Eric Chu, Chao Jia, Chih-Wer Chen, Peter Humphreys, Kate Baumli, Connie Tao, Rajkumar Samuel, Cicero Nogueira dos Santos, Anders Andreassen, Nemanja Rakićevié, Dominik Grewe, Aviral Kumar, Stephanie Winkler, Jonathan Caton, Andrew Brock, Sid Dalmia, Hannah Sheahan, lain Barr, Yingjie Miao, Paul Natsev, Jacob Devlin, Feryal Behbahani, Flavien Prost, Yanhua Sun, Artiom Myaskovsky,
17

---

Published as a conference paper at ICLR 2026
Thanumalayan Sankaranarayana Pillai, Dan Hurt, Angeliki Lazaridou, Xi Xiong, Ce Zheng. Fabio Pardo, Xiaowei Li, Dan Horgan, Joe Stanton, Moran Ambar, Fei Xia, Alejandro Lince, Mingqiu Wang, Basil Mustafa, Albert Webson, Hyo Lee, Rohan Anil, Martin Wicke, Timothy Dozat, Abhishek Sinha, Enrique Piqueras, Elahe Dabir, Shyam Upadhyay, Anudhyan Boral, Lisa Anne Hendricks, Corey Fry, Josip Djolonga, Yi Su, Jake Walker, Jane Labanowski, Ronny Huang. Vedant Misra, Jeremy Chen, RJ Skerry-Ryan, Avi Singh, Shruti Rijhwani, Dian Yu, Alex Castro-Ros Beer Changpinyo, Romina Datta, Sumit Bagri, Arnar Mar Hrafnkelsson, Marcello Maggioni, Daniel Zheng, Yury Sulsky, Shaobo Hou, Tom Le Paine, Antoine Yang, Jason Riesa, Dominika Rogozinska, Dror Marcus, Dalia EI Badawy, Qiao Zhang, Luyu Wang, Helen Miller, Jeremy Greer, Lars Lowe Sjos, Azade Nova, Heiga Zen, Rahma Chaabouni, Mihaela Rosca, Jiepu Jiang, Charlie Chen, Ruibo Liu, Tara Sainath, Maxim Krikun, Alex Polozov, Jean-Baptiste Lespiau, Josh Newlan, Zeyncep Cankara, Soo Kwak, Yunhan Xu, Phil Chen, Andy Coenen, Clemens Meyer. Katerina Tsihlas, Ada Ma. Juraj Gottweis, Jinwei Xing, Chenjic Gu, Jin Miao, Christian Frank, Zeynep Cankara, Sanjay Ganapathy, Ishita Dasgupta, Steph Hughes- Heng Chen, David Reid, Keran Rong, Hongmin Fan, Joost van Amersfoort, Vincent Zhuang, Aaron Cohen, Shixiang Shane Gu, Anhad Mohananey, Anastasija Ilie, Taylor Tobin, John Wieting, Anna Bortsova, Phoebe Thacker, Emma Wang, Emily Caveness, Justin Chiu, Eren Sezener, Alex Kaskasoli, Steven Baker, Katie Millican, Mohamed Elhawaty, Kostas Aisopos, Carl Lebsack, Nathan Byrd, Hanjun Dai, Wenhao Jia, Matthew Wiethoff, Elnaz Davoodi, Albert Weston, Lakshman Yagati, Arun Ahuja, Isabel Gao, Golan Pundak, Susan Zhang, Michael Azzam, Khe Chai Sim, Sergi Caclles, James Keeling, Abhanshu Sharma, Andy Swing, YaGuang Li, Chenxi Liu, Carrie Grimes Bostock, Yamini Bansal, Zachary Nado, Ankesh Anand, Josh Lipschultz, Abhijit Karmarkar, Lev Proleev, Abe Ittcheeiihh, Soheil Hassas Yeganeh, George Polovets, Aleksandra Faust, Jiao Sun, Alban Rrustemi, Pen Li, Rakesh Shivanna, Jeremiah Liu, Chris Welty, Federico Lebron, Anirudh Baddepudi, Sebastian Krause, Emilio Parisotto, Radu Soricut, Zheng Xu, Dawn Bloxwich, Melvin Johnson, Behnam Neyshabur, Justin Mao-Jones, Renshen Wang, Vinay Ramasesh, Zaheer Abbas, Arthur Guez, Constant Segal, Due Dung Nguyen, James Svensson, Le Hou, Sarah York. Kieran Milan, Sophie Bridgers, Wiktor Gworek, Marco Tagliasacchi, James Lee-Thorp. Michael Chang, Alexey Guseynov, Ale Jakse Hartman, Michael Kwong, Ruizhe Zhao, Sheleem Kashem, Elizabeth Cole, Antoine Miech, Richard Tanburn, Mary Phuong, Filip Pavetic, Sebastien Cevey, Ramona Comanescu, Richard Ives, Sherry Yang. Cosmo Du, Bo Li, Zizhao Zhang. Mariko linuma, Clara Huiyi Hu, Aurko Roy, Shaan Bijwadia, Zhenkai Zhu, Danilo Martins, Rachel Saputro, Anita Gergely, Steven Zheng, Dawei Jia, Ioannis Antonoglou, Adam Sadovsky, Shane Gu, Yingying Bi, Alek Andreev, Sina Samangooei, Mina Khan, Tomas Kocisky, Angelos Filos, Chintu Kumar, Colton Bishop, Adams Yu, Sarah Hodkinson, Sid Mittal, Premal Shah, Alexandre Moufarek, Yong Cheng, Adam Bloniarz, Jaehoon Lee, Pedram Pejman, Paul Michel, Stephen Spencer, Vladimir Feinberg, Xuehan Xiong, Nikolay Savinov, Charlotte Smith, Siamak Shakeri, Dustin Tran, Mary Chesus, Bernd Bohnet, George Tucker, Tamara von Glehn, Carric Muir, Yiran Mao, Hideto Kazawa, Ambrose Slone, Kedar Soparkar, Disha Shrivastava, James Cobon- Cobon-Kerr. Michael Sharman, Jay Pavagadhi, Carlos Araya, Karolis Misiunas, Nimesh Ghelani, Michael Laskin, David Barker, Qiujia Li, Anton Briukhov, Neil Houlsby, Mia Glaese Balaji Lakshminarayanan, Nathan Schucher, Yunhao Tang, Eli Collins, Hyeontaek Lim, Fangxiaoyu Feng, Adria Recasens, Guangda Lai, Alberto Magni, Nicola De Cao, Aditya Siddhant, Zoe Ashwood, Jordi Orbay, Mostafa Dehghani, Jenny Brennan, Yifan He, Kelvin Xu. Yang Gao, Carl Saroufim, James Molloy, Xinyi Wu, Seb Amold, Solomon Chang. Julian Schrittwieser, Elena Buchatskaya. Soroush Radpour, Martin Polacek, Skye Giordano, Ankur Bapna, Simon Tokumine, Vincent Hellendoorn, Thibault Sottiaux. Sarah Cogan, Aliaksei Severyn, Mohammad Saleh, Shantanu Thakoor, Laurent Shefey, Siyuan Qiao, Meenu Gaba, Shuo yiin Chang. Craig Swanson, Biao Zhang, Benjamin Lee, Paul Kishan Rubenstein, Gan Song, Tom Kwiatkowski, Anna Koop, Ajay Kannan, David Kao, Parker Schuh, Axel Stjerngren, Golnaz Ghiasi, Gena Gibson, Luke Vilnis, Ye Yuan, Felipe Tiengo Ferreira, Aishwarya Kamath, Ted Klimenko, Ken Franko, Kefan Xiao. Indro Bhattacharya, Miteyan Patel, Rui Wang, Alex Morris, Robin Strudel, Vivek Sharma, Peter Choy, Sayed Hadi Hashemi, Jessica Landon, Mara Finkelstein, Priya Jhakra, Justin Frye, Megan Barnes, Matthew Mauger, Dennis Daun, Khuslen Baatarsukh, Matthew Tung, Wael Farhan, Henryk Michalewski, Fabio Viola, Felix de Chaumont Quitry, Charline Le Lan, Tom Hudson, Qingze Wang, Felix Fischer, Ivy Zheng, Elspeth White, Anca Dragan, Jean baptiste Alayrac, Eric Ni, Alexander Pritzel, Adam Iwanicki, Michael Isard, Anna Bulanova, Lukas Zilka, Ethan Dyer, Devendra Sachan, Srivatsan Srinivasan, Hannah Muckenhirn Honglong Cai, Amol Mandhane, Mukarram Tariq, Jack W. Rae, Gary Wang. Kareem Ayoub, Nicholas FitzGerald, Yao Zhao, Woohyun Han, Chris Alberti, Dan Garrette,
18

---

Published as a conference paper at ICLR 2026
Kashyap Krishnakumar, Mai Gimenez, Anselm Levskaya, Daniel Sohn, Josip Matak, Inaki Iturrate, Michael B. Chang, Jackie Xiang, Yuan Cao, Nishant Ranka, Geoff Brown, Adrian Hutter, Vahab Mirrokni, Nanxin Chen, Kaisheng Yao, Zoltan Egyed, Francois Galilee, Tyler Liechty, Praveen Kallakuri, Evan Palmer, Sanjay Ghemawat, Jasmine Liu, David Tao, Chloe Thornton, Tim Green, Mimi Jasarevic, Sharon Lin, Victor Cotruta, Yi-Xuan Tan, Noah Fiedel, Hongkun Yu, Ed Chi, Alexander Neitz, Jens Heitkaemper, Anu Sinha, Denny Zhou, Yi Sun, Charbel Kaed, Brice Hulse, Swaroop Mishra, Maria Georgaki, Sneha Kudugunta, Clement Farabet, Izhak Shafran, Daniel Vlasic, Anton Tsitsulin, Rajagopal Ananthanarayanan, Alen Carin, Guolong Su, Pei Sun, Shashank V, Gabriel Carvajal, Josef Broder, Iulia Comsa, Alena Repina, William Wong. Warren Weilun Chen, Peter Hawkins, Egor Filonov, Lucia Loher, Christoph Hirnschall, Weiyi Wang, Jingchen Ye, Andrea Burns, Hardie Cate, Diana Gage Wright, Federico Piccinini, Lei Zhang, Chu-Cheng Lin, lonel Gog, Yana Kulizhskaya, Ashwin Sreevatsa, Shuang Song, Luis C. Cobo, Anand Iyer, Chetan Tekur, Guillermo Garrido, Zhuyun Xiao, Rupert Kemp, Huaixiu Steven Zheng. Hui Li, Ananth Agarwal, Christel Ngani, Kati Goshvadi, Rebeca Santamaria-Fermatte Chris Gorgolewski, Sean Sun, Roopal Garg, Xinyu Ye, S. M. Ali Eslami, Nan Hua, Jon Simon, Pratik Joshi, Yelin Kim, Ian Tenney, Sahitya Potluri, Lam Nguyen Thiet, Quan Yuan, Florian Luisier, Alexandra Chronopoulou, Salvatore Scellato, Praveen Srinivasan, Minmin Chen, Vinod Koverkathu, Valentin Dalibard, Yaming Xu, Brennan Saeta, Keith Anderson, Thibault Sellam, Nick Fernando. Fantine Huot, Junehyuk Jung, Mani Varadarajan, Michael Quinn, Amit Raul, Maigo Le. Ruslan Habalov, Jon Clark, Komal Jalan, Kalesha Bullard, Achintya Singhal, Thang Luong. Boyu Wang, Sujeevan Rajayogam, Julian Eisenschlos, Johnson Jia, Daniel Finchelstein, Alex Yakubovich, Daniel Balle, Michael Fink, Sameer Agarwal, Jing Li,, Dj Dvijotham, Shalini Pal, Kai Kang, Jaclyn Konzelmann, Jennifer Beattie, Olivier Dousse, Diane Wu, Remi Crocker, Chen Elkind, Siddhartha Reddy Jonnalagadda, Jong Lee, Dan Holtmann- Krystal Kallarackal, Rosanne Liu, Denis Vnukov, Neera Vats, Luca Invernizzi, Mohsen Jafari, Huanjie Zhou, Lilly Taylor, Jennifer Prendki, Marcus Wu, Tom Eccles, Tianqi Liu, Kavya Kopparapu, Francoise Beaufays, Christof Angermueller, Andreea Marzoca, Shourya Sarcar, Hilal Dib, Jeff Stanway, Frank Perbet, Neje Trdin. Rachel Sterneck, Andrey Khorlin, Dinghua Li, Xihui Wu, Sonam Goenka, David Madras, Sasha Goldshtein, Willi Gierke, Tong Zhou, Yaxin Liu, Yannie Liang, Anais White, Yunjie Li, Shreya Singh, Sanaz. Bahargam, Mark Hpstein, Sujoy Basu, Li Lao, Adnan Ozturel, Carl Crous, Alex Zhai, Han Lu, Zora Tung, Neeraj Gaur, Alanna Walton, Lucas Dixon, Ming Zhang, Amir Globerson, Grant Uy, Andrew Bolt, Olivia Wiles, Milad Nasr, Ilia Shumailov, Marco Selvi, Francesco Piccinno, Ricardo Aguilar, Sara McCarthy, Misha Khalman, Mrinal Shukla, Vlado Galic, John Carpenter, Kevin Villela. Haibin Zhang. Harry Richardson, James Martens, Matko Bosnjak, Shreyas Rammohan Belle, Jeff Seibert, Mahmoud Alnahlawi, Brian McWilliams, Sankalp Singh, Annie Louis, Wen Ding, Dan popovici, Lenin Simicich, Laura Knight, Pulkit Mehta, Nishesh Gupta, Chongyang Shi, Saaber Fatehi, Jovana Mitrovic, Alex Grills, Joseph Pagadora. Tsendsuren Munkhdalai, Dessie Petrova, Danielle Eisenbud. Zhishuai Zhang, Damion Yates, Bhavishya Mittal, Nilesh Tripuraneni. Yannis Assacl, Thomas Brovelli, Pratcek Jain, Mihajlo Velimirovic, Canfer Akbulut, Jiaqi Mu, Wolfgang Macherey, Ravin Kumar, Jun Xu, Haroon Qureshi, Gheorghe Comanici, Jeremy Wiesner, Zhitao Gong, Anton Ruddock, Matthias Bauer, Nick Felt, Anirudh GP, Anurag Arnab, Dustin Zelle, Jonas Rothfuss, Bill Rosgen, Ashish Shenoy, Bryan Seybold, Xinjian Li, Jayaram Mudigonda, Goker Erdogan, Jiawei Xia, Jiri Simsa, Andrea Michi Yi Yao, Christopher Yew, Steven Kan, Isaac Caswell, Carey Radebaugh, Andre Elisseeff, Pedro Valenzuela, Kay McKinney, Kim Paterson, Albert Cui, Eri Chiorre Solomon Kim, William Zeng, Ken Durden, Priya Ponnapalli, Tiberiu Sosea, Christopher A. Mioquette-Choog James Manyika, Brona Robenek, Harsha Vashisht, Sebastien Pereira, Hoi Lam, Marko Velie, Denese Owusu-Afriyie, Katherine Lee, Tolga Bolukbasi, Alicia Parrish, Shawn Lu, Jane Park, Balaji Venkatraman, Alice Talbert, Lambert Rosique, Yuchung Cheng, Andrei Sozanschi, Adam Paszke, Praveen Kumar, Jessica Austin, Lu Li, Khalid Salama, Bartek Perz, Wooyeol Kim, Nandita Dukkipati, Anthony Baryshnikov, Christos Kaplanis, XiangHai Sheng, Yuri Chervonyi, Caglar Unlu, Diego de Las Casas, Harry Askham. Kathryn Tunyasuvunakool. Felix Gimeno, Siim Poder, Chester Kwak, Matt Miecnikowski, Vahab Mirrokni, Alek Dimitriev, Aaron Parisi, Dangyi Liu, Tomy Tsai, Toby Shevlane, Christina Kouridi, Drew Garmon, Adrian Goedeckemeyer, Adam R. Brown, Anitha Vijayakumar, Ali Elqursh, Sadegh Jazayeri, Jin Huang, Sara Mc Carthy, Jay Hoover, Lucy Kim, Sandeep Kumar, Wei Chen, Courtney Biles, Garrett Bingham, Evan Rosen, Lisa Wang, Qijun Tan, David Engel. Francesco Pongetti, Dario de Cesare. Dongseong Hwang, Lily Yu, Jennifer Pullman, Srini Narayanan, Kyle Levin, Siddharth Gopal, Megan Li, Asaf Aharoni, Trieu Trinh Jessica Lo, Norman Casagrande, Roopali Vij, Loic Matthey, Bramandia Ramadhana,
Wojciech Fica, Xinyun Chen,
19

---

Published as a conference paper at ICLR 2026
Austin Matthews, CJ Carey, Matthew Johnson, Kremena Goranova, Rohin Shah, Shereen Ashraf, Kingshuk Dasgupta, Rasmus Larsen, Yicheng Wang, Manish Reddy Vuyyuru, Chong Jiang, Joana Ijazi, Kazuki Osawa, Celine Smith, Ramya Sree Boppana, Taylan Bilal, Yuma Koizumi, Ying Xu, Yasemin Altun, Nir Shabat, Ben Bariach, Alex Korchemniy, Kiam Choo, Olaf Ronneberger, Chimezie Iwuanyanwu, Shubin Zhao, David Soergel, Cho-Jui Hsieh, Irene Cai, Shariq Iqbal, Martin Sundermeyer, Zhe Chen, Elie Bursztein, Chaitanya Malaviya, Fadi Biadsy, Prakash Shroff, Inderjit Dhillon, Tejasi Latkar, Chris Dyer, Hannah Forbes, Massimo Nicosia, Vitaly Nikolaev, Somer Greene, Marin Georgiev. Pidong Wang, Nina Martin, Hanie Sedghi, John Zhang, Praseem Banzal, Doug Fritz, Vikram Rao, Xuezhi Wang. Jiageng Zhang, Viorica Patraucean, Dayou Du, Igor Mordatch, Ivan Jurin, Lewis Liu, Ayush Dubey, Abhi Mohan, Janek Nowakowski, Vlad-Doru Ion, Nan Wei, Reiko Tojo. Maria Abi Raad, Drew A. Hudson, Vaishakh Keshava, Shubham Agrawal, Kevin Ramirez, Zhichun Wu, Hoang Nguyen, Ji Liu, Madhavi Sewak, Bryce Petrini, DongHyun Choi, Ivan Philips, Ziyue Wang, Ioana Bica, Ankush Garg, Jarck Wilkiewicz. Priyanka Agrawal, Xiaowei Li, Danhao Guo, Emily Xue, Naseer Shaik, Andrew Leach, Sadh MNM Khan, Julia Wiesinger, Sammy Jerome, Abhishek Chakladar, Alek Wenjiao Wang. Tina Ornduff, Folake Abu, Alireza Ghaffarkhah, Marcus Wainwright, Mario Cortes, Frederick Liu, Joshua Maynez, Andreas Terzis, Pouya Samangouei, Riham Mansour, Tomasz Kępa, François-Xavier Aubet, Anton Algymr, Dan Banica, Agoston Weisz, Andras Orban, Alexandre Senges, Ewa Andrejezuk, Mark Geller, Niccolo Dal Santo, Valentin Anklin, Majd Al Merey, Martin Baeuml, Trevor Strohman, Junwen Bai, Slav Petrov, Yonghui Wu, Demis Hassabis, Koray Kavukcuoglu, Jeff Dean, and Oriol Vinyals. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https: //wpvvv.avvvvv//aa//aa/eaariiiva org/ abss//////// 24003. 05530.
Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. In A. Globerson, L. Mackey, D. Belgrave. A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neu- ral Information Processing Systems, volume 37, pp. 62432-62451 Curran Associates, Inc., 2024. URL https: : ptw.eedi neurips cttppp.cccceeeeee cwtpccpeppeaaeeeeeeeeeeee wtless/pperr/2222 Coneeree pdf..
Menglin Xia, Victor Ruehle, Saravan Rajmohan, and Reza Shokri. Minerva: A programmable memory test benchmark for language models. arXiv preprint arXiv:2502.03358, 2025. URL petps://arxier /atp://////io..../aa///sss///s orgoaaos 2502. 03358.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2024. URL https:
Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, and Dimitris Papailiopoulos. From artificial needles real haystacks: Improving retrieval capabilities in lims by finetuning on synthetic data. arXiv preprint arXiv:2406. 0224406.19999 19292, 2024. URL https: //rxivxxvvv/v////////////////ava org/ aa////b/a/a/ /2406 19292.
Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long-context language models with lifelong icl and task haystack. arXiv preprint 2xxiv:2407.166990 2024. URL https: /ttppp:::////vvv////////l// www.v.orgg abs/ / 2407. 6695.
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou Junyang Lin, Kai Dang, Kexin Yang. Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1m technical report, 2025. URL https::///........ http//aaa...a/aaaaaaa..a org/abs/ aawaa///.11.. 13883.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXfv:2312.06633, 2024. URL
https://arxixs org/ a 2312. 06635.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answer- ing. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380. Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/D1--1222 1259. URL https: / /acl a anthology.orgee ogg. 8 1259//
20

---

Published as a conference paper at ICLR 2026
Han-Jia Ye and Wei-Lun Chao. How to train your MAML to excel in few-shot classification. In International Conference on Learning Representations, 2022. URL httpttt///peeeeeiiiiiii net/ Eorum?id=49h. _Kkoaaaaaah_IIoJaa....
Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. Longproc: Benchmarking long-context language models on long procedural generation. arXiv preprint wrXiv:2501.05414 2025. URL
org abs aaa./2501. 2201. 05414.
Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2588-2610. Bangkok. Thailand, August 2024. Association for Computational Linguisties. doi: 10. 10.11863/11/2224..o acl-long. 142. URL https: http://cllaattttoo 142//
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language langugae models effectively and thoroughly. arXiv preprint arXiv:2410.02649 2025. URL https: http:p::///axxv..oe/e wrg.iii..//iiooooooooooooooooooe abs 4100. 02694.
Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang. Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval, 2024a. URL attps://arrivv org/ abs /tt////////// 1...966. 19669.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang. Zhiyuan Liu, and Maosong Sun. oBBnnhh Extending long context evaluation beyond 100K tokens. In Lun-We Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 15262-15277 Bangkok, Thailand, August 2024b. Association for Computational Linguistics. doi: 10. 18265//1//002..allloo long.814, 814. URL https : / aaclanthology org/ 2024 mocl-long 814/.
21

---

Published as a conference paper at ICLR 2026
A DATASETS
In this section, we provide details on the datasets used for each task in our experiments and analysis. For each dataset, we visualize an example in Figure 7.
A.1 BABILONG
We construct the train and evaluation data using the data generation framework proposed by Kuratov et al. (2024). As we mentioned in the experiment setup, we mainly focus on tasks involving single-hop (QA1), two-hop (QA2), and three-hop (QA3) reasoning.
Tasks The main task for BabiLong is to reason over facts distributed in long context. For each QA task, we have a set of facts, including distractors and supporting facts. Each fact describes an agent's action or movement (which also describes the agent's 's location). For example, Mary journeyed to the garden is a movement, where Mary is the agent, and garden is Mary's current location. As an example for actions, John picked up the apple there describes John's 's action of picking up apple. Together with multiple agents performing multiple movements and actions, the set of facts became a temporal sequence of events.
QA1 focuses on one-hop reasoning, where the problems can be answered by using a single supporting fact. For example, in sequence of actions and movements, the agent John's 's last movement is John moved to the bedroom. Then the one-hop question "Where is John located now?". By the last movement alone, we can find the answer "bedroom".
QA2 focuses on one-hop reasoning, where the problems can be answered by using two supporting facts, where one fact is movement and the other one is action. For example, in the event sequence, we have John moved to the garden and John dropped the apple there. Using these two facts, we can answer "Where is the apple now?". ?", which is "garden".
QA3 focuses on one-hop reasoning, where the problems can be answered by using three supporting facts, as a combination of movements and actions. For example, we have three supporting facts: Daniel went back to the office, Daniel went back to the bedroom, and Daniel left the milk in order. We can infer the answer the question "Where was the milk before the bedroom?" is "office".
Build Long Context To build the long context, we use the framework to construct the tasks with long natural documents sampled from FineWeb-eds (Lozhkov et al., 2024), specifically from the CC-MAIN-2024-42 subset, ensuring the content postdates base model's knowledge cutoff. Given a particular context window, such as 8192, we sample that many tokens from the corpus as the Haystack long context.
We need to convert a long context into a batch of subsequences, such that PERK can encode them in parallel through test-time gradient-basee adaptation. In practice, set the effective context length (i.c., the number of tokens of subsequence on average) as 256. We divide the long context into chunks where each chunk is approximately 256 tokens long. Then, we randomly distribute the set of facts into these chunks. For each fact, we randomly select chunk and then insert this fact into the chunk at random position. To preserve the temporal order of the facts (since they come from sequence of events with temporal order), we index each fact with a prefix. Finally, we have batch of subsequences where each sequence might contain some facts at random positions.
A.2 STUDENT RECORDS
In the main paper, we propose Drops-in-the-Oceee (DIO), new evaluation setting where long contexts are formed from structurally similar documents. We aim to create problems in which the relevant and irrelevant information will likely be distributionally similar, making more difficult to identify critical facts. We construct synthetic dataset, Student Records, where each context simulates a database containing multiple student records. Each record includes several attributes: ID, name, school, major, and grade.
Tasks
The task operates on top of a database that consists of multiple records. Each record holds information about student. The record is structured follows:
22

---

Published as a conference paper at ICLR 2026
Student Id: 3109998, Student Name: Alison Keith, Year: Sophomore
School: School of Computer Information and Data Sciences, Major: Computer Science, Grade: 72
We generate the student IDs and grades using random number generator. The student names are generated from the Name package of Python. We define a pool of years, schools, and courses. Then, we randomly select each entity for each student. Crucially, to test generalization, specific entities (e.g., IDs, names) and attribute value combinations encountered in the test set are disjoint from those used during training.
Based on the database of student records, we construct three main tasks: Recall (retrieving attributes for a specific ID), Relation (comparing attributes between two IDs), and Aggregate (calculating the maximum, minimum, and average grade across all students).
tests a model's s ability to retrieve an attribute of a student given a specific student ID number. For example, we can ask "Whut major does siudent 1778924 study?". The model should answer "What major does student 1778924 study?". The questions cover all attributes for testing coverage.
Recall
Relation tests a model's ability to compare the attributes of two students. For example. for the attribute of grade, we can ask "Does student 1778924 have a higher grade than student 7077015?". The model should be able to compare their grades and infer "Yes". Or 1778924 and student 7077015 study the same major?", the model should recognize that their majors different and answer "No".
also ask "Do student
Aggregation is the most difficult task in this dataset. This task requires many-hop reasoning ability, where the model needs aggregate attributes from all students in long context make accurate conclusion. We focus on three types of numerical aggregation among the students' grades: (1) Maximization, (2) Minimization, and (3) Averaging. For example, ask the model "What the average grade of all students", the model should be able to compute the sum of all student grades, count the total number of students in the context, and then compute the average, "49.3".
Build Long Context Context length scales directly with the number of student records included. For given context window, we continuously append student records the context until the number of tokens exceeds the context window. To convert the long context into batch of subsequences for PERK, we simply treat each student record as an individual subsequence in the batch, since they are already independent documents by design.
A.3 OPEN-DOMAIN MULTI-DOC QA
We select HotpotQA (Yang al., 2018) and TriviaQA (Joshi et al., 2017) the main evaluation for open-domain QA over long-context documents. HoptpotQA tests the model's 's ability to perform 2-hop reasoning two gold documents randomly located in long list of documents. TriviaQA tests the model's s ability to do reading comprehension over very long documents where the target information can be scattered around the long context. We refer the original papers for QA construction details, data sources, and task examples of these two benchmarks.
Build Long Context To build long contexts for these two benchmarks, given a specific context window (for example, 32768 tokens), we follow Yen et al. (2025)'s process for their RAG evaluation. We extend the original context by injecting retrieved distractor documents (Zhang et al., 2024a) at random positions until reaching the context window limit.
A.4 LOST-IN-THE-MIDDE API RETRIEVAL
To analyze model's robustness against relevant information positions, we follow the setting proposed by Lost in the Middle (Liu et al., 2024). We use API documents from Patil et al. (2024), where each document has associated API call, and create long contexts containing multiple API-call- to-document pairs. We place the relevant API document a particular user instruction at different locations in the context (beginning, middle, end, and random position).
23

---

Published as a conference paper at ICLR 2026
NIAH: BABILONG
PERK
Vhlle Mocett enett that theprima ofi schhossrrrgggg 115 enfaanc supporanc toudbeshccding a
colorial duta lestion, stangy, potartal seatshoto apoilios imallii wnn te sucioo comprom
Pre trained LLM
raaa arga
Scrabchpad
they prosec arrnaciue mast
Context
A: hallway
becone aiv toupe oi privacy loc perepeeeinn acor riubi by inconcc sion Eir Meakk
seelo reaelty Many fveet May d she apple thene waart
Bexsoring
twns acthe
simeaiid Cardel
Svenn e traseledds They oould bilo the sake athool sipp
Question
apple?
STUDENT RECORDS
PERK
enterttn and Semon." "Maae" CCmpuie Schene "Draki "Saccrra T77ESS; "Saaker Langaie tnea Neadi fot Pladdes "seaee Studer Nore! Cuats RNF.
Pro traned
Memory Scratchpad
LLM
("aaket monny Nore! riac "har 100 Your
lats
Context
Stadie
WMO0E "adderharr "Mler Syterwurtty
A: 49.3
Deascrking
Humer tar "Mg oi Tay Marse Creaker Conarr vatkon san PSuahert 3002003 dent hurs! Can Unteguend Noure USaakedd "172000"" SSaat Nuree" DBS, "Non
Question
What is the average grade of students?
Lost-in-the-middle
cod--itttiss4hhn Lurea and baw.hiaaeeetteaaaaa conai: Natur Langu
featus
PERK
bes soc asparnntt- fokk geheer cnntorment rog venert forihemen ourppi ooderttk
Crooto
Pre-trined LLM
Memory Scratchpad
Context
Taoka Tutuu Clas fuuttor canoocc Sone Los boyllyy Whr Draliy doseliction
new suultt anganentitty pothon andioeennt souutee aaginggien hush ad4l pandesl from notiee,
AutoModel from pretrained @nturfechicom- bertibasel
siile Matura Frocossing
Aap- 2tt eoi angumennctree rum, waam ses aepoo mes lengin) porton, anwosmment
Reasoning
Question
Instrcttio White an API implementation that bakes oustomer redeos ad impur and extracts
featunes analyne cuntomer ment.
Figure 7: Dataset Examples: Here show example from the three datasets used: (Top) BabiLong, (Middle) Student Records, and (Bottom) Lost-in-ther bost-in-the-Micee the-Middle API Retrieval. PERK first encode the contexts into the memory scratchpad through self-supervised test-time learning. Next, the user can ask any number of questions about that context, and the LLM can reason over the updated memory scratchpad to generate the answers.
Task We formulate the main task as gold API retrieval. In this task, the model must retrieve the correct API call based on user instructions and documents. Using fixed 4k and 8k token contexts, we train PERK and FT-ICR models on contexts with the target document at varying positions and test them on contexts distributed across all positions.
Build Long Context We follow Student Records and scale the context length with the number of API documents included. For a given context window, we continuously append API documents to the context until the number of tokens exceeds the context window. Similarly, since each API document is independent, we treat a document as subsequence in the batch for PERK. Since the documents are processed in parallel as batch, they do not preserve any ordering information. Thus, by design. P PERK is robust to the position of the relevant information in context.
24

---

Published as a conference paper at ICLR 2026
B TRAINING AND INFERENCE DETAILS
Hardware Information All training runs are conducted on a single Nvidia H100 GPU with 93GB of VRAM. The training jobs run on an internal research cluster managed by Slurm. Each job is provided with 64 CPUs and 64GB of RAM. The GPU is completely utilized without sharing with other concurrent jobs.
Standard Hyperparameters Each training (for PERK and baselines) sets 2 epochs as the maximum runtime with early stopping based on validation loss. In practice, most training runs typically stop early, within one epoch of runtime. We set the baseline training and PERK's outer loop learning rate to 1e-5, and the weight decay 0.01. We use a cosine learning rate scheduler with 0.03 of the total optimization steps as the warm-up phase. We use PyTorch Lightning'' as the trainer for PERK. The Qwen and GPT-2 FT-ICR baselines are trained using the Open-Instruct (Lambert al., 2025) trainer from A12, and the Mamba FT-ICR baselines are trained using the Huggingface Trainer For the stress test of length generalization beyond 64K tokens, we change the default maximum position of 32768 to 131072.
LoRA Hyperparameters We set LoRA rank to 256 for both PERK GPT-2 and PERK Qwen. We use the default values for the other hyperparameters, including setting the alpha to 16 and the dropout to 0.1. We apply LoRA to all modules in the model architecture. For IB scale and below, we apply LoRA to all layers in the model. However, for 7B and 8B models, we apply it only to the top 4 layers in order reduce the memory overhead of meta-learning. We set the scaling factor 16 initially and enable rank-stabilized LoRA fine-tuning Kalajdzievski (2023), which results in target a of 256, corresponding to a rank of 256.
Meta-Learning Hyperparameters For the inner loop adaptation, we use inner loop gradient steps. For the truncated gradient unrolling, we truncate steps and 2 for input context length <4K, and we truncate steps 1, 2. and 3 for context length > 8K. We follow Ye & Chao (2022)'s ' practice on how to train good MAML model and meta-learn set of per-layer-per-step dynamic learning rates for the inner loop as part of the outer loop learning, using 5e-5 as the initial value. This approach learns separate learning rate for each network layer and each adaptation step in the inner loop. These learning rates are optimized during the meta-training process alongside the model parameters. Note that these per-layer-per-step learning rates are learned parameters, hyperparameters that require manual tuning. During meta-training, our algorithm automatically learns to adjust these learning rates for each layer and inner loop step dynamically based on the meta-objective. This approach reduces the hyperparameter tuning burden compared to manually setting the adaptive learning rates. For the language modeling loss in the inner loop, we use weighted version similar to CaMeLS (Hu et al., 2023). The token-level weights are predicted via a weighting model implemented as a two-layer MLP network. The parameters of the weighting model are co-meta-learned along with the LoRA parameters and the adaptive per-layer-per-step learning rates. We use the differentiable version of the Adam (Loshchilov & Hutter, 2017) optimizer for the inner loop optimization, implemented by the Higher library for higher-order optimization. We compute higher-orde derivatives using the automatic differentiation mechanism implemented by PyTorch'. The outer loop optimizer is the standard PyTorch implementation of AdamW.
Long-context Data Collation for PERK At training time, to collate reasoning problem with a long context and a set of associated guestion-answer pairs, we first convert a long context into a batch of subsequences, with relatively even lengths. In practice, we set the effective context length (i.e.. the number of tokens of subsequence on average) as 256,, and divide the long context into N chunks where each chunk is approximately 256 tokens long. For a long context with 8192 tokens, the batch size is thus N = 32. PERK's ss inner loop adaptation encodes the batch of subsequences in parallel through gradient updates. At inference time, if we want to test PERK on a long context with 32K
https///eeeeeeeeeeeeeeeeaaaaaaaa http:r//gttttocoomm/eesserreeees httppp//doogggppooooooooioiiiiia html
25

---

Published as a conference paper at ICLR 2026
tokens, the effective context length stays the same approximately 256 tokens, while the batch size increases to N 128.
Note that splitting long context into batch of shorter sequences either be cormuttaiooninvvartt or maintain the original ordering. PERK induces ordering through explicit indexing in the prompt formatting, which allows the model to learn implicit ordering during the encoding phase. Specifically, when chunking a long context, we can preserve ordering information by indexing each chunk. For example, if we split long document into segments A, B. C. E, format them as: ("Document 1: A", "Document 2: .... "Document 5: E"]. During the inner loop adaptation with the NLL objective, the LoRA parameters learn to associate these indices with their content, effectively encoding the ordering information.
Evaluation Hyperparameters At inference time, we generate the answer for a reasoning problem via greedy decoding (temperature is set 0). The maximum number of tokens that can be generated is 512. The EOS token is set to the default one defined in the tokenizer. The generation is handled by the text-generation pipeline implemented by Huggingface 5, For the length generalization analysis, we apply recent training-free methods Yarn (Peng et al., 2023b) and Dual Chunk Attention (DCA, An et al. (2024a)) to the finetuned Qwen models. We use vLLM (0. 10.2) as the inference engine to apply Yarn DCA. For Yam, we apply factor of 4.0 to the original max position embeddings. 32768, and scale the context window to 131072. For DCA, we use vLLM's QUAL_CHUNE FLASH _ATTN backend for generation.
C ADDITIONAL ANALYSIS
C.I IMPACT OF LORA RANK IN PERK
In this section, we analyze the impact of LoRA's rank on PERK's s long-context reasoning performance. We focus on the BabiLong tasks with a 1024 context window for an efficient ablation study. We apply PERK to Qwen-2.5- 0.5B with LoRA rank of 16 and 256 and evaluate on the BabiLong tasks.
Method
QA1
QA2
QA3
FT-ICR
91.1
70.5
48.6
PERK LoRA-16
99.0
96.9
84.7
PERK L6RA-256
100
98,1
89.1
Analyzing LoRA rank's impact We show the results in Table 2. We also include the per- on PERK'S performance. formance from the FT-ICR Qwen baseline reference. The results show that PERK's performance only decreases slightly when the rank is reduced to 16. However, PERK still maintains a much stronger performance than the FT-ICR Qwen baseline. The analysis shows that PERK can maintain its strong performance with a more parameter-efffcient setting, highlighting its potential for more efficient long-con conteett reasoning.
Table 2:
C.2 INNER LOOP ADAPTATION STEPS
In this section, we analyze the impact of inner loop adaptation steps on PERK's long-context reasoning performance. Similarly, we focus on the Babil Long tasks with 1024 context window. We again apply PERK to Qwen-2.5-0.5Bh for and inner loop steps. With 4 steps, truncate the first steps in backpropagation. With 2 steps, truncate the first step in backpropagation. We show the results in Table 3. The results show that reducing the number of inner loop adaptation steps decreases the performance more noticeably than reducing LoRA rank. However, PERK still outperforms FT-ICR Qwen on all three tasks. The results here suggest that we should select a higher number of inner loop adaptation steps to maximize the performance. If in settings where GPU memory might be limited, reducing the number of inner loop steps can reduce the performance, but it
Method
QA1
QA2
QA3
FT-ICR
91.1
70.5
48.6
PERK Steps
99.2
94.5
82.4
PERK 4 Steps
100
98.1
89.1
Table 3: Analyzing the impact of the number of in- ner loop adaptation steps on PERK's performance.
pipel ines "https://dee v1 att.ee.eilmaaaae/aaattaaa
26

---

Published as a conference paper at ICLR 2026
is still able to outperform the baseline. However, the analysis is done with limited hardware resources, which limits our analysis to 4 steps. To further analyze the impact of the inner loop step counts, more systematic experiments in scalable hardware setting are needed for future work.
C.3 GRADIENT UNROLLING TRUNCATION STEPS
In this section, we analyze the impact of the truncation steps we use when performing gra- dient unrolling (GU) for computing the meta-
Truncation Length
Accuracy
TGU (0)
100
gradients in the outer loop of the meta-learning process. Recall that to reduce the memory cost of full gradient unrolling, which requires sav- ing the computation graph for each inner loop adaptation step, we truncate the unrolling by detaching the early steps' graph to save mem- ory. Bounded by memory constraints, we can only train on K-token contexts with full GU without any truncation. Thus, we conduct our analysis under the 2K context window. Given 4 inner loop adaptation steps, we truncate 0-4 steps with Qwen2.5-0.5B on the BabiLong-Qua task. The analysis shows that we can truncate up to 3 steps and only unroll the last inner loop adaptation while maintaining a high accuracy without significant degradation. However, truncating all steps, which results first-order approximation of the geta-ggra,,nnts shows large performance drop, even on this very simple setting with QA1-IK.
TGU (1)
100
TGU
100
TGU (3)
98.8
TGU (4)
92.4
Table 4: Truncation length ablations on the BA- BILong QAI task with 1K tokens. We conduct 4 steps of inner loop adaptation and truncate 0-4 steps. PERK maintains the accuracy up to 3 trunca- tion steps, but drops when all steps are truncated.
D TRAINING MEMORY COST ANALYSIS
We show PERK's 's training scalability by measur- ing its peak memory use across different context lengths (compared to prior test-time learning method, RECKONING; Chen et al., 2023b).
4 PERK TGJ
PERK TGU (2)
Reckonting
PERK-TGUU
PERK-TGUI83 (3)
Training Memory Complexity (GB)
Setup We complete all runs single Nvidia H100 GPU with 93 GB of VRAM using Hug- gingface's 's Transformers library version 4.51.3. We perform four gradient update steps for the training-time inner loop adaptations for PERK. We test four truncation levels ($3.2): truncate 0 (full gradient unrolling without truncation). 1, 2. and steps. Both methods undergo 10 hardware warm-up iterations.
100
(GB)
Memory (GB)
75
50
2K 4K
8K
Lengthh
PERK is more memory efficient at training. Figure 8a shows that, as the training context length grows from 1K to 8K tokens, RECKON- ING quickly runs into Out- -f--orryemory (OOM) errors at context length 2K. At the same time, PERK, with increasing truncation steps, contin- ues to fit the maximum GPU memory, success- fully validating PERK's training scalability on long contexts.
Figure Peak GPU memory usage during train- ing with context lengths ranging from 1K to 8K tokens for RECKONING and PERK. While PERK successfully scales 8K tokens, RECKONING encounters out-of-memory (OOM) errors at shorter context lengths.
E DETAILED RESULTS
We show detailed quantitative performance of PERK, FT-ICR baselines, and frontier long-context L.Mss on the long-context reasoning benchmarks we evaluate on in Table 5.
27

---

Published as a conference paper at ICLR 2026
BRI
BBL-QA2
BBL-QA3
HetpoiQA
TriviaQA
SR-Recail
SR. Relation
SR SR-Aggregate
AVG
Model
32K
8K
32K
32K
8KK
32K
Commeriiat Froviler Muderis
Geminé-1 1.5ppm
87.7
56.4
75.3
99.4
58.8
Contests
Vwen2.5.7B-IM ProL.ong88B-1228
72.8
41.0
26.2
29.0
27.8
62.4
70.4
60.
$6.4
$3.9
47.0
Thalned on contents with 8K toluna
FT-ICR (Mamboe (48)
24.2
12.4
22.2
18.5
6.8
13.2
9.8
32.1
20.5
12.1
75.0
52.3
30.5
10.9
28.7
15.8
FT-ICR (Qwem2.5-0.50)
76.1
43.8
64.3
30
37.5
12.1
24.4
11.9
47.4
41.7
93.6
84.5
65.3
57.8
51.8
42.4
57.6
41.7
FT-ICR (Qwn225-77B
91.5
80.9
69.4
42.7
59.4
20.8
42.4
39.8
55.4
51.7
98.8
91.5
69.7
62.4
$7.6
45.8
66.1
54.5
FERX (Ouis) motiiedd oncode content with 8/ rokmus
PXNR (Qwen2, (Qwen2.5-0,18)
5-0,58)
99.0
95.3
02.4
80.9
6014
38.7
52.2
49.4
65.5
62.8
95.5
90.4
67.9
60.9
60.1
40.5
74.2
66.0
FSRX (Qwen2.5-0)) 5-70)
100
94.2
$4.4
65.2
55.6
52.2
70.8
100
98.5
75.5
70.3
62.4
53.3
78,2
70.9
Table 5: Model Performance Across Long-context Reasoning Benchmarks and Context Lengths (8K, 32K): We list detailed performance of our method PERK, FT-ICR baselines, and the frontier specialized context LLMs: Qwen2.5-7B I IM (Yang et al., 2025), ProLong-LLaMAAS LLaMA3- 8B3-Instruct-5122 (Gao al., 2025a), GPT-4.1 (OpenAI, 2025), Gemini-1.5-pro (Team et al., 2024). BBL here stands or Babil Long (Kuratov et al., 2024). SR stands for Student Records, our proposed dataset for Drops-in-the-Ocee -n-thh--cean reasoning. PERK and FT-ICR models have only seen training data contexts with 8K tokens, requiring them to generalize to the unseen 32K context window. PERK achieves the highest performance on average for both 8K-token and 32K-token contexts.
28


Reviews:
[{"review_id": "boVpOHBo7J", "reviewer": "Reviewer_bCFY", "summary": "This paper introduces PERK, a method to improve reasoning over long contexts.\n\nPerk learns from the context at test time by updating a small, efficient memory module (using LoRA) while keeping the main model frozen. It uses a two-step training process: first, rapidly encodes the context, and then optimizes how the model uses that info to answer questions.\n\nExperiments on long context benchmarks show PERK significantly outperforms standard long-context finetuning.", "questions": "- One of the mentioned strengths is that it encodes context as permutation-invariant representations, but that might not always be a strength. What if there is a query about positional information? The benchmarks do not consider that.\n- In figure 6, you show that you encode context + question before the LoRA updates. This means that for each new question, you need to do the update procedure. Is there a way to encode only context so that you can ask multiple questions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Paper is well written, and the results are nice\n- PERK demonstrates strong generalization to long context extrapolation (e.g., training on 8K, testing on 128K) and superior robustness to positional biases in the relevant information.\n- The idea of using a LoRA adapter as a differentiable memory module for context is a good alternative to ICL\n- The use of LoRA and TGU is nice, and seems well executed.", "weaknesses": "- The biggest concern is the added complexity. The gains are nice, but I'm not sure they justify using this method over FT-ICR.\n- Related to the added complexity, in general the inference time is significantly longer than ICR (at least up to 34k according to fig 7)\n- PERK is designed for length generalization, as noted in the experiments (train on 8k, eval on 32k). However, FT-ICR is not suited for this kind of evaluation, as is well known. So I'm not sure this is the best baseline to compare against. Especially since we can fine-tune on 32K lengths, I think it would be nice to see an FT-ICR training with 32k length.\n- Permutation-invariance can be detrimental for certain tasks (though maybe rare).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkUwZcjPW9", "reviewer": "Reviewer_d52o", "summary": "This paper introduces PERK (Parameter-Efficient Reasoning over Knowledge), a novel meta-learning framework that enhances large language models’ (LLMs) ability to perform long-context reasoning. Instead of relying solely on in-context learning or expensive fine-tuning, PERK allows models to encode long contexts through gradient updates at test time, using lightweight Low-Rank Adapters (LoRA) as memory modules. The approach employs two nested optimization loops: the inner loop encodes context into LoRA parameters, while the outer loop learns how to use these adapted parameters for reasoning. Across multiple benchmarks, including Needle-in-a-Haystack, HotpotQA, and a new Drops-in-the-Ocean dataset, PERK consistently outperforms long-context fine-tuning baselines and specialized long-context models, showing strong robustness to reasoning complexity, length extrapolation, and positional variation in relevant information.", "questions": "- Will the code be publicly released?\n\n- Reading PERK made me thing about PATHGOOSE [1], where at each LLM token the authors propose to route to a LoRA module that has been finetuned for a specific task. I haven't seen a comprehensive description of any future work you envision for PERK and was wondering if some variant of PATHGOOSE is something you thought about to improve efficiency. \n\n\n[1] Mohammed Muqeeth, Haokun Liu, Yufan Liu, Colin Raffel, Learning to Route Among Specialized Experts for Zero-Shot Generalization,", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper offers a wide-ranging experimental study across multiple long-context reasoning benchmarks, including Needle-in-a-Haystack, HotpotQA, TriviaQA, and the newly introduced Drops-in-the-Ocean (DIO) dataset. The introduction of DIO, which features distributionally similar distractors to better test reasoning precision, strengthens the evaluation by addressing limitations of prior benchmarks. Comparisons against both open-source and commercial LLMs convincingly demonstrate PERK’s superiority.\n\n- PERK exhibits remarkable robustness to reasoning complexity, unseen context lengths (up to 128K tokens), and positional shifts of relevant information. Its consistent performance across model families (GPT, Qwen, LLaMA) and scales indicates that the method learns transferable reasoning strategies rather than overfitting to a particular model or data distribution.\n\n- The approach is built on solid meta-learning foundations, combining bi-level optimization with truncated gradient unrolling to balance effectiveness and efficiency. Its LoRA-based parameter-efficient adaptation enables scalable test-time learning without full model updates, making PERK both methodologically principled and computationally practical.", "weaknesses": "- The authors don't mention any public release of the code\n\n- In section 2 of the appendix I see you only ran you experiments setting temperature to 0. Have you tried other values and what results have you found? I'd be curious to see the robustness of PERK at different decoding parameters\n\n- While I didn't find any major weakness, what prevented me from giving a strong accept to this paper is a clear metric of inference cost for PERK. For this method to be useful to researchers and ML practitioners, data on inference cost is fundamental. While a direct comparison with fine-tuning methods is not fair, at least some stats on time needed by PERK would make the paper contribution much more comprehensive\n\nOverall, I think PERK makes a solid contribution and I vote for its inclusion at the conference as is. With additional inference cost analysis, I'd be happy to further increase my score", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ij6Df6mpfp", "reviewer": "Reviewer_sFKq", "summary": "This work proposes a meta-learning method that learns to encode long contexts through gradient updates to a model adapter at test time. The inner encodes the context to Lora, and the outer loop uses the adapter for the relevant information from the encoded long-context. The experiment proves that the proposed PERK is effective for long-context reasoning.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The test-time learning problem is relatively useful to further improve the model performance.\n* Figure 1 presents the pipeline of this method, and the inner loop is clear.\n* The method compares the performance with FT-ICR, proving that the PERK is better.\n* The experiments have various models, supporting that the PERK is general.", "weaknesses": "* Is the method test-time training? According to the definition of the outer loop, Equation 3 needs the label of the question. However, during test time, there is no such label.\n* Figure 4 is not related to length extrapolation. The Qwen-2.5-0.5B is trained with 32K by the Qwen Team. However, Figure 4 presents the performance with a maximum 32K, which is NOT longer than the Qwen-2.5-0.5B training length. This is a misclaim.\n* It should provide the training cost, such as time cost, for the comparison of PERK and FT-ICR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZKCCiRA1f", "reviewer": "Reviewer_MTjy", "summary": "This paper introduces PERK (Parameter Efficient Reasoning over Knowledge), a novel method that reframes long-context reasoning as a test-time learning problem. Instead of processing a long context in a single forward pass, PERK first encodes the context into the parameters of a Low-Rank Adapter (LoRA) via several gradient-based update steps at inference time. The model then answers a given question using this newly created parameter-efficient memory module, without needing the original long context in its attention window. To enable this, PERK is trained using a bi-level meta-learning optimization scheme: an inner loop learns to rapidly encode context into the LoRA adapter, while an outer loop learns to use the adapted model for reasoning. To make this process scalable, the authors employ truncated gradient unrolling. The paper demonstrates through extensive experiments that PERK significantly outperforms standard long-context fine-tuning, shows superior length extrapolation and robustness to positional biases, and is competitive with or exceeds the performance of specialized proprietary and open-source long-context models.", "questions": "1. Inference Latency Trade-Off: Could you provide a more direct comparison of the end-to-end inference latency (context encoding + question answering) of PERK vs. FT-ICR, particularly in the common 8K-32K context range? How does this latency scale with the number of inner-loop adaptation steps, and where is the practical break-even point where PERK becomes faster than a standard forward pass?\n\n2. Optimality of NLL for Encoding: The inner loop uses a standard NLL objective to \"compress\" the context into the LoRA. Have you considered alternative objectives that might more explicitly encourage the adapter to store salient facts? For instance, would a contrastive loss or an information-theoretic objective that prioritizes reasoning-relevant information lead to a more effective parametric memory?\n\n3. Comparison to RAG: Your method encodes the entire context into parameters, which is a key difference from RAG systems that first retrieve a small subset of the context. Could you discuss the conceptual trade-offs between PERK and a standard RAG pipeline in terms of performance, latency, and robustness to different types of reasoning tasks?\n\n4. Ablation on Training Components: The PERK training process involves several advanced components (TGU, learned per-layer LR, weighted NLL). How critical is each of these to the final performance? For example, what is the performance drop if you use a fixed inner-loop learning rate or a standard (unweighted) NLL loss?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Novel Conceptual Framing:** The paper's most significant contribution is reframing long-context reasoning as a test-time, gradient-based adaptation problem. This is a fundamental and clever departure from the standard in-context reasoning (ICR) paradigm, moving from \"reasoning over context\" to \"reasoning from context stored in parameters.\"\n\n**Exceptional Empirical Performance and Robustness:** PERK demonstrates substantial performance gains (up to +20% absolute) over standard fine-tuning. More importantly, the method shows remarkable robustness where standard models fail:\n- Length Extrapolation: PERK models trained on 8K contexts maintain strong performance on contexts up to 128K, significantly outperforming baselines and even specialized long-context models.\n\n- Positional Robustness: By processing context as a permutation-invariant batch of segments, PERK largely overcomes the \"lost in the middle\" problem, showing minimal performance degradation when the position of relevant information changes, unlike FT-ICR models which fail catastrophically.\n\n**Scalability and Model-Agnosticism:** The combination of LoRA and truncated gradient unrolling makes the complex meta-learning approach tractable . The paper convincingly shows this advantage holds across multiple model families (GPT-2, Qwen, LLaMA) and scales (0.5B to 8B), highlighting the general applicability of the technique.\n\n**Strong Evaluation Methodology:** The introduction of the Drops-in-the-Ocean (DIO) benchmark is a valuable contribution, creating a more challenging and realistic test than standard Needle-in-a-Haystack by using distributionally similar distractors.", "weaknesses": "**Inference Latency Trade-Off:** The paper emphasizes efficiency in terms of training memory, but the trade-off with inference latency is a major practical concern. The method requires performing multiple gradient updates at test time to encode the context before answering a question. While Figure 7b shows PERK can be faster at extreme context lengths (e.g., 64K), it appears slower than FT-ICR for more common lengths. This makes the \"efficient\" framing potentially misleading for many real-world applications where inference speed is critical.\n\n**Complexity of the Training Framework:** The PERK training pipeline is highly complex, involving bi-level optimization, truncated gradient unrolling, learned per-layer-per-step learning rates, and a weighted NLL loss in the inner loop. This high degree of complexity could pose a significant barrier to reproducibility and adoption compared to the much simpler standard fine-tuning approach.\n\n**Sub-Optimal Encoding Objective:** The inner loop's objective is simply next-token prediction over the context segments. This seems to incentivize the model to learn the surface-level statistics of the text, which may not be the most effective way to distill salient, reasoning-critical information into the LoRA parameters. The paper does not explore or discuss alternative, potentially more targeted, encoding objectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)