Self-Adapting Language Models
Authors: Adam Zweiger
Jyothish Pari
Han Guo
Yoon Kim
Pulkit Agrawal
Venue: NeurIPS 2025 poster ()
Keywords: llms
ml
adaptation
meta learning
synthetic data
finetuning
continual learning
reinforcement learning
knowledge editing
Primary area: deep_learning
TL;DR: RL to train LLMs how to generate data and update themselves to adapt to new knowledge/tasks.

Abstract: Large language models (LLMs) are powerful but static; they lack mechanisms to adapt their weights in response to new tasks, knowledge, or examples. We introduce $\textbf{Se}$lf-$\textbf{A}$dapting $\textbf{L}$LMs (SEAL), a framework that enables LLMs to self-adapt by generating their own finetuning data and update directives. Given a new input, the model produces a $\textit{self-edit}$ --- a generation that may restructure the information in different ways, specify optimization hyperparameters, or invoke tools for data augmentation and gradient-based updates. Through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation. To train the model to produce effective self-edits, we use a reinforcement learning loop, using the downstream performance of the updated model as the reward signal. Unlike prior approaches that rely on separate adaptation modules or auxiliary networks, SEAL directly uses the model's generation to parameterize and control its own adaptation process. Experiments on knowledge incorporation and few-shot generalization show that SEAL is a promising step toward language models capable of self-directed adaptation in response to new data. Our website and code is available at https://jyopari.github.io/posts/seal.

Self-Adapting Language Models
Adam Zweiger" Jyothish Pari*t
Han Guo
Yoon Kim
Pulkit Agrawal"
Massachusetts Institute of Technologg (adamz, jyop, hanguo, yoonkim, (alki. edu
Abstract
Large language models (LLMs) are powerful but static; they lack mechanisms adapt their weights in response to new tasks, knowledge, examples. We introduce Self-Adapting LLMs (SEAL), framework that enables LLMs self-adapt by generating their own finetuning data and update directives. Given new input, the model produces a self-edit-a generation that may restructure the information in different ways, specify optimization hyperparameters, or invoke tools for data augmentation and gradient-basee updates. Through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation. To train the model to produce effective self-edits, we use a reinforcement learning loop, using the downstream performance of the updated model as the reward signal. Unlike prior approaches that rely on separate adaptation modules or auxiliary networks, SEAL directly uses the model's generation to parameterize and control its own adaptation process. Experiments on knowledge incorporation and few- shot generalization show that SEAL is promising step toward language models capable of self-directed adaptation in response to new data. Our website and code is available huttps://jjoopart wit.ub..iittt......... wtwpuub..o/osstsseeeeeee
Introduction
Large language models (LLMs) pretrained on vast text corpora exhibit remarkable abilities in language understanding and generation
However, adapting these powerful models for specific tasks [6], integrating new information v or mastering novel reasoning skills [8] remains challenging due to the limited availability of task-specific data. In this paper, we explore an intriguing hypothesis: can an LLM self-adapt by transforming generating its own training data and learning procedure?
3
As analogy, consider human student preparing for the final exam of machine learning class. Many students rely on their notes to prepare for the exam. These notes are often derived from the lecture content, textbooks, content, assimilating and rewriting the information in the form notes often improves the ability of students to understand the content and answer exam questions. This phenomenon of reinterpreting and augmenting external knowledge in way that is casier to understand is not limited to just taking exams, but seems be universally true of human learning across tasks. Furthermore, different humans assimilate information in different ways- some might condense the information into visual diagram, some into text, or some might rely more on concrete mathematical descriptions.
information available the internet. Instead of relying on the raw
Such assimilation, restructuring, or rewriting of data as part of the learning process in contrast with how large language models (LLMs) current LLMs consume and learn from the task data "as-is" via finetuning or in-context learning However, such data may not be in optimal format (or volume) for learning, and
typically trained and deployed. Given new task,
9 10
"qqual contribution. Improbbbll AI Lab, CSAIL MIT
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

RL
Outer Loop Iteration t
RL Data
SE
Optimize
L
Test
LM
0
Ans
Ctx
SE
State
Action Reverd
Ctx
LM
0
SE
Opt imize
L
Policcy Update
01-1
SE
Optimize
L
Test
LM
A
Ans
Figure Overview of SEAL. In each RL outer loop iteration, the model generates candidate self-edits (SE)- direccivtss on how to update the aapphis-applies updates, evaluates performance on a downstream task, and uses the resulting rewards to improve the self-edit generation policy.
current approaches do not enable models to develop bespoke strategies for how to best transform and learn from their training data.
As step towards better language model adaptation, we propose equipping LLMs with the ability to generate their own training data and finetuning directives in response to new inputs. In particular, we introduce reinforcement learning algorithm that trains LLMs to generate "self-edits"- language instructions that specify the data and, optionally, optimization hyperparameters for updating the model's weights (see Figure Tj. We refer to such models as Self-Adapting LLMs (SEAL..
We evaluate SEAL on two applications. We first consider the task of integrating new factual knowledge into an LLM. Rather than finetuning directly on the passage text, we finetune on synthetic data generated by the SEAL model. Our results show that, following reinforcement learning (RL) training, finetuning on self ceff-generated synthetic data improves question-answere performance on the no no-passage-ine cuassag-in-contex variant of SQuAD [3 from 33.5% to 47.0%. Notably, self- data from SEAL outperforms synthetic data generated by GPT-4.1.
We further evaluate SEAL on few-shot learning on a simplified subset of the ARC-AGI bench- mark [14] where the model leverages set of fooly autonomously select both synthetic data augmentations and optimization hyperparameters (e.g., learning rate, training epochs, selective loss computation over token types). Our experiments demonstrate that automatic selection and configura- tion of these tools using SEAL enhances performance compared to both standard in-context learning (ICL) and self-editing without RL training to use the tools effectively. These results collectively show that SEAL is versatile framework for enabling language models to self-adapt.
2 Related Work
Synthetic Data Generation. The creation of synthetic data for LLM training is increasingly
common, from large-scale pretraining datasets [5
18 19 task-specific data augmentation For incorporation of smaller corpus, Yang et al. use synthetic data generation via graph-based prompting. SEAL builds on this line of work by using reinforcement learning to train generative policy that directly maximizes the downstream utility of synthetic data when applied for gradient-based self-updates, rather than relying on static or heuristic generation strategies that are manually tuned.
16 17
20 21 22
and instruction-tuning sets 23 24
25
Knowledge Updating. Several recent works aim to modify or inject factual knowledge into language models via weight updates. Some methods attempt directly locate specific parameters that correspond to individual facts [26] 27 28 Others propose generating additional finetuning We adopt the latter strategy, following Akyürek et al. [30], who propose generating logical implications of fact and finetuning on them, and Lampinen al. [3I] who show that implication- baased finetuning can even outperform in-context learning. We build on these approaches by training models through RL to generate more optimal finetuning data. Park al. [32 show that prompting language models to generate question-answer (QA) pairs directly can outperform implication-styy prompting. Because the SEAL framework is agnostic to the prompt and format of the self-edit data, it can also trained to generate QA pairs or other output formats, explored in |B.11
data using the information context (29
25 B1 32].

---

Test-Time Training, Test- Time Training (TTT) temporarily adapts model weights based on the input the model receives [3
34 35 36]. Akyürek et al. [36] show that combining TTT with ICL enables gradient-updates to outperform standard ICI. in the few-shot setting. SEAL can be viewed as incorporating a round of TTT in its inner-loop optimization, leveraging TTT's efficiency relative to full-scale training to perform multiple updates and reward the generated data that yields the greatest performance gain. Although our method is trained using single-example TTT episodes, we demonstrate in the knowledge incorporation setting that it generalizes to the continued pretraining (CPT) regime-whert placing data directly in context is no longer feasible.
Reinforcement learning has played central role in improving LLM behavior, originally through RLHF (3). More recently, RL with verifiable rewards has been applied to boost reasoning performance by optimizing the model directly for task success [39 40 41). SEAL applies RL not to optimize final answers or trace revisions, but to optimize the generation of self-edit data that is then used for weight updates.
Reinforcement Learning for LLMs.
Meta-Learning and Self-Modifyig Systems. SEAL embodies meta-learning principles
42
44 by learning an adaptation strategy- to generate effective self-edits- -va its outer optimization loop. The goal is to learn how to learn efficiently from task contexts. In reinforcement learning,
meta-learning has been used to train agents that learn new tasks quickly 45
46
47 48). Sun
et al. [49] similarly apply RL to learn task-specific weight modulations, offering an alternative to LoRA finetuning that is orthogonal to our approach. A natural extension of meta-learning is
self-referential networks, where models modify their own parameters [50] [5].. In the domain of large language models, recent work has applied meta-learning to improve LLM adaptation Notably, Hu al. [53] train smaller model output token-specific weights during finetuning, addressing a knowledge incorporation task similar to ours, while Chen et al. (54] propose hypernetwork that generates LoRA adapters conditioned on the input, enabling dynamic and task-specific parameterization. However, SEAL offers greater generality by leveraging the model's existing generative capabilities to parametrize updates.
52
53
54
55
49).
Seff-Improvement Several recent works fall under the umbrella of seff-improvement or self- training. Methods such as RLAIF [56 57] and self-rewarding language models [58] (9) use the model itself to provide reward signals, leveraging the observation that judging outputs is often easier than generating them [60]. Other recent works improve performance on mathematical tasks by using majority-vote or model confidence reinforcement learning rewards, enabling performance improvement without access to ground-truth labels [61 62 63 64 [65. However, all of these methods are fundamentally limited by the model's current evaluation abilities and self-consistency. In contrast, we view self- improvement through interaction with external data as a more powerful and scalable path. SEAL learns how to best utilize this external data for self ceff-improvement.
3 Methods
We propose Self- Self-Adapting LLMs (SEAL), a framework that enables language models to improve themselves by generating their own synthetic data and optimization parameters ("self-edits") in re- sponse to new data. The model is trained to produce these self-edits directly through token generation with the data provided in the model's context. Self-edit generation is learned via reinforcement learning (RL) where the model is rewarded for generating self-edits (SE) that, when applied, improve the model's performance at the target task. SEAL can therefore be interpreted as an algorithm with two nested loops: an outer RL loop, which optimizes the self-edit generation, and inner update loop, which uses the generated self-edit to update the model via gradient descent. Our method can be seen an instance of meta-learning where meta-learn how to generate effective self-edits.
3.1 General Framework
Let 0 denote the parameters of the language model LMg. SEAL operates on individual task instances (C, where is a context containing information relevant to the task, and 7 defines the downstream evaluation used to assess the model's 's adaptation. For example, in knowledge incorporation, C' is the passage intended to integrated into the model's 's internal knowledge, and is a set of questions and associated answers about the passage. In few-shot learning, C' includes shot demonstrations of
3

---

novel task, and T is the query input and ground ground-truth output. Given C, the model generates self-edit SE the form of which varies by domain (see $3.2 and updates its parameters via supervised finetuning: 0' + SE)..
We optimize the self-edit generation process using reinforcement learning: the model takes action (generating SE), receives a reward based on LMg''s performance on т, and updates its policy to maximize expected reward:
CRL (0c) == E(C,))D ESE (10) [r(SE, T, 0:)1
(1)
Unlike in standard RL setups, the reward assigned given action in our setting depends on the model parameters 0 at the time the action taken (since 0 is updated to I', which is then evaluated). As a re- sult, the underlying RL state must include the policy's s parameters and is given by (C,0), even though the policy's 's observation is limited to C' (placing 0 directly
Algorithm 1 Self-Adaptin LLMs (SEAL): Sdit Reinforcement Learning Loop
1: Input: LMg, dataset D = ((C,t)
2:
for outer iteration f = 1, do
3: Sample (C, D
Generate self-edit SE LM2 (- C)
Inner Loop Update: 8% - (F SE)
in context is infeasible). The implication of this is that (state, action, reward) triples collected with previous
6: Evaluate: Ans ~ LM -)
Compute reward: r <- r(Ans, 7)
version of the model, fold. may become stale and mis- aligned for the current model Current- For this reason, adopt on-policy approach, in which self-edits
8: Update: 0.+1 t- RL_ RLL_Update(ee SE)
9:
end for
are sampled from-and, crucially, rewards are computed using-the current model.
We experimented with various on-policy methods such Group Relative Policy Optimization (GRPO) [66] and Proximal Policy Optimization (PPO) [67]. but found the training to be unstable. Instead, we adopt ReSTEM 44), simpler approach based on filtered behavior cloning also known
as "rejection sampling + SFT" (68
69
38
39 70].
ReSTEM can be viewed as an expectation expectation-maxiiizziin maximization (EM) procedure: the E-step samples candi- date outputs from the current model policy, and the M-step reinforces only those samples that receive positive reward through supervised finetuning. This approach optimizes an approximation of our objective (T) under the binary reward:
{1
If on 7, adaptation using SE improves LMg, performance Otherwise
r(SE, T, 0.)
00
More precisely, in optimizing (T.. we must compute the gradient Vo, LRI. However, as we noted, the reward term r(SE, r, 0c) depends on O₂ in our setup but is not differentiable. We address this by treating the reward as fixed with respect to 0g. With this approximation, the Monte-Carlo estimator for a minibatch of N contexts and M sampled self-edits per context becomes
N
M
1
Vo, CRL =
Tsy
Vo, log Po. (SEi) | (.))
(3)
NM
1=
N
M
T
= NM
1
Σ
Vo. log
Po. log
(1d) veo CJ),
(4)
j=1
where pe, denotes the model's autoregressive distribution and ys (1-5 is the 4h token of self-edit SE,j.
the 7th sample for context C, Since sequences with r = can be ignored in (4). we have shown that ReSTeM with simple "SFT on good self-edits." indeed optimizes (T) under the binary reward (
(with a stop-gradient applied the reward term). The SEAL training loop is summarized Alg.
Finally, we note that while the implementation in this work uses a single model for both generating self-edits and learning from these self-edits, it also possible to decouple these roles. In such "teacher-student" formulation [7I]. student model would be updated using edits proposed by a separate teacher model. The teacher would then be trained via RL to generate edits that maximize student improvement.
'The reward may also be assigned to the single self-edit that yields the greatest improvement among sampled candidates, which we do in knowledge incorporation, rather than to all edits that yield positive improvement.

---

3.2 Domain Instantiations
We instantiate the SEAL framework in two distinct domains: knowledge incorporation and few-shot learning. These domains were chosen to highlight two complementary forms of model adaptation: (I) the ability to integrate
information into model's weights that it be recalled without relying on context (evaluated using a no-context variant of SQuAD) and (2) the ability to generalize to novel tasks after seeing only small number of examples (evaluated using ARC).
Knowledge Incorporation. Our goal efficiently incorporate the information provided in passage into the model's weights. A promising recent approach involves using language model to generate content derived from the passage, followed by finetuning on both the original passage and the generated content [29 30 25 32]. While the form of generated content may vary, adopt what we consider the canonical format: implications derived from the passage. This approach, introduced in deductive closure training [30]. converts given context C' into set of implications SE= = {$1, ....... Sn) by prompting the model to "List several implications derived from the content." The output may include inferences, logical consequences,
restatements of the original passage. In [B.IT] we also explore alternative prompts such as "rewrite the passage different ways" or "rewrite in question-answer format" and show that method improves performance by similar greater margins regardless of the base prompt.
Knowledge Incorporation Setup Self-Edit
Passage
Evaluation
titte: Apollc pregram
Apollo program faced apposition from Kennady's science advisor, Jeroma Wiesner, who had...
Who was Kennedy's science adviser that manned spacecraft llights?
NASA reached internal smooth seiling...
from
LM 0g
SFT
LM 0
Figure 2: Knowledge Incorporation Setup. Given a new passage, the model generates synthetic data (the self-edit) in the form of "implications" of the passage. We then finetune on these outputs using LoRA. The updated model is evaluated on questions about the passage without access the original text, and the resulting accuracy serves the reward signal for reinforcement learning.
These self-generated statements form the training data for a supervised finetuning (SFT) update: we compute the standard causal language-modeling loss over each sequence 8; and update the model parameters, yielding 0'. Since the amount of data per update is small and the number of updates do in total large, we use low-rank adapters (LoRA (72)) for efficient, lightweight tuning. Finally, the adapted model LMg is evaluated on the task т. This process is shown in Figure 2
During RL training, the adapted model's accuracy on defines the reward that drives the outer RL optimization. This trains the model to restructure the passage in way that is most effective for assimilation via finetuning.
Few-Shot Setup
Few-Shot Examples
Self-Edit (SE)
Eva Evaluatio ion
basic augmentations: true size size.augeentaiion: false chain_augmennations false repeat angeettaugnentationss
LM 02
LM
0
strategy:
learning rate: 1e-05
SFT
0
epochs:
Figure 3: Few-Shot Learning with SEAL. Left: example ARC demonstrations. Center: the model generates self-edit specifying augmentations and training hyperparameters. Right: the adapted model evaluated on a held-out test input.
5

---

Few-Shot Learning. The Abstraction and Reasoning Corpus (ARC) [8] is a benchmark designed test abstract reasoning and generalization from very limited examples. Each task includes a small set of input-output demonstrations and held-out test input whose correct output must be predicted.
We adopt the test-time training (TTT) protocol of Akyürek et al. [36] where augmentations of the few-shot examples are used to perform gradient-based adaptation. Rather than relying on manually tuned heuristics for selecting augmentations and optimization settings, we train SEAL learn these decisions. This setting tests whether SEAL can autonomously configure the adaptation pipeline- determining which augmentations to apply and what optimization parameters to use.
To implement this, we define a set of tools, each of which is a pre-defined function from Akyürek et al. [36] that transforms data specifies training parameters. These include:
rotations, flips, reflections, transpositions, resizing operations (c.g., changing grid resolution), and chained repeated transformations.
Data augmentations:
Optimization parameters: learning rate, number of training epochs, and whether the loss is computed over all tokens or only output tokens.
The model is prompted with a task's s few-shot demonstrations to generate a self-edit, which in this case is a specification of which tools to invoke and how to configure them, as shown in Figure 3 The self-edit is then applied to adapt the model via LoRA finetuning. The adapted model is evaluated on the held-out test input, and the result determines the reward for the self-edit generation.
4 Results
In this section we empirically evaluate SEAL across our two adaptation domains: few-shot learning and knowledge incorporation. Full training, hyperparameter, and evaluation details are provided in and B
4.1 Few-Shot Learning
We conduct our experiments using Llama-3. 1-1B-Instruct, a small open-source model with no ARC-specific pretraining. Since most ARC tasks are challenging for models that have not been pretrained on ARC, we curate subset of 11 tasks from the ARC training set and from the evaluation set, filtered to ensure that they are solvable under optimal TTT configurations for a base 1B- 2- Instruct. While this is small number of examples, note that Akyürek et al. (36
Llama-3.. 2- used the same TTT configuration for all tasks, and thus we do not need large training set for learning a fixed self-edit. More details are included in SA
The model is trained using ReSTEM by sampling 15 self-edits per training task. Each self-edit is applied individually generate 15 updated models, which are then evaluated the corresponding held-out test example. We reinforce only those self-edits that lead to correctly adapted models, i.c., models that produce the correct output for the test input after adaptation.
After training, we evaluate the model by generating 5 self-edits per held-out evaluation task and apply each one independently. We then report the percentage of self-edits that lead to correct outputs, yielding success rate that reflects the quality of the learned self-edit generation policy.
We compare against the following baselines:
1. ICL (In-Context Learning): Llama-3 -. 2-1B-Instruce is prompted with the given few-shot examples using Akyürek et al. (36)'s 's protocol, and directly queried on the test input.
2. TTT Self-Edit (w/o prior RL): Llama-3 3. 2-1B- Ins-nnutuct performs test-time training (TTT) using few-shot examples and synthetic augmentations, but without any prior RL to optimize which augmentations or training configurations to use.
Oracle TTT: The model performs test-time training (TTT) using the optimal human-crafter
configuration from Akyürek et al. [36). This provides an upper bound of our method.
We record results in Table 4.1
SEAL substantially improves adaptation success rate compared to baselines: 72.5% vs. 20% (with self-edits from the base model without RI. training) and 0% (no adap- tation), though performance remains below Oracle TTT, suggesting room for further improvement.

---

Method
Success Rate ( (%)
ICL
0
TTT + Self-Edit (wo prior RL)
20
SEAL
72.5
Oracle TTT
100
Table I: Few-shot Abstract Reasoning
4.2 Knowledge Incorporation
We experiment with Qwen2.55 .-BB on incorporating novel factual content from SQuAD passages [3. We use the relatively simple SQuAD dataset because its passages can be fully "understood" by the base model in-context, yet the model cannot reliably answer questions about them without that context. We do 2 rounds of ReST with batch of 50 contexts (see $B for further details). We compare SEAL on knowledge incorporation against the following bascline approaches:
1. Base Model:
The pretrained model is evaluated on downstream QA tasks without any adaptation or access to the passage.
The model is finetuned directly on the passage using the standard language modeling loss, without any synthetic data.
Train on Passage Only:
3. Train on Passage Synthetic Data: The model is trained the passage along with self-generated implications. This is the same setup as SEAL but without any prior RL training.
4. Train on Passage + GPT-4.11 Synthetic Data: The model is trained on the passage along with model-generated implications collected from G via the OpenAl API.
Table 4.2 reports mean no-context SQuAD accuracy under two regimes: single-passage updating (with LoRA), and small-scale continued pretraining (with full finetuning). We run continued pretraining (CPT) experiments with n 200 documents, as well the full SQuAD validation set of n 2067 documents. In the single-passage setting, finetuning directly on the passage yields negligible gain over the frozen base model (33.5% vs. 32.7%), confirming that using the raw data alone insufficient. Augmenting with synthetic implications generated by GPT-4.1 boosts accuracy to 46.3%, an improvement of 12.8 percentage points over the passage-only baseline. Using synthetic data produced by the base Qwen-2. 2. 5-7B model yields 99.7%. 6.2-point increase. After reinforcement learn- ing, SEAL further improves accuracy to 47.0 %, notably outperforming using synthetic data from GPT--411, despite being a much smaller model.
Single Passage Knowledge Incorporation
Krowledge Incorporation (%6)
Gaven 2.5 73 Buse
Run Pasaage
Gwenn Base Synthekk
Syothel
34
SCAL
2
ReST-EM Treining Iterations
Figure 4: Accuracy over RL itera- tions. Each iteration consists of a mini- batch of 50 contexts, each with 5 sam- pled self-edits. SEAL surpasses GPT- 4.1 synthetic data after two iterations of ReSTEM on the no-context SQuAD set.
In the CPT setting, the model assimilates information from many passages in a single continued pretraining run. It is then evaluated on the union of all corresponding ques- tions. In this setting, we sample 5 self-edit generations for each passage and take the aggregate synthetic dataset for continued pretraining. As shown in Table 4.2 we observe a similar ranking of methods as in the single- passage case, but with synthetic data from GPT-4.1 slightly outperforming SEAL. In the n = 200 setting, SEAL achieves an accuracy of 58.2%. exceeding its single-passage performance. We attribute this gain to the aggregation of multiple self-edit generations. Overall, the strong continued pretraining results of SEAL suggest that the self-editing policy generalizes beyond the original RI. setup of creating synthetic data single generation for a single passage.
Figure tracks accuracy after each outer RL iteration. Two iterations suffice for SEAL to overtake GPT-4.T data; subsequent iterations yield diminishing returns, suggesting that the policy quickly converges to an edit style that distills the passage into easily learnable atomic facts (see qualitative examples in Figure 5). All results use tuned hyperparameters (see {
7

---

Method
Single Passage
Continued Pretrain-
Continued Pretraining
(n 1; LoRA)
ing (n = 200; full-FT)
(n = 2067; full-FT)
Base model
32.7
32.7
29.0
Train on Passage
33.5
36.0
31.2
Train on Passage+ + Synthetic
39.7
50.6
43.4
Train on Passage GPT-4.1 Synthetic
46.3
59,4
49.2
SEAL
47.0
58.2
46.4
Table 2: Knowledge Incorporation Performance Across Passage Settings.
Imput Context
Q1: whaa and of sending tactnology la being used
Using who les in minforets of southom Surnonn mag sut thet ancesuaa kads to help stregthhin teir tonntoiial clains Cu Curcerly, neet tribse smapon do have cvnrty defioed boundaiee, making it ene for mends vertures to target their tertitoiee
The nemote waming the consenation Arruae to ans mga used the indipennzx trises of the beean protect
triee,
protect lancs the Amasont
Q what tilbe uses GPS dav cos to map lands?
Q
technollogy?
Reurite Iteration
Rewrit Iteration
A1: Cronee being
pronect the kente of the triee in the Amrccnr Xf
gamous
technaalogy protect their lands from con commens an intoresta 2. Claar striis for libaa santic con prevent commreent encidl
sibes protuct thair ands.
GPS their encectra lande in the Anooonn ninhhreet X
A2: vanommni
2. The use GPS devices and prog like Google Earth
tenioriie.
argeting
3. Crearing foreats for agricultre llndd cattle
technalogy to protedt iree lansh from comnr encial insensiaa /
A3:
fanchess a majo case dolo wussion the Arr frnazon
Regel Arsanon rairfo
mensiaas fignt
A1: moodel iesporee emate" x
Rewrite Iteration
5. The ecpans on of intrastmuctue, such reads and cama la abso make concemn conservation of the Amazznn
Maaaggo be case cPS devices mop
1. Remose censing technology can bos offectiveyy utilbee for conseraation
Arsason that are most waneraabe deforectation and other sveats
A3: some sties the Amnazon infernet use sensing
Amazton reinfornet.
prownndd
anceotra Time heice them better protect thas lamtoriaa from core noremennt intavests. of
protect their ands frrm commercaa vards
Amazon reiriorest crucill for consenvtion
a The uso of nanthed des esioee and pr progons lik mappl no
mapting ancestra lanis by and grnous mooo
lants
& Cleanin of boardariza for inekgances tamboriss can help
2. The use of tomoto senoing can help mon for the Amazon
rights land.
cowngthen
A1:
used protect intel lnnte Amazzn
techesibay
provent commerccial verozes targeting these the
cainbrrsst track changes
devioes
invo indigeouu ernorvation wfterts of the Anaoon in inforet can wesd to more scatainatle future for the reg on.
Aenozon inforeet ecsental the preoersstion of biodiowrily and the manteranne of eccayater services.
pronection
tibe intoreets southern being used help protect their stcestra lanck. A3: scree
prpttect thai lands from commercaal interests
Figure 5: Example Knowledge Incorporation Self-Edits Across RL Iterations. In this example, we see how RL leads to the generation of more detailed self-edits. which in turn results in better performance. While the progression is clear in this case, the differences across iterations sometimes more subtle in other examples. We show in &B. that prompting for longer self-edits is effective, and that RL training further improves performance by a similar margin.
Limitations
Catastrophic forgetting. One key motivation we had for enabling language models to self-edit is to move towards models
Catastrophic Forgetting
[1:35
(U:2
ELUe
0L23
OLE3
CUS4
C29
025
0.00
the ultimate goal of continual dearning-allowingg to incorporate new information over time, whether through agentically interacting with an environment or through standard training. While our earlier experiments assess how well SEAL adapts to individual edits in isolation, more ambitious goal is to support sequences of edits: can the model adapt to new information repeatedly while preserving prior knowledge?
0.38
Setf-Edit Iteration
0.36
0.59
0.29
0.45
0.45
0.00
<-
0.27
0.37
0.37
0.44
0.35
0.38
0.23
0.29
0.28
0.36
0.4
This question relates directly to the challenge of catas- trophic forgetting [73] [4], where new updates interfere destructively with past learning. We do not explicitly op- timize for retention in our current training setup. but we aim to establish a baseline for how well SEAL handles sequential self-edits without dedicated mechanisms for handling catastrophic forgetting. To test this, we simulate a continual learning setting in the knowledge incorpora- tion domain. The model receives stream of test passages, each triggering new self-edit. After each update, we
7- 0.19
0.25
0.22
0.25
0.28
0:36
0:38
0.0
8. 0.19
0.22
0.16
0.24
0.22
0.27
0:30
0.34
Passage Index
Figure 6: Catastrophic forgetting from continual self-edits. We sequentially update the model on new passages and track degradation prior tasks. Entry- wise standard errors are reported in [B6
8

---

re-evaluate the model on all previously seen tasks to measure retention. This setup tests the model's ability to integrate new edits without forgetting earlier ones.
As shown in Figure performance on earlier tasks gradually declines as the number of edits increases, suggesting that SEAL is still susceptible catastrophic forgetting. Still, it can perform multiple updates without complete collapse, indicating possibility for improvement. Future work could enhance this ability through reward shaping [75 76 77 to penalize regressions on earlier tasks, or by integrating continual learning strategies such as null-space constrained edits [78] or representational superposition [7.. In addition, since RL has been shown forget less than SFT, SEAL's inner loop could also employ RL instead of SFT [80].
Computational overhead. The TTT reward loop is significantly more computationally expensive than other reinforcement learning loops used with LLMs. For instance, reward signals based on human preferences typically involve a single model forward pass, and those using verified solutions may rely on simple pattern matching (e.g., regex). In contrast, our approach requires finetuning and evaluating an entire model to compute the reward- each self-edit evaluation takes approximately 30-45 seconds, introducing substantial overhead (see $B.5)
Context-dependent evaluation. Our current instantiations assume that every context is paired with an explicit downstream task: few-shot demonstrations arrive with held-out query pair, and each passage comes bundled with reference QA. This coupling simplifies reward computation but prevents RL training of SEAL from scaling to unlabeled corpora. A potential solution is to let the model generate not only self-edits but also its own evaluation questions-e.g. draft QA items or synthetic test cases for each passage-while the original content is still in context. These model-written queries could provide the immediate supervision required for reinforcement learning, broadening applicability to general training domains where external crosstionande and-answer answer sets unavailable.
6 Discussion and Conclusion
Villalobos et al. [8] project that frontier LLMs will be trained on all publicly available human- generated text by 2028. We argue that this impending "data wall" will necessitate the adoption of synthetic data augmentation. Once web-scale corpora are exhausted, progress will hinge on model's 's capacity to generate its own high-utility training signal. A natural next step is to meta-train a dedicated SEAL synthetic-de future models to scale and achieve greater data efficiency without relying on additional human text.
generator model that produces fresh pretraining corpora, allowing
We imagine future in which LLMs can ingest new data, such academic papers, and generate large quantities of explanations and implications for themselves using their existing knowledge and reasoning with the in-context data. This iterative loop of self-expression and self-refinement could allow models to keep improving on rare or underrepresented topics even in the absence of additional external supervision.
In addition, while modern reasoning models are often trained with RL to generate Mhaiin-ff-thougee (CoT) traces, SEAL could offer complementary mechanism, allowing the model to learn when and how to update its own weights. These two approaches could synergize: the model may choose to perform weight updates mid-reasoning to guide its current trajectory, after completing reasoning to distill key insights into its parameters-improne
future inference through internalized learning.
This continual refinement loop is also promising for building agentic systems woselss-moeels that operate over extended interactions and adapt dynamically to evolving goals. Agentic models must incremen- tally acquire and retain knowledge structured self-modification: after an interaction, the agent could synthesize a self-edit which triggers weight update. This could allow the agent to develop over time, aligning its behavior with prior experience and reducing reliance on repeated supervision.
they act. Our approach supports such behavior by enabling
SEAL demonstrates that large language models need not remain static after pretraining: by learning to generate their own synthetic self-edit data and to apply it through lightweight weight updates, they can autonomously incorporate new knowledge and adapt to novel tasks. Looking ahead, we envision extending the SEAL framework to pretraining, continual learning, and agentic models, ultimately enabling language models to self-learn and scale in a data-constraimed world.

---

Acknowledgments and Disclosure of Funding
We would like to thank Shivam Duggal, Idan Shenfeld, Seungwook Han, Jeremy Bernstein, Akarsh Kumar, Linlu Qiu, Juno Kim, Brian Cheung, Moritz Reuss, Ayush Sekhari, Zhang- Zhang-Wei Hong, Mehul Damani, Leshem Choshen, and Ryan Yang for their valuable discussions and feedback. We acknowledge support from ARO MURI grant number W91 1NF-23311-007 This research was also partly sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19- 19- 2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the United States Air Force or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein. We acknowledge the MIT Office of Research Computing and Data for providing high performance computing resources that have contributed the research results reported within this paper. This research was also partly supported by the Stevens Fund for MIT UROP research and by the MIT-IBM Watson Al Lab.
References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Pra- fulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Pro- cessing Systems, 2020, URL
neurips..... www.ips.cc/papprr.eeeeep wttp:ccpppppe////////opp h www.raatttttat.. html
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL http:::ttt//rrte
13971
[3] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models, 2024. URL https::///// htt///axivv../// .rg/ abs/2407 .27783
[4] Dirk Groeneveld, Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Sehwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannanch Hajishirzi. OLMo: Accelerating the science of language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for Computa- tional Linguistics, 2024. URL :tttttpp://aaaatttollii
org/ 2024. acl -ong..nn....m 841/
[5] Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang. Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang. Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang. Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. httLtttt: wtwpi//.wrrrab///222222 15115
[6] Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In
10

---

Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computa- tional Linguistics, 2020. URL patttps:/aaclanttollo/ .rg.
org/ 2020 a.l00...llmmaa. acl-main. 740/
[7] Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu. and Sanjiv Kumar. Modifying memories in transformer models, 2020. URL https://arxa org/abs/2012. 00363
[8] Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. ARC prize 2024: Technical report, 2025.
htapttps://axxi..ooom httg//bb/24113 .46044
[9] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL lattps://openrevieee stte/forummid-ggzz0000
[10] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong. Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open foundation models for code, 2024. httpthttss/aaaateei org 2308 12950
[11] Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad. Vinitra Swamy, Igor Krawezuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. MediTron- 70B: Scaling medical pretraining for large language models, 2023, URL https://arx:
16079
[12] Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. SauILM-7B: A pioneering large language model for law, 2024. httpss////ar abs/ 205/24433038883
[13] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2016. EtR.https:// htt.ttss/aaaaaahollll. org/D16-1264/
[14] François Chollet. On the measure of intelligence, 2019. URL 1911. 1911.01547
[15] Ronen Eldan and Yuanzhi Li. TinyStories: How small can language models be and still speak coherent English?, 2023. /tt./tttss//aaxxi. http://awwwv..rrr/aa/2////0.0../
[16] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Conti Kauffmann, Gustavo Henrique de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Behl, Xin Wang, Sebastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2024. URL https:///pperreviet net http:/orrmmmmddmtttme
[17] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2024. URL htttss///axxiii http://aaxxvvvor/raea// atggabs//3099.... 14316
[18] Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly, Rephrasing the web: A recipe for compute and data-efficient language modeling. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedingy of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguisties, 2024. URL httptpss//aclaaahologg..oo org/ 2024. acl-long, 757/

---

[19] Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegi. Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron CC: Transforming Common Crawl into refined long-horizon pretraining dataset, 2025. URL https::////aaa wwwpexiww...rrveeooo org/ abs/
02595)
[20]
Ruixiang Tang, Xiaotian Han, Xiaogian Jiang, and Xia Hu. Does synthetic data generation of LLMs help clinical text mining?, 2023. URL hattss// wwaeeiiiw..rrraeeoooo aa////////3333 04360
[21]
Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better synthetic data by retrieving and transforming existing datasets. In Lun-Wei Ku, Andre Mar- tins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics. Association for Computational Linguistics, 2024, URL https:://aclatttlllsaaa org. org/2024 findings- acl.3335 385/
[22] Yangjun Ruan, Neil Band, Chris J. Maddison, and Tatsunori Hashimoto. Reasoning to learn from latent thoughts, 2025. Ut.pttts::/ org/ w.tp//503.18..
[23] Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hanmaneh Hajishirzi. Self-Instruct: Aligning language models with self- instruc- tions. In Anna Rogers, Jordan Boyd-Grabed and Naoaki Okazaki, editors, Proceedings of the st Annual Meeting of the Association for Computational Linguistics.. Association for Compu- tational Linguistics, 2023. BtL.https:// htttttpss//aaaaalllllaaaooo
org 2023 acl-long 754/
[24] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with GPT-4, 2023. URL BtLLhttp::///
0
[25] Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. Synthetic continued pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL whtpt::tt///// http:::://ppeeeveeeeeee net/ net/fforunPid=Oyyyy00S11
[26] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning, Fast model editing at scale. In The Tenth International Conference on Learning Representations, 2022. L /tttpp:///penreviice
net/ forum??dd-DczzeittOP
[27] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing fac- tual associations in GPT. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems. Curran Associates, neurips. cc/paper_ httpl/appreflsssll////eeaaeee htlle//paper/2022////52
Inc., 2022. URL http::tts/////ccddinngg 6f Fr12556a222776900663333888222222 Paper- Conference pdf
[28] Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass- editing memory in transformer. In The Eleventh International Conference on Learning Representations, 2023. URL
[29] Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Eyal Shnarch, and Leshem Choshen. Achieving human parity in content grounddd datasets generation. In The Twelfth International Conference on Learning Representations, 2024. URL htttps:/oppnreee fett/forumiddRRjKKee YKTOOLW
[30] Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. Deductive closure training of language models for coherence, accuracy, and updatability. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computationnal Linguistics. Association for Computational Linguistics, 2024. URL https
www..ooy.ooooooooooo.o....o..... //aclanthology... .rg. 2024 1tt.22/2....nnnaacllla findings-acl..a
[31] Andrew K. Lampinen, Arslan Chaudhry, Stephanic C. Y. Chan, Cody Wild, Diane Wan, Alex Ku, Jörg Bornschein, Razvan Pascanu, Murray Shanahan, and James L. McClelland. On the generalization of language models from in-context learning and finetuning: a controlled study, 2025. htttttttt//axxiiii org/ abs/2505. 200.//0505000001111
[32] Core Francisco Park, Zechen Zhang, and Hidenori Tanaka. New News: System-2 fine-tuning for robust integration of new knowledge, 2025. httpstts:// wtwpiivv..rra///////eeea/ 22500
12

---

[33] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei A. Efros, and Moritz Hardt. Test- time training with supe-suuer for generalization under distribution shifts. In Proceedings of the 37th International Conference Machine Learning. PMIR, 2020. URL http:// proceedings.. mml.iingg.........mm.mm..m..e www.s/eessviiee htt99///nn..... .html
[34] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2022. URL wwtpthtttss//tpoeeeeetee fateelcc226644433laaaae520000000
www.rps..c//.epeeeeeeeee cc. /ttppp/e..llee/ppppp////////////
[35] Yu Sun, Xinhao Li, Karan Dalal, Chloe Hsu, Sanmi Koyejo, Carlos Guestrin, Xiaolong Wang. Tatsunori Hashimoto, and Xinlei Chen. Learning to (learn test time), 2024. URL https
7/arxiv. org/ brg/abs//a /2100. 13807
[36] Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning, wwtp///a.......
2025. URL htttts://arxiiii
[37] Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Chris- tiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2022. URL URL httppsss//proceeeddaas nwwrins.rnncmmm pttpppeefllesepperr/////// httppper_.ieeeeppprr///////// pttp:/paaap//22//leee (
pdf
[38] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL httph:://///t //wpiv/.///ii.../ www.xiiv...rreeeoo wbw. 05862
[39]
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. STaR: Bootstrapping reason- ing with reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2022. URL httt:ttttt::::/oeeeeettt www.:eeedddnnnnnnnen............ neurips. cc/ #3328a.72c04fbb64477666222d22200
httpiippppcppppp.ppeee htpppaaprrr____ filersfflll///e wttp//pppp//////11//222 Paper- Conez-Confeeee conference.pp pdf
[40]
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexan- der A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy EI- sayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pen- nington, Jiri Hron, Kathleen Kenealy. Kevin Swersky, Kshiteej Mahajan. Laura A Culp. Lechao Xiao, Maxwell Bileschi. Noah Constant, Roman Novak. Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha bohl-Dicksteen, and Noah Fiedel. Beyond hu- man data: Scaling self self-training for problem-solving with language models. Transactions on Ma- chine Learning Research, 2024. Utt.httts::/ppperviiiii n /tt//eerurt//e//e dooummidd-aAuunnnaeoooo
[41] DeepSeck-AL. Deepseek-R1: Incentivizing reasonin capability in LLMs via reinforcement learning, 2025. htt.:ttpp://arrvv.oo//aaa///////
[42] Jürgen Schmidhuber. Evolutionary priccco in self-referential learning, 1987. URL https
pdf
[43] Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning to learn using gradient de- scent. In Georg Dorffner, Horst Bischof, and Kurt Hornik, editors, ICANN. Springer Berlin Hei- delberg, 2001. https:/ttt:////n// h/ti://.....iiiingrrrrrrrr com/ cwwppeer/1..100773350000----0000
13
13

---

[44] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model- agnostic meta-learning for fast adapta- tion of deep networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMIR, 2017. URL. paattps://proceeeee
mlr wtt.a..peess////// http://iinnnnaaaaa..a htmll
[45] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Pieter Abbeel. RL?: Fast reinforcement learning via slow reinforcement learning, 2016. URL httph://arxxii org/abs/ httpp//1111.277711
[46] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn, 2017. URL hattps://artee
org/abs/1611 01761.0576....
[47] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In The Sixth International Conference on Learning Representations, 2018. URL
https://opeereviee:
[48] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta-reinforremmntt learning of structured exploration strategies. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems. Curran Associates, Inc.. 2018. URL atttps://proceedeeee htw.eeooeddiinnnnnnnnnreeeo pdf (007322381166555ecccccee111990
[49] Qi Sun, Edoardo Cetin, and Yujin Tang. Transformer-Squared: Self-adaptive LLMs, 2025, URL
[50] Jurgen Schmidhuber. Steps towards 'self-referentie neural learning: A thought experiment,
1992. URL httphtttt/pppopee..ieeee idsia httpc//eeeeeeeeeeeeeereeeeeeeeee pdf
[51] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. A modern self-referential weight matrix that learns to modify itself. In International Conference on Machine Learning. PMLR, 2022. UtL/attts://rrcceciinn
mlr ppress/v62/irreeee
html
[52] Chenmien Tan, Ge Zhang, and Jie Fu. Massive editing for large language models via meta learning, 2024. URL http::/tp://a wttp://.rrggbbb////1/// 04661
[53] Nathan Hu, Eric Mitchell, Christopher Manning, and Chelsea Finn. Meta-learning online adaptation of language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023. URL htap::::///allahhooog/y org/
honlp-maing 268/
[54] Tong Chen, Hao Fang, Patrick Xia, Xiaodong Liu, Benjamin Van Durme, Luke Zettlemoyer, Jianfeng Gao, and Hao Cheng. Generative Adapter: Contextualizing language models in parameters with single forward pass. In The Thirteenth International Conference on Learning Representations, 2025. URL htttp:::/ppenrevieeett
[55] Dan A. Calian, Gregory Farquhar, Iurii Kemaev, Luisa M. Zintgraf, Matteo Hessel, Jeremy Shar, Junhyuk Oh, András György, Tom Schaul, Jeffirey Dean, Hado van Hasselt, and David Silver. DataRater: Meta-learned dataset curation, 2025. URL
http://axxvv.rr/g//
2505. 17895]
[56] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen- Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield- Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from Al feedback, 2022. URLpst :/ap::://aaae http///x.vvvrr///// abs/
2212.08073.
14

---

[57] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHE: Scaling reinforcement learning from human feedback with Al feedback. In Proceedinggs of the 4/st International Conference on Machine Learning, ICML. '24. JMLR.org. 2024.
[58] Jing-Cheng Cheng Pang. Pengyuan Wang, Kaiyuan Li, Xiong-Hur Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self- improveeeve by reinforcement learning contem- plation. In The Twelfth International Conference on Learning Representations, 2024. URL
https:://ppenrevie. wwweeoooumddd-eeeeee
[59] Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, and Huaxiu Yao. CREAM: Consistency regularized self-rewarding language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https //openreview.o htt.://w.eeetttooommiaaaaaeaoodd
[60] Yuda Song, Hanlin Zhang, Carson Eisenach, Sham M. Kakade, Dean Foster, and Udaya Ghai, Mind the gap: Examining the self-improvement capabilities of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https
/openreview wwwperrvieew.et/forrrne
[61] Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguisties, 2023. URL https: http///acaaantthoooo/o
org/ 2023. emnlp- mwwn.miinn........ 67/
[62] Archiki Prasad, Weizhe Yuan, Richard Yuanzhe Pang, Jing Xu, Maryam Fazel-Zarandi, Mo- hit Bansal, Sainbayar Sukhbaatar, Jason Weston, and Jane Yu. Self-consistency preference optimization, 2024. URL httpp:://rxiiv.. org/
[63] Audrey Huang, Adam Block, Dylan J Foster, Dhruv Rohatgi, Cyril Zhang, Max Simchowitz, Jordan T. Ash, and Akshay Krishnamurthy. Self-improvement in language models: The sharp- ening mechanism. In The Thirteenth International Conference on Learning Representations, 2025. URL httpsttps:///
[64] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang. Xinwei Long, Ermo Hua, Biqing Qi. Youbang Sun, Zhiyuan Ma, Lifan Yuan, Ning Ding, and Bowen Zhou. TTRL: Test-time reinforcement learning, 2025. URLhtttps: /awxiv/.iv//v.////////a////.a/// org/
abs/2504. 16084
[65] Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?, 2025. URL htttp:::/aax....rg////////
2505 21444
[66] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https: :tttpp:///rziv.... www. /bb////a////a org/aba/
03300
[67] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL httptps://aaxvv.or////aa///
06347
[68] W. R. Gilks and P. Wild. Adaptive rejection sampling for gibbs sampling. Journal of the Royal Statistical Society, 1992. URL httpbtttt//ww.jjtttttt wttp::jjssooorrogooaboooooo wtw.:oorrrrrg/raaaa///////// 2347566
[69] Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. When should we prefer offline reinforcement learning over behavioral cloning? In The Tenth International Conference on Learning Representations, 2022. URL https: h/tp:::////peeeee //tppp/pppeeiiieee// net wwt/teeruuueee ooofm 7id= AP JP1MKT37rJ
[70] Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models, 2023. URL / /arxxvv..vvv///////a org/ abs/ 2208 01825
[71] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. HRRLhttps: //wxxxvxivvaa///////////aaaa// wtrg/abs/1500
15

---

[72] Edward Hu. yelong shen, Phillip Wallis, Zeyuan Alue Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022. URL https
//openreview.ne// http://wwee...eroomooodddae/aaaa
[73] Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. 1989. htstps::/www.siiencediiie.. www.scooocccccooc sciene//
arricle/p111000777741008080
[74] Ian J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. In The Second International Conference on Learning Representations, 2014. URL URI httpst////peeeeeeeeee vet/forrun2dd=o3557larra
[75] Yujing Hu, Weixun Wang, Hangtian Jia, Yixiang Wang. Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2020. URL
neurips... catps.cc/pape cccppppe__fite httpppprr..ileeeppppr/////////// 0709177559902241106222b8888 Paper.pd.
[76] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Tex:2Reward: Reward shaping with language models for reinforcement learning, 2024. httpp://////xii org/ a339 11489.
[77] Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in RLHF, 2025. URL https://arxi http:///axxvvv.///aaee/ hwrgg/ab/25502 18770
[78] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. AlphaEdit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https www.:eeooumddddaaayttttteeeeeeee
/
[79] Brian Cheung, Alexander Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Super- position of many models into one. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Buc, E. Fox, and R. Gareett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2019. URL httph:ttp////tt Utt/btttp://prrreedding .eurips. http:/ppperffleeppppppeeeeee 2atp://in4444aff3322022222666662 Paper. .df
[80] Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less, 2025. URL httpt:://axxiii org/ atsp//bb////099 04259
[81] Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will run of data? Limits of LLM scaling based human-generated data, 2024. BtRLhttss://xxxi>> org/ abs 2211. 04325
[82] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. GPT-4 technical report, 2024. URL httpst:/ta/aaaiiii org/ abs/ absgg//bb/333 08774
[83] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. DeepSpeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 20. Association for Computing Machinery, 2020. URL https / //doi. org/10.010.000 1145/ 3304406634067033
[84] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. URL. https:: :/dl acm. orm/doi/11// 1145/ 0145//300006..oo 3613165
16

---

[85] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, 2023. URL https: / arxiv. org/ 2tt.//311000060
17

---

A Experimental Details:
Few-shot Learning
A.1 Model and Setup
For the few-shot learning experiments, we use Llama-3. 2-1B-Instruct 3 as the base language model. Since this model has no specialized training on ARC, its ability solve ARC tasks is limited. To enable controlled evaluation, we curated small set of ARC problems from the training and evaluation splits that are solvable with optimal TTT hyperparameters.
Training Set: We selected 11 ARC tasks from the training set as the environment for RL optimization. Evaluation Set: We selected 8 distinct ARC problems from the evaluation set for measuring generalization performance. These were explicitly filtered for being amenable to TTT out of the full evaluation set.
These sets were chosen isolate the effect of self-edit learning rather than general ARC ability.
A.2 Training Procedure
We trained SEAL using a single outer loop of reinforcement learning on the training problems. For each problem, the model generated 15 self-edit configurations consisting of:
Data augmentation strategy: Controls whether to include variations such as basic exam- ples, size-based augmentations, chained reasoning, and repeated inputs: include_ basic, include _ize,, include _hhin, include peepeat.
Optimization configuration: Specifies hyperparameters such as learning rate, number of epochs, and loss function type (e.g., whether compute loss over all tokens or only the final output tokens).
Each configuration evaluated via test-time training (TTT), and assigned binary reward: if the adapted model produced the correct solution, 0 otherwise using Akyürek et al. [36] 's evaluation protocol. To encourage efficient adaptation, discarded configurations requiring more than 375 training steps, retained only configurations with reward and acceptable cost for LoRA-based SFT.
SFT for TTT was done with the following hyperparameters:
Parameter
Value
LoRA rank
128
LoRA alpha
16
Learning rate
N/A (model chooses
num Sxrain_epocha
N/A (model chooses it)
LoRA adapters were inserted into the following modules: q_proj, v_proj, gate_proj. down_proj. up-proj
For the RL SET used the following hyperparameters:
Parameter
Value
lora_rank
16
lora_alpha
16
num_traie rim_train_eepoht
8
sii.ddvcceetrrin_bbbe
5
learning_rate
5e-5
When generating the self-edits, add the following prompt suffix to the few-shot examples.
18

---

You configuring model training pipeline by selecting from predefined tools. You must make two decisions:
1. Data Generation Tools - For each of the following, choose true false:
use auggmntttgtsss
use sie_size_rre aungenttmennnn
use use_chain aatainn_augmentttionnss ise_repeat_angmmentttionn
2. Training Configuration- of: tokenss
Also specify:
learning_na _atennggreeeeaa (float)
num train _eteeeeeechsseeee (integer)
Output Format
Respond with valid JSON object. Do not include any explanation, markdown, or extra text. Use lowercase true/false for booleans and ensure correct JSON syntax. Example output:
"data_ gentraagnerttion: {
"use www.baaicccaummentttiiiii
"use_ssiaa_aummmntttiiisss""
"uss__chainn_agggennttttto""""" iuse_repeat.auggeentaiioom
"training": {
"strategy":
"learning_rate rate": ....
coum_train_epochet
}
A.3 Evaluation Details
For each of the held-out evaluation tasks, the model generated self-edit configurations, yielding total of 40 configurations. Success was measured the percentage of configurations that led correct outputs after adaptation. We followed the evaluation protocol from Akyürek et al. [36].
For the Oracle TTT used the following configs:
Parameter
Value
lora_rank
128
lora_alpha
16
num_ tettttrainnepeete
batch_size 2
learning_rate
lc-4
A.4 Compute Resources
performed training runs on a single A 1000, H100, or H200. Each TTT per problem requires between half a minute to few minutes, which is also why we limited the number of samples for ReSTEM and additionally limited the number of gradient steps allowed per self-edit TTT. Overall RMSTEM took around 2-3 hours.
19

---

B Experimental Details: Knowledge Incorporation
B.1 Model and Setup
We the (rren-255-77 base model [S] in the knowledge incorporation experiments. We repurpose the SQuAD dataset v1.1 [13] for the task of answering questions without the passage in-context. We use the training set for RL training and 200-article subset the evaluation for evaluation. Within the training set and evaluation set, there are some overlapping topics of passages, but there no overlap between these sets, so we can be sure that there is no data contamination of the test passages due to RL training.
B.2 RL Training Procedure
We run 2 rounds of ReSTEM training 440] On each round, take batch of 50 context-questions answers triples from the SQuAD training set. For each context, we sample self-edit generations at temperature 1. We evaluate each self-edit over random seeds, training on the sequences and then evaluating the updated model on the corresponding questions. We average each generation's results over seeds and then keep the single best generation for each of the 50 contexts. Finally, finish the round of ReSTEM we perform supervised finetuning on the 50 resulting prompt-completion pairs.
Supervised finetuning here is done with batch size of 10, for 2 epochs, with learning rate 3e-4, using LoRA 72 with rank 64 and alpha 128, applied to all MLP and attention projection layers.
B.3 Synthetic Data Generation and Finetuning Details
In all models, we generate synthetic data by prompting to generate implications of the passage:
Let's read the following passage and produce list of implications derived directly or indirectly from the content.
Passage: (passage)
Implications:
We then take the resulting generated sequence. In the single-passage case, we split it by newlines into a set of training documents. the multi-passage case, we use the full generated sequence as a single training document. In the case of synthetic data from GPT-4.1 (gpt-4. 1 20-220020--11 instruct-model. we additionally have the following rule: If the second line begins with a "1." then we omit the first line from the training set. This is because we found that the first line often contained filler text (e.g. "Sure, here is the list of implications:").
an
We then use the following training hyperparameters:
Table 3: Single-Passage Knowledge Incorporation Hyperparameters
Parameter
Search Space
LoRA Rank (r)
[32, 64]
LoRA Alpha (a)
[32, 64]
Learning Rate
[1e-4, 3e-4, 5e-4, 1e-3, 2c-3]
Epochs
[1, 10, 15, 20]
Batch Size
[1,4]
In the multi-passage 200 case, sample self-edit completions for each passage and take the aggregate dataset of all self-edits across all passages to train on.
To answer the corresponding questions, use the following prompt:
20

---

Table 4: Multi-Passage Knowledge Incorporation Hyperparameters
Parameter
Search Space
LoRA Rank (R)
[32, 64]
LoRA Alpha (o)
[32, 64]
Learning Rate
[1c-4, 3c-4, 5e-4, 1e-3, 20-3]
Epochs
[1. 5,
Batch Size
[1. 4, 16]
Let's answer question directly and concisely.
Question: {question)
Answer:
B.4 Evaluation Details
evaluate on a 200-passage subset of SQuAD evaluation set, consisting of a combined 974 evaluation questions (roughly corresponding each passage). The pipeline of generating synthetic data and finetuning on it the same as above. For automated grading, we use gpt-4. 4. 1 1000---20000 11025-04-111 via the OpenAI API with greedy decoding.
82
The grading prompt is as follows:
You grading assistant. Your job is to determine whether student's 's answer correctly answers the question based solely on the provided gold answer. Do not use any outside knowledge. The student answer can include additional information, but it must at least fully convey the gold answer and must not contradict it. Ignore style, phrasing, or extra details that do not affect correctness. Respond ONLY with 'yes' 'no'.
Question: (question)
Gold answer: (gold)
Student answer: (pred)
the student answer correct based solely on the gold answer? Respond 'yes" or 'no'.
B.5 Compute Resources
All experiments are performed on 2 H100 or 2 H200. We use DeepSpeed ZeRO-3 83 for SFT in RM training. We use vLLM (84) for efficient inference. The most compute- compute-intensive intensive portion of our training and evaluation is the E-step of ReSTEM training, where the model generates completions and is graded through the inner-loop process of finetuning and running inference. Doing single round requires a batch of 50 passages over completions and 3 runs per completion, meaning 750 inner loop iterations. This takes about 6 hours on 21000..
B.6 Standard Error of the Mean in Catastrophic Forgetting Experiment
The standard errors of the mean (SEM) for each entry in Figure 6 is shown below in Table B.6
B.7 Scaling Model Size
We further experimented with the 3B 3B-parameter Qwen variant, with the same singlc-passage setup as in Figure The results are given in Table 6
To compare the benefit of SEAL over using self-edits generated by the base model, we compute the ratio of SEAL's improvement over the base model to the improvement from base model self-edits. This ratio is 1.75x for the 3B model and 2.04x for the 7B model. The relative improvement is greater for the 7B model, which provides some evidence that not only are stronger base models more effective at leveraging synthetic data for colf-adaptation, but reinforcement learning may have compounding
21

---

Table 5: Entrywise standard errors of the mean (SEM) across continual self-edits experiment.
2
3
4
6
7
0.0306
0.0315
0.0263
0.0318
0.0297
0.0370
0.0310
0.0284
0.0273
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
2
0.0305
0.0277
0.0000
0.0000
0.0000
0.0000
0.0000
0.0000
0.0277
0.0358
0.0406
0.0000
0.0000
0.0000
0.0000
0.0000
0.0272
0.0303
0.0337
0.0320
0,0000
0.0000
0.0000
0.0000
0.0296
0.0342
0.0290
0.0298
0.0319
0.0000
0.0000
0.0000
0.0289
0.0334
0.0271
0.0258
0,0320
0.0337
0.0000
0.0000
0.0255
0.0313
0.0264
0.0253
0.0309
0.0331
0.0363
0.0000
0.0237
0.0307
0.0211
0.0267
0.0273
0.0271
0.0358
0.0263
Table 6: Model Size Scaling Performance (%).
Model
Base Model (No Training)
Base Model Self-Edit
SEAL
Qwen2.5-3B
25.1 32.7
31.9 39.7
37.0
Qwen2.5-7B
47.0
benefits as model capacity increases. We acknowledge that is hard to draw conclusions though without actually scaling up further.
B.8 Comparison to Generative Adapter
We additionally compared with Generative Adapter [54]. a hypernetwork approach that generates LoRA weights from context, using our evaluation setup. Table reports results for both single- passage (n=1) 1) and continued pretraining (n 200). We use the Mistral-7B-ban Generative Adapter, since that was the closest model for comparison. All values are on the same evaluation set, but CPT batches updates over all documents while single-passage trains and evaluates an adapter separately for each document. Generative Adapter achieves strong performance in the n=1 case, but underperforms SEAL in the CPT setting. SEAL's parameterization of weight updates through synthetic data generation allows reuse of generated data for CPT, application to arbitrary base models, and flexibility to learn updates from diverse interaction types beyond LoRA finetuning.
model [85 for
Table SEAL vs. Generative Adapter Performance (%).
Model
Base
Single-passage (n=1)
CPT (n 200)
SEAL
32.0
47.0
58.2
Generative Adapter
24.4
66.8
28.0
We note that parameterizing weight updates via synthetic data generation rather than directly predict- ing LoRA weights has several advantages: (1) generated data can be reused for CPT or applied to arbitrary base models, (2) models can leverage reasoning and restructuring as document scale and complexity grow, and (3) the framework is not restricted to LoRA finetuning, allowing for many different update types, including those arising from environment or user interactions. By contrast, it is unclear how hypernetwork-Be -based approaches would scale to such settings, while next-token prediction on generated data naturally exploits model's in-context learning capabilities.
B.9 Comparison to Entigraph
We additionally compare SEAL to Entigraph (25] in the Synthetic Continued Pretraining (SCPT) setting on SQuAD. Results for both 200 and 2067 passages are shown in Table g SEAL uses the same 5 synthetic data generations per document. For Entigraph, we sample 5 synthetic data generations involving pairs and 5 triplets of entities per document. Entigraph with all 10 synthetic data generations sampling is competitive with SEAL, especially at the larger scale. These results
22

---

suggest that RL-trained self-edits and structured heuristic methods are both strong approaches for synthetic data generation.
Table Synthetic Continued Pretraining (SCPT) on SQuAD (no passage in context), Best in each column is bolded.
Method
Continued Pretraining (u=200)
Continued Pretraining (n=2067)
SEAL
58.2
46.4
Entigraph (pairs)
46.2
38.6
Entigraph (pairs+triples)
56.0
48.6
B.10 Proxy Reward
We experiment with replacing the inner loop with proxy reward based on human-crafted rubric with categories: length, diversity, quality, and correctness. GPT-4.1 grader scores each category on a 1-5 scale, and the sum of these scores is used the RL reward. Table repoors final results and RL training times.
Table 9: Full Reward vs. Proxy Reward Performance (%).
Model
Base
Post-RL
Time
SEAL
32.0
47.0
=6 hr
SEAL w/ Proxy-Reward
32.0
45.6
R5 min
While further tuning of the rubric or metric design could strengthen the reward signal, the advantage of the full SEAL loop is that no such manual specification is required-thh model directly learns which edits improve own performance. Both approaches appear promising for scaling to larger model sizes and compute budgets: proxy metrics offer dramatically lower cost, and with refinement, they may even surpass the "true" reward of directly optimizing for post- post-finetuning performance.
B.11 Prompting
Recent works have shown that reinforcement learning baselines and outcomes can highly sensitive to prompting. We experiment with 6 additional self-edit prompts in the snowledge-incorporan setting. The seven prompts-impil implications implications-long implications -erettiion-veerlooooo -very-long, ampplicaaions-chain-aa -ff-huught, rewrite, self-qa, and to-prompt-an are shown below. All results the main content of the paper used the implications prompt, which we consider to be the most prototypical [30 31. However, prior work has found prompts involving rewriting or generating question-ansee answer pairs can be more effective, as discussed $
Furthermore, as we see qualitatively in Figure RL appears to have dramatically increased the length of the response of the example. We therefore experiment with prompting for longer generations with impl catciannnnooonn and vverccatooossvry--aeeee through prompting alone.
to test if we can achieve similar gains
The results are shown in Table 10 Notably, the baselines for
and rewrite the RL-trained version of implications However, using these prompts the base of RL training yields even greater improvements. In all cases, RMSTEM enhanced performance by roughly to 11 percentage points.
Here, "Chain-of- taiiio-f--touggtteeel""" refers to having the model reason before answering the questions (letting the model "pull out" information from its weights), rather than chain-of-thouge before generating synthetic data, which is done with the base coplications prompt. However, we did not notice substantial difference in our setting when thain-of-thoughe was applied, whether before answering and before writing synthetic data.
Letting the model "determine its own" self-edit format, with no-prompt, was not able to achieve the same results as predefined prompting formats in our experiments, achieving only 18.9% after rounds of training.
23

---

Method
Original
Round 1
Round 2
gpt-4.1 synthetic
No self-edit
33.5
Implications
39.7
43.7
47.0
46.3
Implications-long
49.3
52.4
54.4
54.1
Implications-very-long
45.0
51.5
52.1
40.9
Rewrite
49.4
55.3
55.6
54.4
Self-QA
42.8
48.7
39.2
No-Prompt
13.8
12.7
18.9
28.6
Impiications-ccaan-----oooggg
38.7
Chain-af-thought-eval
37.8
Table 10: Performance across rounds of ReSTEM RL training on various prompts in the single- document knowledge incorporation setting. The gpt-4. column reports performance using synthetic data generated by gpt-4.1 with the corresponding prompt format.
The five prompts are shown below.
impl catticctionn ins
Let's read the following passage and produce list of implications derived directly indirectly from the content.
Passage: {passage)
Implications:
implications -lng
Let's read the following passage and produce a long list of implications derived directly or indirectly from the content.
Passage: {passage)
Implications:
implicatioos--very-ll
Let's read the following passage and produce very long list of implications derived directly or indirectly from the content.
Passage:
}passage)
Implications:
implications chain--chii--aee of of- -hain-of-thought
Let's read the following passage, think step by step, and then produce list of implications derived directly or indirectly from the content. We should first generate a "Thought Process" and then "Implications"
Passage: (passage)
Thought Process:
24

---

no co-prompt
passagee }
rewrite
Let's read the following passage and rewrite it in few different ways, each one separated by a newline.
Passage:
{passage)
Rewritten passages:
self-qg
Let's read the following passage and rewrite it in question-answer format.
Passage: (passage)
Question 1:
For self we apply additional formatting so that training documents consist of ques- tion-answer pairs, rather than using our standard approach of splitting by newline characters. Specifi- cally, we split the output using occurrences of "Question n:" instead of newlines.
Note:
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Abstract & $1 clearly list contributions and scope; results sections match them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
$5 "Limitations" enumerates strong assumptions, compute cost, and
catastrophhi--foreettnng issues.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
The paper contains no formal theorems only algorithmic descriptions ($3).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All training/evaluation pipelines, hyperparameters, and prompts detailed in Appx. A-B.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer:
[No]
Justification:
Code and data will be released upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips.com wttpcccpppbccccceec guwde//ddeeeeee Cooeeooodeesssseeeeeee http:///d///uuiiiiddooo////ee/e/
ssinisiiccyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Datasets, splits, model sizes, LoRA ranks, LR grids, etc. specified in Appx. A-B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Discrete success-rate metrics reported without error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Information compute resources is given in Appx. A-B.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/neurips. http:///ccctctttecttttttett
Answer: [Yes]
Justification: Authors reviewed NeurlPS Code of Ethics and verify compliance. We do not foresee any potential harms.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not foresee immediate societal impacts, as this is foundational research focused on self-adaptive model training. The method is not deployed or tied to sensitive applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do not anticipate any risks for misuse. Our models have similar risk of misuse as the base Qwen and Llama models used.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets (Qwen-2.5, Llama-3, SQuAD, ARC) cited, with licenses properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Study uses only public text corpora; no human-subject experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human-subject research performed, hence IRB needed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
31

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used as important, original, or non non-standard components in the development of the research methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttps//////u.p...eeee. wwt.rrippp.ccooooeeeeeeeee /t/Cooffeeencee/2222222 for what should or should be described.
32


Reviews:
[{"review_id": "R4fg7qfpgG", "reviewer": "Reviewer_nQa9", "summary": "The paper proposes a new framework, SEAL, for model learning from self-generated data. SEAL includes a RL phase to optimize data generation, aiming to better adapt the model to given downstream tasks. The authors analyze the effectiveness of SEAL by evaluating it on two tasks: the few-shot ARC benchmark and knowledge incorporation.", "questions": "I am curious whether model collapse becomes a problem when training on self-generated data. Is this related to the number of adaptation steps or the amount of synthetic data used during adaptation?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strength\n\n- Self-adaptation is an essential ability in human learning, but effective methods for enabling it in transformer-based language models are currently lacking.\n- Learning from self-generated data is a general and practical direction for enabling self-adaptation.\n\nweakness\n\n- Although self-adaptation is an important challenge, SEAL only explores a narrow case: test-time adaptation to new instances of the same task seen during RL training, rather than to unseen tasks. This limits the broader significance of the paper. Both training and testing focus on visual reasoning and knowledge incorporation. It would be more compelling to include experiments where RL is applied on few-shot tasks from benchmarks A, B, and C, and then tested on distinct benchmarks X, Y, and Z.\n- Given that the experiments focus on adapting to new instances within known tasks, it is unclear how much the RL training improves over carefully designed synthetic data pipelines crafted by humans. For example, in knowledge injection, [1] seems to provide a more structured approach for document augmentation. In few-shot learning, it would be interesting to see comparisons with synthetic data generated by different models.\n- Some additional ablation studies would strengthen the paper. For instance, why was ReST chosen as the RL algorithm? Does the size of the RL training dataset affect performance? Is it necessary to generate different augmentation strategies or optimization configurations?\n- Regarding parameter adaptation from demonstrations or instructions, prior work on hypernetworks should be discussed more thoroughly. For example, [2] is relevant and should be considered.\n\n[1] Yang et al., \"Synthetic Continued Pretraining.\" ICLR 2025\n\n[2] Chen et al., \"Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass.\" ICLR 2025", "was_revised": true, "final_justification": "How to generate synthetic data for a given task is an important question in practice. The paper provides an RL framework to improve data synthesis, given the downstream evaluation as the reward. The paper has a bit of weakness in its evaluation setup. They only evaluate on two tasks: knowledge injection and few-shot learning, while in practice, we can imagine there are many more scenarios that involve adaptation, like personalization chatbots and customized tool agents where the model needs to learn user preferences, procedural knowledge, and new function calls, etc. Also, they train a data generator for each task separately. This limits their method from being used as a general tool to adapt the model to any task. Despite this, during the rebuttal, the authors provide a thoughtful discussion on a comparison with human-designed data synthetic pipelines and other model adaptation paradigms like hypernetworks, which significantly strengthens the paper. So, I recommend a score of 5.", "rebuttal": "We thank the reviewer for their attention and helpful feedback! We will address each weakness and question.\n\n**Response to weaknesses:**\n\n1. **Generalization.** Our evaluation still uses an unseen test set from the distribution, and our model’s ability to self-edit generalizes within that scope. The general knowledge experiments were meant to simulate learning from arbitrary training data. We focused on document QA for its ease of verification, but concede that RL training on only this likely would not generalize to many other types of LM training data. We also don’t claim that self-editing on ARC tasks would transfer to entirely different few-shot learning tasks. However, we believe that even within these datasets, generalization is a challenge.\n\n2. **Prompting & Heuristic Pipelines.** This is a great point. We have experimented with a variety of prompts and structured methods for synthetic data generation—ranging from different prompting strategies for SEAL or GPT‑4.1 to using entigraph [1] for continued pretraining—within the general knowledge incorporation domain. The table below includes the additional experimental results, showing that SEAL consistently delivers performance gains across each base prompting strategy, and matches or exceeds [1] for continued pretraining in our setting. This table will be included in the updated version of the paper. \n\n3. **Algorithm Ablations.** We chose ReST-EM because it was simple and worked well, and we had a difficult time stabilizing GRPO in our meta-learning setting. However, to scale up SEAL, we agree that moving to more sophisticated RL methods would be beneficial. In the few-shot domain, we already know that augmentation strategies and the optimization parameters make a difference (e.g. moving any optimization parameters off of the sweet spot would lead to worse performance). It’s unclear what would be a “fair” point to fix one while letting the model generate the other, so we did not run any additional experiments here.\n\n4. **Hypernetworks.** Thank you for pointing out the related work. We will include additional discussion on hypernetworks. We ran Generative Adapter [2] with our evaluation setup and have discussed the results at the bottom of this response.\n\n**Response to Questions:**\n1. **Model Collapse.** Our experiment in the Limitations section shows that continual gradient steps (without maintaining optimizer states or batching) leads to model collapse. However, batching the updates recovers much of the performance, at least on the task at hand (see table 2 on continual pretraining experiments). Additionally, we’ve run larger continual pretraining experiments with full finetuning rather than LoRA and provided them in the table below. Notably, accuracy does not degrade but rather increases to 58.2% for n=200 and remains high at 46.4% for n=2067 when updates are batched, compared to training and evaluating a separate LoRA adapter for each document (47.0%).\n\n## Main Additional Experiments\n\nSince there is no option for a global response, we provide the bulk of our additional results and explanations below, and reference them throughout our individual responses.\n\n### Prompting\n\n| Method                     | Original | Round 1 | **Round 2** | gpt-4.1 synthetic |\n| -------------------------- | -------: | ------: | ----------: | ----------------: |\n| No self-edit               |     33.5 |       – |           – |                 – |\n| Implications               |     39.7 |    43.7 |    **47.0** |              46.3 |\n| Implications-long          |     49.3 |    52.4 |    **54.4** |              54.1 |\n| Implications-very-long     |     45.0 |    51.5 |    **52.1** |              40.9 |\n| Rewrite                    |     49.4 |    55.3 |    **55.6** |              54.4 |\n| Self-QA                    |     37.3 |    42.8 |    **48.7** |              39.2 |\n| No Prompt                  |     13.8 |    12.7 |    **18.9** |              28.6 |\n| Chain-of-thought-self-edit |     38.7 |       – |           – |                 – |\n| Chain-of-thought-eval      |     37.8 |       – |           – |                 – |\n\nA central critique was that our self-adaptation procedure is not sufficiently general, or that we did not compare with other methods for generating synthetic data. In the knowledge QA task, we’ve added results showing that SEAL remains effective across a variety of self-editing instructions. This demonstrates that within the general knowledge incorporation domain, SEAL is not tied to a single self-edit format. \n\nThe “implications-long” and “implications-very-long” columns prompt for longer generations.  While none of the reviewers pointed this out explicitly, we wanted to test whether prompting could match the gains from RL, since we saw that RL training naturally increased the lengths of generations (for example, see Figure 5). We find that many prompts achieve higher scores than our main “implications” prompt, but that RL on top of them increased results by similar margins, meaning that the improvement could not simply be attributed to length. \n\nHere, “chain-of-thought-self-edit” refers to having the model reason before providing the synthetic data, while “chain-of-thought-eval” refers to having the model reason before answering the questions (letting the model “pull out” information from its weights). Both are done with prompting for \"implications.\" The \"gpt-4.1 synthetic\" column refers to finetuning the base Qwen2.5-7B model using data generated by gpt-4.1 using the corresponding prompt. \n\nFurther analysis is given in specific responses to reviewers.\n\n### Synthetic Continued Pretraining\n\n| Method                    | Continued Pretraining (n=200) | Continued Pretraining (n=2067) |\n| ------------------------- | ----------------------------: | -----------------------------: |\n| Base                      |                         32.0 |                          29.0 |\n| Train on Passage            |                         36.0 |                          31.2 |\n| Train on Passage + Base Model Synthetic Data           |                         50.6 |                          43.4 |\n| SEAL                      |                     *58.2* |                          *46.4* |\n| Entigraph (pairs)         |                         46.2 |                          38.6 |\n| Entigraph (pairs+triples) |                         56.0 |                          *48.6* |\n| Train on Passage + GPT-4.1 Synthetic Data         |                     **59.4** |                      **49.2** |\n\nHere, we run additional continued pretraining experiments. These experiments are done with the same methodology as the ones in the original paper, except we do full finetuning rather than LoRA. We experiment with n=2067 documents, which is the full validation set. Entigraph [1] has two settings—the second (pairs+triples) creates 10 generations per document, while the first (pairs) and all other methods create 5. \n\nSurprisingly to us, with full finetuning, we find that performance *improved* over the n=1 regime in which we RL-trained our model to self-edit. Performance was highest using a batch of n=200 documents, but remained strong at n=2067. We conjecture that this is due to issues with gradient updates over small amounts of data that are encountered with n=1 document, even with strong hyperparameters.\n\n### Comparison to Generative Adapter\n\n| Model              | Base | TTT (n=1) | CPT (n=200) |\n| ------------------ | ---: | --------: | ----------: |\n| SEAL               | 32.0 |      47.0 |    **58.2** |\n| Generative Adapter | 24.4 |  **66.8** |        28.0 |\n\nWe ran Generative Adapter [2], a hypernetwork approach to generating a LoRA adapter based on context, with our evaluation setup. All 6 values in the table are on the same evaluation set, but CPT batches updates over all documents while TTT trains an adapter and evaluates each one separately. We used the Mistral-7B-based model for Generative-Adapter, and it achieved a performance of 66.8% on n=1 document, outperforming SEAL, and 28.0% on n=200 continued pretraining, greatly underperforming SEAL. \n\nHowever, we would like to highlight some key differences between the methods. SEAL parameterizes weight updates in the generation of training data, which (1) allows for reusing data for continued pretraining or on arbitrary models and (2) allows models to leverage reasoning as documents scale in size and complexity. SEAL is also not restricted to LoRA finetuning, and is much more general in that it potentially allows for many different types of updates given a base document beyond just memorizing a piece of data. For example, perhaps we wish to learn and do a weight-update from an environment/user interaction. It is unclear how hypernetworks could scale to such tasks, while generating data for next-token prediction could leverage the in-context learning abilities of the model."}, {"review_id": "fPeGjeso0d", "reviewer": "Reviewer_D8Zo", "summary": "This paper studies the direction of enables LLMs to self-adapt by automatically editing the data format. First, the model generates the refined context and then adapts the original LLM on this context. The contexts, which correspond to improved downstream performance, are marked as high rewards, and the model would be updated to maxmize these rewards.", "questions": "Q1. In Step 4 of Algorithm 1, how can the authors ensure that the self-edit generated based only on $C$ is suitable for downstream $Q$? Is it possible that one context $C$ corresponds to multiple different $Q$, making a context-only self-edit suboptimal for all associated $Q$? Would it be more appropriate to generate self-edits conditioned on both $C$ and $Q$? Furthermore, how is the type of self-edit (e.g., rewriting, augmentation) selected or controlled, given that different $Q$ types may require different editing strategies?    \nQ2. The formula is missing a sequence number.   \nQ3. In Section 4.1 and 4.2 (the main results), the adopted baselines are too simple. None of the recent work in Related work are used in this experiment, making the results more like ablation study.  \nQ4. In few shot learning, what will happen if the self-edit does not include optimization configurations, e.g., learning rate, number of training epochs?   \nQ5. The authors have give some examples to intuitively explain why editing the data format is helpful. However, an in-depth theoretical analysis can be more convincing and improve the quality of this paper.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:  \nS1. The studied problem is important and interesting.  \nS2. The paper is well-motivated and easy to follow.  \nS3. The proposed method is novel.  \n\nWeaknesses:  \nW1. The methodology lacks clarity on how context-specific (context denoted as $C$) self-edits generalize across multiple downstream tasks (downstream tasks denoted as $Q$).  \nW2. The formula is missing a sequence number.   \nW3. Lack of competitive and recent baseline methods.  \nW4. The contribution of different components of self-edits (e.g., optimization configurations) is unclear.   \nW5. Lack of in-depth theoretical analysis about why editing the data format can benefit the downstream performance.\n\nThe details and corresponding questions about these weaknesses can be found in Questions below.", "was_revised": true, "final_justification": "I recommend Accept. My main concerns are addressed after author rebuttal.", "rebuttal": "We thank the reviewer for their constructive feedback! We address the reviewers weaknesses and questions below.\n\n**Response to Weaknesses/Questions:**\n\n1. **Prompting With Both Context and Questions.** Prompting the model on both C (context) and Q (downstream task) might be useful in some settings, but we wish to be able to self-edit in response to any document C without knowing ahead of time what downstream tasks Q we will encounter. We also agree that our current setup uses task-specific self-edit strategies, while further work might allow for the model to employ an even larger pool of strategies in response to different types of data. One potential extension, as you suggest, is to train across a diverse set of tasks, each with its bespoke self-editing method, and allow the model to learn to select an appropriate self-editing strategy for new tasks.\n\n2. **Typo in Formula.** Could you clarify which formula you are referring to?\n\n3. **Comparison with Related Work.** Thank you for the feedback. We’ve added comparisons to a number of recent baselines and prompting methods for generating synthetic data for a task. See the table in our response to reviewer nQa9 for these new results. The key point is that some prompting methods do better than \"implications,\" but RL on top of these prompts improves accuracy further, by similar margins. Entigraph, a human-crafted synthetic data generation pipeline, does comparably to SEAL in our continued pretraining setup. Generative-Adapter, which aims to directly generate the LoRA adapter weights, does better in the single-document regime, but has many other limitations. See our discussion at the bottom of our response to reviewer nQa9.\n\n4. **Contribution of Components of Self-Edits.** Without training for self-edit generation, performance in the few-shot domain drops to 20%. For each of the parameters such as learning rate, number of epochs, etc., setting them far off the “sweet-spot” leads to a substantial decrease in performance. In this sense, all of these parameters are important. It’s unclear what would be a “fair” point to fix one while letting the model generate the other, or what this would tell us, so we did not run any additional experiments here. These experiments are meant to show that signal from transiently adapted models can reinforce better self-generated hyperparameter selection through RL, which we believe was shown.\n\n5. **Theoretical Analysis.** While we do not currently provide theoretical contributions, we agree that this represents an interesting direction. Prior work ([1], Yang et al., Synthetic Continued Pretraining) demonstrates from a theoretical perspective how strategically rewriting data can greatly improve the scaling behavior of learning on a set of data. Here, we do not enforce any particular structure such as in [1], so it’s hard to give additional theoretical insight. Instead, we see our method as towards automating the discovery of good synthetic data formats such as that identified in [1]."}, {"review_id": "EY8RpxSeUx", "reviewer": "Reviewer_hhPg", "summary": "The paper proposes SEAL, a framework using RL to guide models in generating useful self-edit data.", "questions": "1. Beyond Figure 5, is there systematic analysis of what makes a `good` self-edit (e.g., atomicity, abstraction)?  \n2. Can the inner-loop reward signal be approximated more efficiently (e.g., proxy models) for scalability, especially in pretraining?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:  \n1. The idea is interesting and intuitive, addressing the limitation of static and passive learning in LLMs.  \n2. SEAL succeeds in two distinct tasks (i.e., knowledge integration and few-shot abstract reasoning) demonstrating its versatility. In knowledge integration, SEAL's synthetic data outperforms GPT-4.1, highlighting the potential of tailored learning materials. On ARC tasks, success rates improved from 0% to 72.5%.  \n3. The paper is well-structured, with clear explanations and helpful figures (Figure 1, 2, 3).  \n\nWeaknesses:  \n1. The generalization of learned policy is unclear. The framework optimizes predefined self-edit formats (e.g., implications) but may not adapt to entirely different formats (e.g., code, math). Its robustness on harder, uncontrolled tasks (e.g., full ARC benchmark) is uncertain.  \n2. Experiments are limited to small-scale scenarios (single or 200 paragraphs), far from real-world continuous learning or pretraining. Larger-scale tests are needed to assess interference and shortcut learning risks.  \n3. Binary rewards (success/failure) may encourage hacking—generating edits tailored to specific evaluation sets rather than genuine understanding. Many real-world scenarios lack predefined evaluation metrics.  \n4. Fairness in comparisons: GPT-4.1 used a generic prompt; optimized prompts might improve its performance. \n5. The computational cost of SEAL is high (6 hours per RL round).  \n5. Only smaller models were tested. Would larger models (e.g., 70B) benefit less from SEAL due to better inherent self-editing ability?  \n6. Flexibility is overstated. Self-edit formats are predefined (e.g., implications, JSON), making SEAL more like automated prompt engineering than true `learning to learn`.", "was_revised": true, "final_justification": "The rebuttal resolved my main concerns.", "rebuttal": "We thank the reviewer for their thought-out response! We address the proposed weaknesses and questions below.\n\n**Response to Weaknesses:**\n\n1. **Generalization.** The QA task is meant to simulate general training data. We focused on document QA for its ease of verification, but concede that RL training on only this likely would not generalize to many other types of LM training data. While we don't have experiments specifically on math/code, we believe the method would still work. \n\n2. **Larger-Scale Experiments.** We have to start somewhere! We've added a larger scale CPT result (all 2067 passages of the validation set, with full finetuning rather than LoRA) to the paper, discussed in the response to reviewer nQa9. This shows the self-edit policy scales well as we increase the amount of ingested data, so long as they are applied in a batch. As we point out in our limitations section though, sequential self-edits (without maintaining optimizer states, replay, etc.) remain prone to catastrophic forgetting.\n\n3. **Reward Hacking.** We believe rewarding the ability to answer questions about a passage, as well as applying a few-shot rule to a novel input, are fairly robust to reward hacking. Note that these questions are not seen by the model while generating the self-edit. Is there a specific failure case you imagine?\n\n4. **Prompting.** We've added a much more in-depth analysis across different prompts (see response to review nQa9).\n\n5. **Computational Cost.** Computational cost is indeed high, but this is a fixed cost. After training the model to perform self-edits (or training a teacher model to generate strong synthetic data), we can then use this model for data generation indefinitely without further expensive meta-learning.\n\n6. **Scaling Model Size.** We didn't have the capacity to run experiments on 70B models (especially with our meta-learning RL setup). However, we hypothesize that scaling will *greatly improve* self-editing ability (both before and after RL). In particular, the ability to generate synthetic data given a document is governed by the model's in-context learning ability, and its ability to generate long sequences of data conditioned on the original passage. Scaling increases context/generation length and the ability to synthesize \"new\" information in context, given the original piece of data and prior knowledge. To provide some evidence that SEAL might show greater benefits over *base-model* synthetic data, we've run an additional experiment scaling *down* to 3B. See our results given in the response to reviewer oBBv. The key point is that the the relative increase in performance with SEAL over using the base model for synthetic data generation increases with model size.\n\n7. **Self-Edit Formats Predefined.** We've added results with no predefined self-edit format, as well as a variety of other prompting formats (see our rebuttal to reviewer nQa9). Predefined prompting formats won out in our experiments, but SEAL improved performance by similar margins *no matter the prompt (or lack thereof)*.\n\n\n**Response to Questions:**\n\n1. **What Makes a Good Self-Edit.** We’ve noted that self-edits in knowledge-incorporation are longer. However, as we explain in the new results and response to reviewer nQa9, this does not account for all of the gains. Qualitatively, in the case of the “implications” prompt, good self-edits seem to contain a long but diverse list of short atomic statements that follow from the base text. We will also expand the appendix with additional qualitative examples and will provide a link to the repo containing all of the data for any future analysis.\n\n2. **Proxy Reward.** Note that RL would not be done *throughout* pretraining. We see it more as a warmup such that the model can then self-edit effectively (or produce synthetic data for a student model). However, we think that using a proxy reward models is a good idea! As with any RL setup, we could train a reward model and then use that instead, or we could try to use a faster-to-compute proxy metric for what is a good self-edit. To explore this idea, we've run another experiment using a rubric-based reward, where we query gpt-4.1 to score the length, diversity, quality, and correctness (whether it follows from the contexts) of the response, and use this as a reward rather than our current inner-loop. This is significantly faster than our original inner loop, but performance did not reach the level of the full meta-learning inner loop.\n\n## Additional Experiment: Proxy Reward\n\nWe experiment with replacing the inner loop with a proxy reward based on a human-crafted and tuned rubric with 4 categories: length, diversity, quality, and correctness. A gpt-4.1 grader model ranked each category on a 1-5 scale, and the sum of these scores was used as the RL reward. The final results are shown below, as well as the overall RL training times.\n\n| Model                |  Base | Post-RL | Time   |\n| :-------------------- | ----: | ------: | :------ |\n| SEAL                 | 32.0% |   **47.0%** | ~6 hr  |\n| SEAL w/ Proxy-Reward | 32.0% |   45.6% | **~5 min** |\n  \nWe believe further rubric/metric tuning could lead to a stronger reward signal. However, the benefit of the full SEAL loop is that this is not necessary, and we directly learn what edits are useful for the base model. We believe both approaches are promising when scaling to larger model sizes and greater compute budgets."}, {"review_id": "VkvUIk3jId", "reviewer": "Reviewer_oBBv", "summary": "This paper proposes SAEL, a novel algorithm for training LLMs to adapt themselves to particular tasks with RL. SEAL works by sampling outputs produced by the LLM which parameterize novel (synthetic) data that the LLM is subsequently finetuned on with LoRA. The LLM is trained with RL to output synthetic training data that improves its performance on the task at hand.\n\nSEAL is evaluated on a tool-use setting for ARC, as well as in a QA setting with natural language. In both cases SEAL works well. The authors also showcase example of the edits, and highlight some important limitations.", "questions": "* How do the gains from SAEL scale with the size/quality of the base LLM?\n* How does this technique compare to RL/reasoning methods? Simply comparing SEAL to a CoT approach without training would add value here.\n* Are there any qualitative differences between the self edits that improve Knowledge Incorporation and those that do not? From Figure 5 it seems that the LLM learns to output longer, more specific edits, but this seems pretty subtle. Would love to see some more detailed examples of successful vs unsuccessful edits.\n\n\nMinor remarks:\n* It's not obvious from the text that the base LLM used in the QA experiment is Qwen-2.5-7B. It's stated in the appendix, but should be clear in the main text.\n\nOverall I think the idea and algorithm is nice and makes a lot of sense. Unfortunately the evaluation of the algorithm is too superficial for me to recommend more than borderline accept. However, if the authors can address the questions above I am willing to increase my score.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The algorithm is novel and interesting.\n* SEAL also looks to work quite well in the domain tested.\n* Algorithm is fairly simple to understand.\n\n\n\nWeaknesses:\n* In SEAL, the algorithm generates data for itself, which means that if it's not very knowledgeable about a topic in the first place, it will be challenging to generate edits that improve performance. In this case, training on synthetic data from a frontier model like GPT4.1 seems to have advantages.\n* I think a slightly more extensive evaluation of the algorithm is necessary for publication. Particularly, I wonder how the performance scales with the quality of the base LLM used. Are stronger models also better at generating synthetic training data for themselves? I think rerunning the QA experiment with differently sized Qwen models could answer this question.\n* How does SEAL compare to Chain of Thought (CoT) reasoning? It seems plausible that CoT could pull out relevant pieces of information that the LLM knows and which could improve performance too when added to the context.", "was_revised": true, "final_justification": "The method introduced paper is intuitive and original. I had some doubts about the evaluation, particularly how SEAL compares to more conventional CoT prompting, which the authors have addressed in the rebuttal.", "rebuttal": "We thank the reviewer for their constructive feedback! We will address the proposed weaknesses and questions. \n\n**Response to Weaknesses:**\n\n1. **Smaller Models Might Not Have Enough Knowledge.** We agree that SEAL’s ability to generate useful self-edits is tied to the underlying capabilities of the base model. However, our results show that even small 7B models can substantially benefit, outperforming using data generated by GPT-4.1, despite starting with much lower baseline knowledge. We agree though that starting with a stronger base model would likely greatly improve the ability to self-edit, as scaling increases generation length and the ability to synthesize \"new\" information in context, given the original piece of data. \n\n2. **Scaling Model Size.** To further explore this, we conducted an additional scaling experiment across different Qwen model sizes (3B and 7B). We observed that the difference in performance between the base model and the SEAL model increased with the larger model, indicating that stronger base models are indeed better at generating synthetic data for self-adaptation, and that the benefits of RL increase as well. We will include these results in the revised paper and have provided them at the bottom of this response.\n\n3. **Chain-of-Thought.** We’ve run experiments incorporating CoT in two different ways. First, in producing a self-edit, we experiment with having the model write a CoT before the generation of synthetic data (see table under response to reviewer nQa9). We found little difference in performance in our document QA domain (39.7% vs 38.7% with base model), likely due to the fact that generating implications of our relatively simple documents require little in-context “reasoning.” In addition, we ran an experiment where given the questions during evaluation, we only do CoT with no training on the original document, to see if CoT can “pull out relevant pieces of information from the LLM,” as you suggested. We believe this intuition makes sense, but did not find any substantial difference in our setting (37.8%).\n\n**Response to Questions:** \n\n1. **Scaling Model Size.** See response to weakness 2.\n\n2. **Chain-of-Thought.** See response to weakness 3.\n\n3. **Qualitative Difference in Self-Edits.** We appreciate the suggestion to include more qualitative analysis of self-edits. Our initial inspection of edits did not reveal clear differences beyond length, specificity, and perhaps diversity (as noted in Figure 5), but we agree that more illustrative examples would be helpful. As we note in our response to reviewer nQa9, length clearly does not fully explain the difference. We will expand the appendix with additional qualitative examples and will provide a link to the repo containing all of the data. Furthermore, we show in our response to reviewer hhPg that a proxy reward based on these attributes can achieve good results with much less computation.\n\n**Response to Minor Remarks:**\n\n1. **Clarify Paper Text.** We will revise the paper to clearly state that the base LLM used in the QA experiments is Qwen2.5-7B.\n\n## Additional Experiment: Scaling Model Size\n\nWe rerun our experiments with the 3B-parameter Qwen variant. The results are given below.\n\n|      Model | Base Model (No training) | Base Model Self-Edit |   SEAL |\n| ---------: | -----------------: | ---------------: | --------: |\n| Qwen2.5-3B |              25.1% |            31.9% | **37.0%** |\n| Qwen2.5-7B |              32.7% |            39.7% | **47.0%** |\n \nWe can calculate the relative improvement of SEAL over the base model compared to using self-edits produced by the base model over the base model:\n\n$\\\\begin{align*}&\\\\text{3B: }\\\\frac{37.0−25.1}{31.9-25.1}=1.75\\\\\\\\ &\\\\text{7B: }\\\\frac{47.0−32.7}{39.7−32.7}=2.04.\\\\end{align*}$\n\nThe relative improvement is greater for the 7B model, which provides some evidence that not only are stronger base models more effective at leveraging synthetic data for self-adaptation, but reinforcement learning may have compounding benefits as model capacity increases. We acknowledge that it is hard to draw conclusions though without actually scaling up."}]

Author rebuttal:

Meta-review: Overview:

This paper proposes SEAL, a method for models to learn from self-generated data.  The proposed method incorporates model-generated self-edits, followed by evaluation on a target task, and a subequent model update based on target task performance.  Effectiveness of SEAL is demonstrated on two downstream tasks and outperforms training on other synthetic datasets such as that of GPT-4.


Strengths:

- The paper is very well-written.  The Figures 1-3 add a good amount of clarity to the method.  The method itself is also intuitive to understand.  

Weakness:

- Experiments are run mostly at smaller scales including 3B and 7B model sizes, and on smaller scale evaluations.  The experiments are also only limited to two tasks which are somewhat non-standard compared to other evaluations at this size such as Alpaca, MT-Bench, IF Eval, MMLU, etc.  

- Reviewers pointed out that there are some missing comparison with more SOTA work. As the paper primarily focuses on learning from self-generated data, the best comparison at the time was likely GPT-4 generated data as an upper bound on synthetic data generated for improvement.  While there were still some concerns about the comparison of the GPT generated data, the performance comparisons are reasonable and encouraging.

Main Reasons to Accept or Reject:

- Accept: A clear, well-written paper with a simple and intuitive method that performs well.  Authors have added several experiments expanding comparison.

- Reject: Experiments are only limited to two tasks, which are somewhat non-standard and limited in size (initially 200 but expanded in rebuttal).  The method is general but not applied in more settings.  The authors note it could be applied in math, code, etc. but do not give any particulars.  

Rebuttal Period:

All reviewers engaged in the rebuttal leading to increases scores where all reviewers are positive.  Reviewers note that most of their concerns have been addressed by the rebuttal and the authors have run quite a few additional experiments with another model and more baselines, and addressed some potential limitations such as run time.  However, the largest outstanding concern to me is the lack of multiple evaluation sets.  This seems like a general method that could be applied in more domains and on any generation benchmark as self-generating data for a target task has general use.

Final Recommendation:

The paper proposes a novel method for training language models on self-generated data. There are a few comments from reviewers and additional experiments that the authors should incorporate into future versions.
Decision: Accept (poster)