RL's Razor: Why Online Reinforcement Learning Forgets Less
Authors: Idan Shenfeld
Jyothish Pari
Pulkit Agrawal
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Large Language Models
Catastrophic Forgetting
Primary area: foundation or frontier models, including LLMs

Abstract: Comparison of fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT) reveals that, despite similar performance at a new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task, whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to a smaller KL change. We term this principle $\textit{RL’s Razor}$: among all ways to solve a new task, RL prefers those closest in KL to the original model.

RL'S RAZOR: WHY ONLINE REINFORCEMENT LEARNING FORGETS LESS
Idan Shenfeld
Jyothish Pari" Pulkit Agrawal
Improbable AI Lab, MIT (idanshen, jyop, golkitaggemit..dd
ABSTRACT
Comparison of fine-tuning models with reinforcement learning (RL) and super- vised fine-tuning (SFT) reveals that, despite similar performance new task, RL preserves prior knowledge and capabilities significantly better. We find that the degree of forgetting determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated the new task. Our analysis reveals that on-policy RL is implicitly biased towards KL-minimal solutions among the many that solve the new task. whereas SFT can converge to distributions arbitrarily far from the base model. We validate these findings through experiments with large language models and robotic foundation models and further provide theoretical justification for why on-policy RL updates lead to smaller KI change. We term this principle RL's Razor: among all ways to solve a new task, RL prefers those closest in KL to the original model.
KL(
KL!
Prior Task Performance
Policies with on besk
SFT
Low
Low
New Task Performance
Figure 1: Bias toward KL-minimal solutions reduces forgetting. Left: Among policies that solve the new task, RL converges to those closest in KL to the base model. Right: This KL bias yields higher prior-task retention at matched new-task performance compared to SFT.
INTRODUCTION
Foundation models have rapidly become the backbone of modern AI, powering applications in language, vision, robotics, and beyond. Despite their remarkable capabilities, today's models are largely static once deployed: they excel tasks learned during pre-training post-training, but not designed to self-improve and continually acquire new capabilities. We imagine a future where deployed models are long-lived agents assisting humans in the long-term and continuously adapting to new needs. As such, models must improve and adapt to new data, environments, and objectives al. 2025b); Simonds & Yoshiyamal Gao et al. (2025) Dao Le (2025): Moradi al. 2025 (2025) Zweiger et al. (2025)
'Equal contribution. Correspondence to: Idan Shenfeld (idanshen@mi t. edu)) Jyothish Pari (jyopémi edu).

---

A central challenge to this vision is catastrophic forgetting the tendency for models to lose previ- ously acquired capabilities when trained on new tasks McCloskey & Cohen 1989); French| [1999;;
Kirkpatrick et al. (2017); Luo et al. (2023). Although scaling model size and pre-training data : Luo et al. ): Cossu et al. (202)) catastrophic forgetting remains a persistent obstacle, undermining the promise of continual improvement [Bom-]
improves robustness Ramasesh et al. (2021)
imasani (202I): Guo et al. (2025b); Zweiger et al. (2025). To enable foundation models to serve as long-term agents, we need develop post-training methods that allow models to acquire new skills without erasing old ones.
To further this goal, we analyze the performance of two widely used post-training schemes of su- pervised fine-tuning (SFT) and reinforcement learning (RL). Our experiments reveal a surprising finding: even when SFT and RI. achieve the same performance on the new task, we observe that SFT often achieves new-task gains by erasing prior knowledge, while RL. better preserves old skills. Figure | (right) illustrates this tradeoff: although both methods can reach high performance on the new task, RL maintains substantially higher performance on prior tasks compared to SFT.
This striking empirical gap raises the question: what underlying mechanism allows RL to improve on new tasks, but unlike SFT, minimally impacts the model's prior knowledge?
Previous approaches to catastrophic forgetting targeted specific factors such as constraining weight updates (Kirkpatrick et al.) 2017 Aljunni et al. | Zenke et al. 2017). preserving learned features
(Rannen et 2201 || et al. 2019). or regularizing shift in output distribution (Li & Hoiem| [017] Stiennon et al.] (020). While these methods reduce forgetting, they focus its effects rather than its underlying cause. Consequently, it remains unclear what truly governs forgetting or why different training algorithms behave so differently. Some prior work claimed that forgetting can be
determined by how much the model's distribution shifts on past tasks (Rebuffi et al. 2017 Castro al. 2018 Chaudhry et al. 2018 Wu et al., 2019). Yet in practice, this is infeasible to measure in foundation models, where the set of prior tasks is vast or even unbounded. To search for a more useful principle, we systematically ablated many candidate variables. Surprisingly. we find that forgetting can instead be predicted using only the new task distribution. Specifically, we uncover an empirical forgetting law: When fine-tuning a model π on a new task т, the degree of forgetting is accurately predicted by Ex-r [KLTTTl|m) [T]] the KL divergence between the fine-tuned and base policy evaluated on the new task. This law is practically useful since it can be measured, and even influenced, during fine-tuning, without requiring access to past-task data. Although the mechanism remains to be fully understood, the consistency of this law across models and domains suggests it reflects fundamental property of forgetting.
This law also clarifies the surprising difference between SFT and RL. Our analysis reveals a simple but powerful principle we call RL's Razor: among the many high-reward solutions for a new task, on-policy methods such as RL are inherently biased toward solutions that remain closer to the original policy in KL divergence. Figure [ (left) highlights this effect: among the many policies that reach a high success rate on the new task, RL is biased toward KL-minimal solutions, while SFT can converge to distant ones. This bias arises directly from RL's on-policy training: by sampling from the model's own distribution at every step, RL constrains learning to outputs already given non-negligible probability by the base model. To improve reward, these samples are reweighted and used to update the model, which gradually shifts the policy rather than pulling it toward an arbitrary distribution. Thus, when multiple equally good solutions exist for a new task, RL tends to find solutions close to the original policy, while SFT can converge to solutions much farther away, depending on the provided labels. Theoretical analysis in a simplified setting confirms this view, showing that policy gradient methods converge to KL-minimal solutions even without explicit regularization.
Finally, to validate the KL hypothesis, we construct "oracle SFT" distribution that provably mini- mizes KI. divergence while achieving perfect accuracy. Training on this oracle distribution produces even less forgetting than RL itself. This demonstrates that RL's advantage does not stem from be- ing inherently different, but from its implicit KL minimization. Whenever training is biased toward KL-minimal solutions, forgetting is reduced.
Our main contributions are:
We show that RL fine-tuning forgets less than SFT, even when both reach the same performance on new tasks.
2

---

We uncover an empirical forgetting law: the KL divergence the base policy, measured on the new task, as a strong predictor of catastrophic forgetting across objectives and hyperparameters.
We provide empirical and theoretical evidence that the on-policy nature of policy gradient methods leads to smaller KI. shifts and explains RL's advantage.
Together, these findings suggest new perspective on post-training: to achieve continual adapta- tion without forgetting, algorithms should explicitly aim minimize KL divergence from the base model. This principle opens the door to designing future training methods that combine RL's ability to preserve prior knowledge with the efficiency of SFT, enabling foundation models that can truly learn for life.
RELATED WORK
Foundation Models and Post-training In modern deep learning, large-scale models pre-trained on broad, diverse datasets (usually termed Foundation models) serve general-purpor back- bones (Radford et al. [021]] |Achiam et al. [023] Touvron et al., 2023 Hu et al. 2023] Li et al. 2024a) with broad domain knowledge and some zero-shot learning abilities (Radford et al. 2018 Brown et al.) 2020). However, pre-trainn pre-trained models may not directly meet the requirements of specific applications align with lomain-specifer constraints. Post-training methods address this gap by adapting foundation models to downstream tasks through supervised fine-tuning on curated datasets (Howard & Ruder 2018 Dodge et al., 2020 Wei et al. [202]] Chung et al.] 2024). reinforcement learning from human or automated feedback (Ziegler et al., 2019 Ouyang et al. [2022] Guo et al.)
2025a Zhai et al. 2024), and other techniques (Ratailov et al. 2023). In this work, we study how different post- -raintrain methods affect forgetting, focusing on supervised fine-tuning and reinforce-
ment learning.
Catastrophic Forgetting. While fine-tuning primarily aims to improve performance on a new specific task, preserving the model's pre-existing general capabilities is equally critical. Unfortu- nately, fine-tuning often leads to catastrophic forgetting-4 phenomenon where learning new in- formation significantly deteriorates previously acquired knowledge McCloskey & Cohen (1989); French ( (1999): |Kirkpatrick et al.] (2017): (Ouyng et al. (2022); [uo et al. (2023). Many works have sought to reduce forgetting by constraining updates, for example, by penalizing the magnitude of change in the model parameters, features, or matching the output on previous tasks/datasets (Wang al. 2024). These methods are effective heuristics, but they address the symptoms of forgetting rather than explaining its cause. Our aim is to identify simple and predictive metric that explains when and why forgetting occurs across different training algorithms.
We do not introduce new training algorithm, but instead identify a simple empirical forgetting law: the KI. divergence between the fine-tuned and base policy. measured on the new task, reliably predicts the degree of forgetting. The law also sheds light why some mitigation strategies work. For example, methods like Elastic Weight Consolidation (Kirkpatrick et al. (017)) can be seen as approximations to KL minimization (Chaudhry et al.) 2018). Interestingly, practitioners have also observed that KL regularization used in RL tuning of LLMs as a heuristic for stabilizing opti- mization or preventing reward hacking Stiennon al. (2020); Gao al. (2023), also helps reduce catastrophic forgetting (Ouyang et al. 2022). Our contribution is to show that KL divergence is not merely a useful heuristic, but reliable predictor of forgetting across settings.
SFT versus RL. Prior comparisons between SFT and RL have focused on new task performance. A seminal result in sequential decision making is that on-policy learning can achieve stronger per- formance even when the expert providing supervision is the same one used to generate the offline dataset (Ross et al.) [201]). Recent empirical studies have also found that RL fine-tuned models often exhibit superior generalization beyond the training distribution Han et al. (2025); CCu et al. (2025); [Chu
Li al. (2025a) and transfer more effectively to related tasks Huan et al. (2025) compared to SFT. However, prior works haven't 't examined the relative susceptibility of RL and SFT to catastrophic forgetting, which is the focus of our study.
Concurrently, Lai et al. (025) reports that RL forgets less than SFT, but ascribes RL's advantage to learning from negative examples and not to the on-policy nature of RL. Results in Section
3

---

LLM, Math
LLM, Science Q&A
0.57
0.57
ON FraNdeus
0.98
0.55
Scate
RL
Aeg.
0.53
RL
0.40 0.50
0.52
0.54
0.55
0:58
0.80
0,62
03
0.4
0.5
0.6
07
New Task Ancuracy
New Tack Anouracy
Tool Use
Robotics, Pick and Place
W0 Score on Prezious Tasto
0.42
0.56
0.58
0.54
0:58
0.53
0.36
R.
SFT
0.40
0.50
0.55 0.60
0.85
0.70
0.3
0.4
0.5
0.8
New Tesk Acourasy
New Task Accuracy
Figure 2: Pareto frontiers of RL and SFT. Comparing the performance of fine-tuned model on the new task (x-axis) and prior task (y-axis). Each point corresponds a model trained with different set of hyperparameters, and the curves trace the Pareto frontiers for the two methods. RL achieves new-task improvements while maintaining prior knowledge, whereas SFT improves new- task performance at the expense of forgetting the prior task.
contradict their explanation of why RL forgets less, showing that the on-policy nature of RL is key. We also contribute the empirical forgetting law, the RL Razor, and its theoretical justification.
3 REINFORCEMENT LEARNING FORGETS LESS THAN SFT
We report results comparing the degree of catastrophic forgetting against new-task performance induced by RL and SFT on various large language model (LLM) and simulated robotic tasks.
3.1 PERFORMANCE TRADE-OFFS
Experimental Setup. For each new task, we fine-tuned models using the same of prompts. One group of models was trained with SFT, and another with RL using GRPO Shao et al. (2024). Evaluation was performed along two axes:
New task Performance: We measured performance the held-out test set of the newly introduced task to assess the performance gain from the training.
Previous tasks Performance: We measured performance on a diverse set of unrelated benchmarks. A drop in these benchmarks taken as measure of catastrophic forgetting.
Importantly, our RI. implementation uses GRPO with single gradient steps (effectively REINFORCE with normalized rewards), no KI. penalty, and no clipping.
Since different hyperparameters can lead to varying trade-offs between learning and forgetting, we trained dozens models under diverse hyperparameter settings for both SFT and RL. To compare methods fairly, we identify the Pareto frontier in the two-dimensional plane of new-task performance versus previous-task performance. The Pareto frontier represents the set models for which no fur- ther improvement on the new task is possible without incurring greater forgetting. Figure 2 (right) reports these frontiers: each point corresponds to a trained model with a different set of hyperpa- rameters, and the Pareto-frontier curve indicates the best achievable trade-off for each method.
4

---

20
70
52
S20 Prettvu Teak
40
20
Scote
88
90
52
26
36
15
SFT dist.
SFT on dist.
SFT on optimal dist.
KL. POg.
Figure 3: KL divergence predicts catastrophic forgetting. (Left) Gearring-Forgettinnn Trade-offs. SFT outperform RL only when an oracle distribution used as a source annotation. (Middle) Forgetting aligns to single curve when plotted against KL divergence, showing KL strong predictor across methods. (Right) RL improves new-task accuracy with much smaller KL shifts than SFT, highlighting the conservativeness of on-policy updates.
Tasks and Datasets.
We perform experiments across three LLM and a single robotic tasks:
LLM, Math reasoning: Qwen 2.5 3B- IB-Inutr (Qwen et al. 2025 trained on math questions from the Open-Reasoner--Zerr dataset (Hu al. 2025).
LLM, Science Q&A: Qwen 2.5 IB-Instruct trained on Chemistry L-3 subset of SciKnowEyal (Feng
et al. 2024).
I.M, Tool use: Qwen 2.5 3B-Instruct trained on ToolAlpaca dataset (Tang et al. 2023
Robotics, Pick and Place: Open VLA 7B (Kim et 2024) trained in the SimplerEnv environment (Li et 2024b) on the task of picking up a can.
To measure forgetting, evaluated the finetuned models established benchmarks covering di- verse prior capabilities. For LLMs, we used Hellaswag (Zellers et al.) 2019), TruthfulQA (Lin et al, 2021, MMLU (Hendrycks al. 2020), IFEall (Zhou al. 2023). Winogrande (Sakaguchi al. 2021. and HumanEval (Chen et al. 2021). For robotic policies, we evaluated on the open/close drawer SimplerEnv tasks, excluding the one used for fine-tuning. These benchmarks act as prox- ies for prior skills that should be preserved during adaptation. Full details on SFT data sources, hyperparameters, and training/evaluation protocols are provided in Appendix C
Results. Figure Sigureerrporee reports the trade-off between new-task performance and retention of prior abili- ties. For RL, as accuracy on the new task increases, performance on previous benchmarks remains nearly unchanged.
contrast, SFT improvements on the new task consistently come at the cost of substantial forgetting. This difference is most pronounced in Math, where even small gains on the fine-tuned task correspond sharp reduction in prior-task performance. In Science Q&A and Tool Use, SFT retains some ability on prior tasks at lower accuracy levels for the new task, but performance deteriorates rapidly the model approaches higher accuracy on the new task.
Takeaway 1
RL is able to learn tasks while incurring minimal forgetting, whereas SFT reaches similar new-task performance only by sacrificing prior knowledge.
4 SMALLER KL DIVERGENCES LEAD TO LESS FORGETTING
As shown in Section RL fine-tuning achieves comparable new-task performance to SET while consistently forgetting less. Explaining this gap requires identifying a variable that determines the degree of forgetting across methods. We therefore searched for predictor that could account for forgetting independently of the training algorithm or hyperparameters. Such a predictor would both explain the empirical difference between RL and SFT and offer unifying principle for catastrophic
5

---

forgetting. Prior work has proposed candidates such as the magnitude of weight changes, sparsity of updates, or gradient rank. Across our experiments, however, none of these variables consistently empirical forgetting law: the KI. divergence between the fine-tuned model and the hase model, measured the new task, reliably predicts the degree of forgetting.
aligned with the observed forgetting behavior (see Section (6). What did emerge
seems more natural to measure KL on prior tasks, and not on new tasks, and indeed that was the focus of multiple prior works (Li & Hoiem 2017, Kirkpatrick et al.) (017). However, since foun- dation models undergo many training stages, old-task distributions are ill-defined or inaccessible. Moreover, we' found that prior task KL and new task KL are highly correlated, as can be seen in Figure 13
Testing this hypothesis large LLMs is challenging, since RL training computationally expensive and cannot easily be run to convergence. Moreover, the search for predictors requires repeating fine- tuning many times under diverse conditions. To address these limitations, we designed a controlled toy setting, ParityMNIST, that allows us to replicate the RL-SFT gap under full convergence and perform systematic ablations.
Parit is derived from MNIST (Deng 2012), but reframes the task as predicting parity (even vs. odd). An image of an even digit is correctly classified if the model predicts any even digit label, and likewise for odd digits. Multiple output distributions are thus equally valid, mirroring a key many distinct policies can achieve the same property of the generative tasks we studied in section 3 performance.
We pretrained a 3-layer MLP jointly on subset of ParityMNIST and FasHionMNIST (Xiao et al, 2017). then fine-tuned only on ParityMNIST while measuring forgetting FashionMNIST.. This design provides a minimal, tractable setting for investigating predictors forgetting. To parallel the main experiments:
the SFT setting, the model was trained on labels sampled from a single arbitrary distribution out of the many possible correct ones.
In the RL setting, the reward was correctness with respect parity, leaving the model free to converge to any valid distribution.
For more details, see Appendix C..3 This design allowed us replicate the phenomenon where RI. reached high accuracy on the new task with substantially slower degradation of prior knowledge, while SFT exhibited steeper trade-off (Figure 3 left). Importantly, reproducing the effect in this simple MLP setting shows that is not specific to large scale transformers, but a more general property of fine-tuiing deep generative models.
KL Predictor. Plotting forgetting against the KL divergence from the base model ParityM- NIST reveals a single functional relationship across both RL and SFT (Figure 3 middle). This indicates that forgetting is determined by KL divergence, not by the choice of training algorithm. A quadratic achieves R² = 0.96 in this setting, underscoring the strength of the relationship. To test robustness, we repeated the experiment with two different arbitrary SFT labelings. Although their Pareto frontiers differed, the forgetting-KL. curves coincided, confirming that KL consistently pre- dicts forgetting irrespective of training method label distribution. The same correlation appears in our LLM experiments, with quadratic fit achieving R² = 0.71 (Figure 12). While weaker, the residuals are mean-zero and can be attributed noise from approximate KI. and accuracy estima- tion.
Optimal SFT Distribution. To validate that KL divergence is the predictor variable, we con- structed an oracle SFT distribution. In ParityMNIST, the simplicity of the task allows us to analyt- ically identify the labeling that minimizes KI. divergence to the base model among all distributions achieving 100% accuracy (Appendix C.3p. KL divergence fully determines forgetting, then train- ing SFT on this oracle distribution should yield the optimal forgetiy trade-off. The results in Figure 3 confirm this prediction SFT trained on the oracle distribution retained more prior knowledge than RL, achieving the best trade-off observed. RL performs well because its on- policy updates bias the solution toward low-KL regions, but when SFT is explicitly guided to the KI-minimal distribution, it can surpass RL. As additional validation, trained SFT model

---

Pos + Neg Examples
Pos Examples
On Policy
0.70
1-0 Reinforce
GRPO
0.6
0.07
0.55
0.33
0.95
0.50
C.45
Offlie
SIMPO
c32
0.01
Figure 4: Comparison of algorithm classes. (Left) The four quadrants illustrate algorithm types, defined by whether they are on-policy or offline and whether they incorporate negative gradients. (Middle) On-policy methods retain prior knowledge more effectively. (Right) Both GRPO and 1-0 Reinforce achieve higher new-task accuracy while incurring smaller KL shifts from the base model, showing that on on-policy methods consistently induce more conservative KL updates.
on data generated by an RL-trained model. The distilled SFT matched RL's forgracyg trade-off (Figure [0), reinforcing that the distribution Icarned, rather than the optimization algo- rithm, governs forgetting. Finally, we also added KL regularization to SFT, and found that it only minimally improves the loraeng Pareto frontier (Appendix(A)
Takeaway 2
Catastrophic forgetting in both SFT and RL is predicted by the KL divergence between the fine-tuned and base models on the new task.
ON-POLICY METHODS LEADS TO SMALLER KL DIVERGENCE
Having established that the KL divergence between the trained model and its base distribution on new task predicts catastrophic forgetting, we now ask: why are RL fine-tuned models able to achieve strong task performance while moving less in KL than SFT models?
5.1 EXPERIMENTAL EVIDENCE
To understand the difference in KL behavior, it is useful to contrast the training objectives of SFT and RI. For discrete outputs, SFT minimizes cross-entropy against a supervision distribution distribution of inputs D:
(T) -EEwD [log (1)
In contrast, RL with policy gradients optimize
( Eyy-D.ye (a, y) log π (y]a:)]
where A(a, y) is an Advantage function, which is the reward of y normalized with respect to other rewards for the same Two features distinguish this from SFT:
1. Sampling Distribution. While in RL the training was done on outputs drawn from the model's own distribution, in SFT they come from fixed external annotations.
Negative Examples. While sampling from ग, some the responses will be incorrect. These are usually assigned negative coefficient A(x,y). This pushes probability mass away from poor outputs, a mechanism absent SFT.
'Notice that in practice, the policy gradient trick (Sutton et al. 1998 ensures gradients are taken only through the log-probability term, not through the sampling distribution inside the expectation.

---

Our hypothesis is that one of these two differences is what causes RL's resistance to forgetting. To examine our hypothesis, we perform experiments with four different objectives:
GRPO, An on-policy objective that utilizes negative examples. Here, A(z, y) is the normalized reward.
1-0 Reinforce. An on-policy algorithm that does not use negative examples. Here, A(x, for correct responses and 0 for incorrect ones. This is equivalent to sampling from the model and performing SFT on correct answers only.
SFT. An offline objective that does not use negative examples.
SimPO. An offline objective that utilizes negative examples. We create negative examples by sampling incorrect responses from external model, and use the SFT data for positive examples. The SimPO (Meng et al. 2024) loss compares correct and incorrect outputs via a logistic term:
LSIMPO ( - -E D.nn--yyurrppp. -
[log (log ग (Mwe (z) - log T (u (x) 1)]
where aT and g denote distributions for correct and incorrect responses, respectively. We used SimPO rather than naive iikelihood/negativee likelihood because the latter was unstable to train.
We compared the four objectives on the Science Q&A task, measuring their learning-foretttnng trade-offs as in Section The results, shown Figure reveal that 1-0 Reinforce behaves similarly to GRPO, while SimPO resembles SFT. Thus, the critical factor is not the presence of negative gradients but the use of on-policy data. Plotting KL divergence confirms this conclusion: on on-policy methods (GRPO and 1-0 Reinforce) reach the same task performance with significantly smaller KI. divergence from the base model than offline methods (SFT and SimPO).
5.2 THEORETICAL PERSPECTIVE
Beyond the empirical results, it is useful to ask why on- policy methods naturally induce smaller KL shifts. One
Policy (YY
R=1
way to see this is through the lens of projection in prob- ability space: policy gradient methods can be understood conservative projection that keeps the policy close to its starting point while reweighting toward higher-reward outcomes. At each step, the policy samples outputs it al- ready finds likely, then re-weights those samples accord- ing to reward, shifting probability mass toward higher- reward outcomes while suppressing lower-reward ones. Crucially, because updates are defined relative to the model's 's own distribution, they nudge the policy toward nearby re-weighted distribution, rather than pulling it toward potentially distant external distribution (as in SFT). This explains why policy gradient methods tend to remain close to the base model in KL divergence.
Figure 5: KL-minimal path to opti- mality. Alternating I-projection into the set of optimal policies and M M-projection into IT carries To into p while prefer- ring the closest solution in KI.
This perspective can be formalized by observing that, in the binary-reward case, the re-weighted distribution targeted by policy gradient is exactly the minimum- KL projection of the current policy onto the set of optimal ones.
Lemma 5.1. Let p be a distribution over a finite set Y. and let R Y - {0.1} be a reward function. Rejection sampling from p with acceptance condition R(y) yields distribution qus. This distribution can be equivalently characterized as the solution to:
QRS
arg min (xl(p) s.t. [[()) =1
Building this, we show that policy gradient converges the KL-minimal optimal policy within the representable family. A detailed version with proofs is provided in Appendix
Theorem 5.2. Let be a finite set and let II C A(Y) be a convex family of feasible policies (e.g., an exponential family). Let R: Y - {0, be binary reward function and P = {g :yy [R]
8

---

the set of optimal policies. Then, under suitable regularity conditions, solving the reinforcement learning objective with policy gradient converges to
π arg
min ( mo); xEP*NIl
where mo is the initialization. In other words, policy gradient selects, among all optimal repre- sentable policies, the one closest in -l.diivegeence to the starting policy.
Our theoretical analysis uses simplified convex setting to isolate the on-policy effect. Extending our analysis to fully characterize deep network dynamics remains important future work.
Takeaway 3
On-policy training explains why RL maintains smaller KI. divergence than SFT. Sampling from the model's own distribution keeps it close to the base model, while SFT pushes it toward arbitrary external distributions.
6 ALTERNATIVE HYPOTHESIS
Science advances not only by identifying the right explanations, but also by eliminating incorrect ones. To this end, systematically evaluated alternative variables potential predictors of catas- trophic forgetting, grouped into four categories:
Weight-level changes. Many prior work tried to mitigate forgetting by constraining the change in parameter space (Kirkpatrick et al., 2017; Alluund et al.) 2018 Zenke et al.) (017). We mea- sured parameter changes under L1. Fisher- weighted L2, and spectral norm metrics. The Fisher matrix was computed on the basis of the model parameters, with expectation over inputs from the previous task. These metrics correlated only weakly with forgetting: large parameter shifts could occur without forgetting, and conversely, forgetting sometimes occurred despite small parameter movement.
Representation-level changes. Some other papers focused on maintaining the previous features (Jung et al., 2018 Hou et al.) [2019] Dhar et al.) 2019). We examined hidden activation shifts (LI and L2 distances) as proxies for changes internal representations. Although we found that there is representation drift during training (see Appendix D.I. the curves were distinct between training objectives, meaning that it not good predictor.
Sparsity and rank of updates. Motivated by Mukherjee al. (025). who argue that RL updates are sparse while SFT weight updates are dense, we explicitly tested this hypothesis. We found that the reason for the observed sparsity was the of oatl for model training. Since bfloatle 6 has a limited mantissa, small parameter updates (such as those produced by RL) can fail to cross the representational threshold, effectively causing no update at all. Performing the same training with float32 resulted in models with identical performance but without any sparsity in their weight updates. The rank of all weight updates was full.
Distributional distances. We considered multiple measures of output distribution change, all measured over inputs from the new task т: Forward KL. (En=, [L(To]]))]) Reverse KL. (Eper
KL π [T)))) Total Variation, and Lg distance between distributions.
Table summarizes these results for the MNIST task. Across candidates, KL divergence (both forward and reverse) between the fine-tuned and base model evaluated on the new task emerges as the only consistent and high-fidelity predictor catastrophic forgetting.
DISCUSSION AND CONCLUSION
Our study reveals that catastrophic forgetting is governed not by the choice of training algorithm, but by the KL divergence from the base policy evaluated on the new task. This explains why RL forgets less than SFT, as on-policy training naturally biases updates toward KL-minimal solutions, preserving prior knowledge while acquiring new skills.
However, still lack mechanistic account of why larger KL shifts the new task disrupt prior knowledge- whether whhthhe through representational interference, implicit capacity limits, or other dy- namics. Moreover, while demonstrate the forgforgggnngg link across moderate-scale LLMss and

---

Variable
R (2nd deg. polynomial)
KL, forward
0.96 0.01
KL, reverse
0.93
TV
0.80 + 0.01
Distribution change, L.
0.56 0.02
Weight change. L1
0.34 + 0.02
Weight change, Fisher Weighted L2
0.58 + 0.02
Weight change, spectral norm Sparsity of weight change
0.58 + 0.02
N/A
Rank of weight change
N/A
Activation change, L.1 Activation change, 1.2
0.52 0.02
0.55 + 0.02
Table 1: Predictive power of alternative variables compared to KL.
toy models, its behavior at frontier scales and in more diverse generative domains remains unknown. In addition, we didn't 't study online but off-policy algorithms, which are popular in RL. Addressing these gaps will be essential for grounding the principle and extending
real-world deployment.
Taken together, our results motivate a new design axis for post-training research: algorithms should be judged not only by how well they optimize new tasks, but also by how conservatively they move in KL relative to the base model. Importantly, this does not mean offline data cannot help, but that continual learning requires updates to keep learning close to the KL-minimal path, Embracing this principle may allow us to build agents that not only learn new skills, but also truly learn for life.
USE OF LANGUAGE MODELS
The authors used large language models polish and revise the writing of the manuscript. The models were not used to generate ideas, perform analysis, or produce original scientific content.
ACKNOWLEDGMENT
We want express gratitude Nitish Dashora, Seungwook Han, Moritz Reuss, Zhang-Wo Wei Hong, Leshem Choshen, Ahmad Beirami, Mchul Damani, Akarsh Kumar, and members of the Im- probable AI lab for the helpful discussion the paper. We are grateful MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources. The research was sup- ported in part by Hyundai Motor Company, Qualcomm Innovation Fellowship. Google, and Ama- zon. The research was sponsored by the Army Research Office and was accomplished under Grant Number W91 1NF--1-11-000 was accomplished under Grant Number N00014-22 2114-2-----740 1-2740. Research was also sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Coop- erative Agreement Number FA8750-19 19-2-199---000. The views and conclusions contained in this docu- ment are those of the authors and should not be interpreted as representing the official policies, either expressed implied, of the Department of the Air Force the U.S. Government. The U.S. Gov- ermment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office. Naval Research Office, Air Force, the U.S. Government.
The research was also sponsored by the Office of Naval Research and
AUTHOR CONTRIBUTIONS
Jyothish Pari co-developed the project and contributed in all aspects of experiments and writing. Idan Shenfeld co-developed the project and contributed in all aspects of experiments and writing. Pulkit Agrawal co-developed the project direction, advised IS and JP, and played significant role in paper writing.
10

---

REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 08774, 2023.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. Proceedings of the European confer- ence on computer vision (ECCV), pp. 139-154, 2018.
Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American Mathematical Soc., 2000.
Rishi Bommasani. On the opportunities and risks of foundation models. arXiv:2108.00 0725588. 2021.
arXiv preprint
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arKiv:2212.068) 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Francisco M Castro, Manuel Marín-Jiménce Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference computer vision (ECCV), pp. 532-547, 2018.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint 17Xiv:2501.17661, 2025.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnstruction-finetune lan- guage models. Journal of Machine Learning Research, 25(70): 2024.
Andrea Cossu, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision. Neural Networks, 179: 179410 2024.
Imre Csiszár. Information geometry and alternating minimization procedures. Statistics and Deci- sions, Dedewicz, 1:205-237, 1984.
Alan Dao and Thinh Le. Rezero: Enhancing Ilm search ability by trying one-more-time. arXiv preprint arXiv:2504.1100) 2025.
Arthur P Dempster, Nan M Laird, and Donald Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1): 1-22, 1977.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6): 141-142, 2012.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5138-5146, 2019.
11

---

Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint ar 1.Xv::20020063005. 2020.
Kehua Feng, Keyan Ding. Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen. Sciknoweval: Evaluating multi-level scientific knowledge of large language models. arXiv preprini arXiv:2406.09098, 2024.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Huan-ang Gao, Jiayi Geng, Wenyue Hua, Mengkang Hu, Xinzhe Juan, Hongzhang Liu, Shilong Liu, Jiahao Qiu, Xuan Qi, Yiran Wu, et al. A survey of self-evolving agents: On path to artificial super intelligence. arXiv preprint arXiv:2507.21046 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835 10866. PMLR, 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
Asela Gunawardana, William Byrne, and Michael Jordan. Convergence theorems for generalized alternating minimization procedures. Journal of machine learning research, 6(12), 2005.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025a.
Haiyang Guo, Fanhu Zeng, Fei Zhu, Jiayi Wang, Xukai Wang, Jingang Zhou, Hongbo Zhao, Wen- zhuo Liu, Shijie Ma, Da-Han Wang, et al. A comprehensive survey on continual learning in generative models. arXiv preprint arXiv:2506. 13045, 2025b.
Seungwook Han, Jyothish Pari, Samuel J Gershman, and Pulkit Agrawal. General reasoning requires learning to reason from the get-go. arXiv preprint 250. 2025.
Dan Hendrycks, Collin Burns Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 03000, 2020.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilci Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 831-839, 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint /80/.06 00080.000666...... 2018
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner-zzeo:: An open source approach to scaling up reinforcement learning the base model. arXiv preprint arXiv:2503.24296 2025.
Yafei Hu, Quanting Xie, Vidhi Jain, Jonathan Francis, Jay Patrikar, Nikhil Keetha, Seungchan Kim, Yaqi Xie, Tianyi Zhang, Hao-Shu Fang. et al. Toward general-purper robots via foundation models: A survey and mnta-sna arXiv preprint arXiv:2312.08878 2023.
Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general Ilm capabilities? understanding transferability of Il reasoning. arXiv preprint arXiv:2507.00432 2025.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987, 2024.
12

---

Heechul Jung, Jeongwoo Ju, Minju Jung, and Junmo Kim. Less-forgetful learning for domain pansion in deep neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint 1rXiv:2406.092445 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-33 ---226,,, 2017.
Tomasz Korbak, Ethan Perez, and Christopher L Buckley, RI with kl penalties is better viewed as bayesian inference. arXiv preprint (xXi:2205.112777 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMIR, 2019.
Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi, Min Xie, Qingfu Zhang, al. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. arXiv preprint arXiv:2507.05386 2025.
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang. Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general purppose assistants. Foundations and Trends@ in Computer Graphics and Vision, (6(1-2):1-211 2024a.
Tianle Li, Jihai Zhang, Yongming Rao, and Yu Cheng. Unveiling the compositional ability gap in vision-language reasoning model. arXiv preprint arXiv:2505.19406 2025a.
Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sich, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies simulation. arXiv preprint ar www.x::405.099/4. 2024b.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Iransactions on patlern analysis and machine intelligence, 40(12):2935-2947) 2017.
Zhongyang Li, Ziyue Li, and Tianyi Zhou. C3po: Critical-layer, core-expert, collaborative pathway
optimization for test-time expert me-mixing. ArXiv, 0bs/2504.07964 2025b. URL https::/
api semant www.hcccchhaaaccccco www.aaani.cccoll.....ooooooppapa
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07955. 2021.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.2078. 2025.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint N3Xi:2300.08844 2023.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-Fir reward. Advances in Neural Information Processing Systems, 37:124198-121 -224235, 2024.
13

---

Mohammad Mahdi Moradi, Hossam Amer, Sudhir Mudur, Weiwei Zhang, Yang Liu, and Walid Ahmed. Continuous self-improvement of large language models by test-time training with verifier-driven sample selection. ArXiv, Abs/2505.19475 2025. URL https://api.
semant: 1 wwwhhcacc.rcooooo Cotp.ooor//oorpusii::3300000 pee.aaattt.sssso....rr.rrrrrrorr
Sagnik Mukherjee. Lifan Yuan, Dilek Hakkani-Tur, and Hao Peng. Reinforcement Jearmin finetunes small subnetworks in large language models. arXiv preprint arXiv:2505.11711 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL https: /ttpaaaxvvv...o .rgg abs/ 22//2241 2412. 15115
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. arXiv preprint arXiv:2303.08774, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763.8773..08 PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in neural information processing systems, 36:53728 53741, 2023.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International conference on learning representations, 2021.
Amal Rannen, Rahaf Aljundi, Matthew B Blaachko, and Tinne Tuytelaars. Encoder based lifelong learning. Proceedings of the IEEE international conference on computer vision, pp. 1320- 1328, 2017.
Sylvestre-AIie Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lamperr. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference on artificial intelligence and statistics, pp. 627 635. JMLR Workshop and Conference Proceedings, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.033000 2024.
Toby Simonds and Akira Yoshiyama. Ladder: Self-improving Ilms through recursive problem de- composition. arXiv preprint wrXiv:2503.00733 2025.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
14

---

Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2123302..3777 13971, 2023.
Nino Vieillard, Tadashi Kozuno, Bruno Schemer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: an analysis of kl regularization reinforcement learning. Advances in Neural Information Processing Systems, 33:12163- 2020.
Liyuan Wang. Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEEE transactions on pattern analysis and machine intelligence, (188):536-5388. 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
CF Jeff Wu. On the convergence properties of the em algorithm. The Annals of statistics, pp- 95-103, 1983.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 374-382, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 2017.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv: 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al. Fine-tuning large vision-language models as decision-makin agents via reinforcement learning. Advances in neural information processing systems, 17:110955-119911, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followingg evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08535 2019.
Adam Zweiger, Jyothish Pari, Han Guo, Ekin Akyürek, Yoon Kim, and Pulkit Agrawal. Self-adapting language models. ArXiv, abs/2506. 10943, 2025. URL https://api. ssemant cscho ar. www.aaooroooorrrrao.......
15

---

A THE EFFECT OF KL REGULARIZATION
In main experiments, did not employ explicit KL regularization. Nevertheless, finding that forgetting is closely predicted by the KL divergence to the base model naturally raises the question: can directly regularizing KL divergence mitigate forgetting? This is especially relevant given that KL penalties are widely used in reinforcement learning fine-tuning of large language models (Stiennon et al. 2020 Ouyang et al. 2022 Vieillard et al. 2020).
Empirical observations. We revisited the ParityM- NIST setup from Section 4 this time adding explicit KL penalties to both SFT and RL training. For each method, conducted hyperparameter sweep and varied the reg- ularization coefficient over 0.1, 0.2, 0.5. Figure reports the resulting Pareto frontiers of the forrenngnorgetting trade-off. The effect is strikingly asymmetric:
70
Avg. Score on Previous Tasks
60-
40
In RL. KL regularization substantially improves the trade-off. By explicitly discouraging large deviations from the base model, amplifies RL's inherent bias to- ward KL-minimal solutions, enabling gains on the new task while preserving performance on prior tasks.
RL
30-
SFT
SFT
KL reg.
Accuracy
In SFT, KL regularization has only marginal effect. While slightly restrains the model from drifting too far, the op- timization remains tied to external supervision distribu- tions, which may themselves be far from the KL-minimal solution. As result, the overall frontier is essentially un- changed.
Figure 6: explicit KL regularization helps RL retain prior skills, but barely affects SFT training.
These results suggest that explicit KL regularization cannot rescue SFT from its fundamental limita- tion: SFT forced to imitate whatever distribution is provided, and cannot search for new solutions.
Theory This intuition can be formalized. RL with KL regularization effectively restricts opti- mization to policies achieving a given reward level and then selects the KL-minimal one. Thus, whenever the optimal reward is attainable, RL with sufficiently small KL penalty converges the minimum- KL optimal policy. By contrast, SFT with KL regularization minimizes cross-entropy to fixed annotator distribution plus KL penalty, and in general cannot guarantee alignment with the minimum- KL. solution. Formally:
Theorem A.1. Let A be the set of probability measures on y. and 11 C A a nonempty feasible policy class. Fix a base policy To and a reward R y E. let
Riax = sup E, [R),
p== { E IT: [ Rmax
TEI
For 8 > 0 consider the RL with KL regularization objective:
arg max
sEll [R[
Rmax is attainable by the policy class then there exists B > 0 such that for all B < B.
ag E arg min (T(()))m)) xEP
Now, define annotator distribution objective:
A. For B consider the SFT with KL regularization
SFT n
arg min Eynu llg r (v)) + (KL(T|)))) well
In general, there is no 3 > 0 for which 7 equals the minimum- KL optimal policy, even when itself is optimal (B.gl R Rmax).
16

---

B THEORY
B.1 IMPLICIT BIAS OF ON-POLICY RL
Lemma B.1 (Rejection sampling as an I-projection), Let distribution over a finite set Y, and let R Y + {0, } be reward function. Rejection sampling from with acceptance condition
R(y) yields distribution qus- This distribution can be equivalently characterized as the solution to:
QRs - arg min (xl(alp) s.l.
Ey-n[R()]]== =1
Equivalently, qRs is the I-projection of p onto the set {q: Eg =
Proof. Let e R(y) = 1}. Rejection sampling produces the conditional distribution
p(y) P(S)
y E
QRS (v)
o S,
where p(S) =[ys= p(y) and we assume P(S) > 0.
Now consider the optimization problem. The constraint By [R] = means
Ed( = Σ <v()
BEY
VES
so y must put all of its mass on S. Thus the feasible set is exactly all distributions supported on S.
For any g supported on we can write ply) = p(yls) for y E S, and then
(1)) Σ q(v) log
aly) log ((S) 1 {3)) ply S)
DKI.(glPp)
Σ ((v) log
p(y)
yES
yES
PE9
DKL (p]pp( S)) log P(S)
where used Eyes (8) in the last step. The second term is constant in q, so minimizing DKL(gllP) is the same minimizing DKL(q) [(-IS)). By strict convexity of DKL(-ll-) in its first argument, the unique minimizer is p(- S) QRS-
Lemma B.2 (Policy gradient as an M M-projection). Let a finite set and let S A(Y) be admissible policies (distributions over Y). Consider the single-step reinforcement learning objective
max Eag max
where R: Y Y reward function. By the policy gradient theorem, this objective is equiva- lently optimized by
max Eay [R(y) log T (y)]
where गे indicates that gradients are not propagated through the sampling distribution. Define the distribution
R(y)
<(v)
Z z= Σ R(v)
Z
BEY
Then laking a policy gradient step is equivalent taking a grudient step on the following objective:
min Eay- (1))
In other words, long a q fixed, optimizing the RL objective using policy gradient is equivalent to finding the M --proeection of 4 onto the set feasible policies π using gradient descent.
Proof. Expanding the policy gradient objective gives
Eyy= (R(y log ( = Σ (V)) log 1ogmm( (vv)
WEY
17

---

Let Z Ever xly)R(y). Define q(v) (ly)R(y)/Z Then the above becomes
Σ (og (()) -
Σ log)) Ey~allog (log@(())))))
Since does not depend on æ in the gradient computation (it treated as a constant in the sense), maximizing the original objective is equivalent to maximizing By-g Fyrgllog (v)].
Finally, recall that the . -projectton of a distribution q onto set of distributions IT is given by
min
K m) B, log _ EEglog ql =ell
E, n|
since Eyllog q] does not depend on म, the maximizer of Ea Rog т] over II coincides with arg min xL) Thus, the policy gradient update corresponds to the M -projecction of q onto the policy class.
Theorem B.3 (RL with binary reward EM algorithm). Let Y be finite set and let IT (YY) be a set of feasible policies. Let R : + {0. } be binary reward function and P the set of all optimal policies p = {q- E, 1}. Then, solving the Single-step reinforcement learning objective using policy gradients is equivalent to performing the following optimization procedure:
q: = arg min (L)))
T++1
arg min KI. ([T)
qc
well
This procedure is also known as EM with information projection.
Proof. Sampling y and accepting iff R(y) is exactly rejection sampling onto the event S = Ey E Y: R(y) = 1}. The resulting distribution is x--[S). By Lemma A.1 with t n, this (-1SS) solves
min (gllT) 9
establishing the I-projection. Applying Lemma A.2 on the RL objective gives us the M-projection.
Proposition B.4 (Convergence to minimum KL solution). Under the setting appear in theorem B.3 and assume is an e-flat (exponentialfammll)) model with full support, the optimal set P is nonempty and realizable (i.e., I* + D). Then:
(1) If the M-projection is exact at every step, then (Tt) converges to
π =ra min TIP
DT xp)
(2) If the M-projection is inexact but, for some errors 2 0, it holds that
00
PKL(gel(TeHal min DKL (gellT) e, with well
Σ
E 00
then ne also converges the same limit t..
Proof. The I-step is always exact I-projection (Lemmaa A.1). 1). In the case of an exact M-step, the iterative process is EM with information projections. The e-/m-flat geometry vields the Pythagorean
identities implying convergence to x! (Dempster et al. 1977 Csiszár 1984 Amari & Nagaoka 2000). When the M-step only ensures a mmnimmiziiiiiiiio up to summable errors, the iteration is GEM: monotone improvement and convergence follow from the GEM theory of (1983 together with generalized alternating minimization for Bregman divergences (Gunawardana et al. 2005), which, under the same e-/m-flat assumptions, selects the same minimum-Ki KI. limit .
(1983)
18

---

Practical considerations. Our theoretical equivalence should be interpreted with the following caveats:
Beyond REINFORCE. In practice, many policy gradient algorithms such as GRPO and PPO re- place the reward R(y) with an advantage estimate (ty) Since this substitution is control variate technique, it leaves the expected gradient direction unchanged while reducing its variance. interpretation continues to hold.
Thus, our projection-base
In practice, the M- M-projection step is rarely done until convergence. More common is collect small dataset from the current policy, and then take a few gradient steps on mini-batches from this data before recollecting more.
The optimal policy set Pa defined by the linear constraint E, [R] 1 is m-flat family, but the representable policy set induced by a neural network parametrization This may prevent exact convergence to the minimum-Ke KL solution described above. Nevertheless, our theorem provides a principled explanation for the bias observed in practical RL algorithms.
not in general e-flat.
B.2 KL REGULARIZATION
We will start by analyzing the setting RL with KL regularization:
Theorem B.5 (The solution to RL with KL regularization). Let A be the set of probability measures on V, and II S A a nonempty feasible policy class. Fix a base policy To E II and a reward R y R. For O, consider the penalized problem
max MEll
ET
[R] (KL(T(
(1)
and let v be any maximizer with value Tg = Ev [R]. Then n also solves the constrained problem
min rell
KL(T|ITO)
s.t.
|=[R==1 =13-
(2)
Conversely, â solves equation for some feasible target n, then there exists 3 such that solves equation Aand R4 [t]
= 1).
Proof. From Korbak et al.
(2022) we know that if II A, the solution to Equation is the expo-
nentially tilted distribution
To (y)e
23:=| = f RLy\/B f Toly)
ga
ZB
Zz
For the more general case where 1I A whenever Zg 00, we can write for any 11:
(TT(( = / RT - R /loog
E. R
To
-3 / log
π
@R/3TD
π
- (KL(
-log Zna)
Hence maximizing equation I over is equivalent to
min KL(T]lga) well
By optimality, for every æ E II,
KL KL(TTllas) < (L(T
Using the decomposition KL [I(T||=o) 8- + log Zg, obtain for all æ E II:
KL(TT|lTO) 1 [R] |L( E [R].
Rearranging,
(TT(m] [T(T|T)) (aan)--(EEAA (E n88
ITTIT
19

---

n/s. Plugging this equality into the inequality above kills the Now, fix any II such that E,, [R] last term and yields
KL (L I (L(=|(=)
i.c., among all x II with Ex na, T minimizes KL(-I|=o). This proves that Th solves equation 2
Now for the other direction. Suppose स E II solves equation 2 at some feasible Consider the Lagrangian:
Lx(=) = (=(= + A(n - 1 (R))
By the KKT conditions for the equality constraint, there exists X > such that á minimizes Cx- over II. Equivalently, A maximizes E, [R] (KL(T]|Fo) over II with B 1/X", and necessarily
Ea [ = holds at the maximizer. Thus solves equation [
Corollary B.6. Assume there exists x² II achieving the maximal attainable expected reward Rmax su [R]. Let lopp := {TT :1: [x]R = R.ax). Then there exists B > 0 such that for every B E (0, B), any maximizer r of equation 7 |satissies BRg [R] = Rmax and
Th E arg min xEllopt
(L(TT|
In words: once the KL penalty is small enough that the optimal reward is still achievable, the KL- regularized objective selects the minimum- KL optimal policy.
Now we will move to analyzing the setting of SFT with KL regularization:
Lemma B.7 (The solution SFT with KL regularization). Let A be the set probability measures on y, and II C A nonempty feasible policy class. Fix base policy E II and as the distribution producing the annotations for the SFT training. For O, consider the following objective:
min (EA())
-Eyng log Tt)] 3KL( mo)
(3)
Assume II A. then the unique minimizer Th is given by
{(v)
(4)
4(y)
B BW(
W(
A(3) moly)
where W is the principal branch of the Lambert W function and the scalar A(3) > 0 is chosen to satisfy the normalization E, zy( =
If q(v)) for some y E supp(zo), the formula holds in the limit, yielding Tz(()) - no(y). If
Tol (y) = O and q(v) > 0, the objective value too and no finite minimizer exists.
Proof. Write the Lagrangian
C(m, A) - a(v) log (() Σ (ty) log *(v) + > Σ [()) -1
To
y
8
Stationarity dL/Om(y) = gives
<(v)
ল (
log (य)
g(v)
(1+ )
+ 3 log
+ + -= =0
π(リ) To (v)
To
B x
Let A exp(1 + 2/3) 0. Then:
T ( To(y) 1
exp(
9()) 3Tt
π (ty) exp
9(t)
To(y)/A
Fa(w)
= g(v) Setting :- Tा(a) yields u.e
9(v) J.A [oj whence My = [W (o(
q(v) and thus equation
The normalizer A(3) is uniquely determined by E, (v) 1. Strict convexity of equation the simplex implies uniqueness.
20

---

Theorem B.8 (SFT with KL reg. does not guarantee the minimum-Ki KL optimal policy). Let R.
y + R be a reward and Rmax = sup, R(y). Define the minimum KL optimal policy as the solution
of
min
(T) s.1. E. [R]
Rmax
(5)
REll
Let zi denote the unique minimizer of equation from Lemma
B.7
Then, in general, there is no
B >> such that r equals the minimum- KI. optimal policy.
Proof. (Counterexample), Let y = (1,2,3). R(1) R(2) 1, R(3) 0. Take wo =
(0.6, 0.2, 0.2), hence q KL = (0.75, 0.25, 0). Let q (0.5, 0.5, 0), which is optimal
(g(S) = 1). We show that no 3 >> 0 yields T3 min KL,
Using the stationary condition from the proof of Lemma B.7 and assume T min KL, We get
this set of equations:
0.5
0.75
0.5
0.25
+ 3 log
+1) + +
+ 8 log
0.2 +1) + +
0.75
0.6
0.25
which has no solution, thus a contradiction.
Take y = (1,2.3] with R(1) = R(2) = 1, R(3) = 0. Let To = (0.6, 0.2, 0.2), then the minimum-KI. optimal policy is n min KL (0.75, 0.25, 0). Choose an optimal annotator q = (0.5, 0.5, 0).
Counterexample.
Assume for contradiction that there exist 3 > 0 and À E R such that m3 min KL The first-order condition from Lemma B. reads, for each y.
q(v) + 33 log
* (w) To
+ 1 + = 0.
(y)
Plugging क min (1)(1) we get
(1) 0.75, min KT. (2(())
0.25, To (1) 0.6, =2 0.2, q(1) q(2) 0.5,
0.5
0.75
+ +
log
+ 1
+ 1 +1= 0,
0.75
0.6
0.5
0.25
+
+
log
+
+ =
0.25
0.2
Subtract the first equation from the second to eliminate A. Using log 0.75 the 8-terms cancel and obtain
log 0.25 0.2
= log((.22,
0.0
2 - (- 3) = - 00
a contradiction. Hence no 0 yields T3 min KL in this setting.
C TRAINING AND EVALUATION DETAILS
C.I LLM EXPERIMENTS
Unless otherwise stated, all reinforcement learning experiments were conducted using GRPO (Shao et al. 2024).
For the Math reasoning task, the training set provided final answers but lacked reasoning chains required for SFT training. To obtain these, we queried DeepSeek RI (Guo et al.) 2025a), sampling up to 16 responses per prompt and retaining single response that matched the correct final answer. This yielded valid annotations for 96% of the dataset. For the Science Q&A task, we applied the same procedure with GPT-4o, obtaining correct annotations for the entire dataset.
To construct the low:
trade-off curves (c.g.. Figure 2). we followed the protocol be-
1. Hyperparameter sweep. We trained multiple models under a broad sweep of hyperparame- ters (see Table 2)
21

---

2. New-task evaluation. For Math and Science Q&A, accuracy measured by comparing the model's final answer to the ground truth, ignoring intermediate reasoning chains. For Tool Use, extracted API calls from the output and matched them against ground-truth calls via regular expressions.
3. Previous-task evaluation. We assessed performance on unrelated benchmarks as described in Section 3.1 using the Language Model Evaluation Harness (Gao et al.
2024).
4. Pareto filtering. From the trained models, we retained only those lying within 2 accuracy points of the Pareto frontier.
5. Curve fitting. An exponential function was fit to the filtered points to produce the trade-off curves.
3.50
3.52
100
23.0
3415
0,55
Accurit
Figure 7: Example for the process of creating the pareto frontier plots
Hyperparameter
SFT/SIMPO
RL
Base Model
Qwen2.5 3B-Instruct
Qwen2.5 3B-Instruct
Learning Rate
{1e-5, 3c-5, 5c-5, 7c-5, 9c-5) adamw
{le-5, 2c-5, 3e-5, 4e-5, 5c-53 adamw
Optimizer
LR Scheduler
(constant w. warmup, cosine w. warmup)
constant warmup
Warmup steps
50
50
Epochs
[1.2)
Batch Size
16,32,64,128)
See Below
Max Grad Norm
1
bfloat16
True
True
Weight Decay
GRPO-only hyperparumeters
KI. reg.
Group Size
64
Prompts per generation num iterations (14)
{1,2}
Loss type
Dr. GRPO (Liu et al. 2025
Table Hyperparameters used for the LLM experiments. Curly braces O indicate a sweep over the specified values. Additional parameters such weight decay and max gradient norm were manually ablated; since they showed no significant effect on results, they were not included the final sweep.]
C.2 ROBOTIC EXPERIMENTS
We evaluated the RL-SFT forgetting gap in a robotic control setting using the OpenVLA-7B model (Kim et al. 2024) our base policy in the SimplerEnv environment (Li et al. 2024b). The fine- tuning task was a pic aee---ee scenario requiring the robot to grasp and lift can, while forgetting measured on distinct manipulation task of drawer opening/closing. This setting complements our LLM results by probing whether the KL-forgetting relationship generalizes to embodied poli- cies. To construct the froeto- we follow the same protocol in the LIM experiments.
22

---

Data Collection. Training data were collected by varying object placement over 10 x 10 grid of initial positions: j-j-init [-0.35 0.12]. ob j-iinityy E [-0.02, 0.42). For evaluation, sampled 100 random object locations uniformly in this area.
Supervised Fine-TTrre Tuning (SFT). For each grid point, we collected 10 successful trajectories using the RT-1 (Brohan al. 2022) model and filtered for successful trajectories. We trained models with batch sizes {16, 32, 64] and learning rates {1x10 3x10 5x10 7x10 9xx10 1x10 05, 3x 10 10-5), Other hyperparameters were: AdamW optimizer, training epoch, max gradient norm of 1, weight decay of O, warmup of 10 steps, constant seittant-wit-wwnrne warmup scheduler, and bf ofloott precision.
Reinforcement Learning (RL). For RL, we trained using REINFORCE with an reward normal- ization baseline, without explicit KL regularization. At each iteration, 5 trajectories were collected per grid point. Rewards were binary success indicators of task completion. RL training used the same training config SFT.
C.3 MNIST EXPERIMENTS
All MNIST experiments were conducted using a 3-layer MLP with input dimension 785, hidden layers of sizes 512 and 256, and output dimension 10. The input consisted of a flattened 28 x 28 image concatenated with a binary indicator: 1 for ParityMNIST and -1 for FashionMNIST.
Pretraining. We pretrained the network jointly on ParityMNIST and FashionMNIST using small subsets of the original datasets (500 images from each). For MNISS, the label was chosen uniformly at random among all digit labels with the correct parity.
Fine-tuning methods. In our experiments, we evaluated five fine-tuning strategies:
1. GRPO.
2. GRPO KL regularization with coefficient 0.1.
SFT all even digits mapped to label O, all odd digits to label
SFT even digits randomly mapped to {0, 4}, odd digits to (1.5).
5. SFT with oracle distribution: annotations drawn from the minimum-KL. distribution consistent with task correctness.
Oracle distribution. Motivated by the forgettgn connection, define the oracle distribution as the one that achieves perfect task accuracy while remaining closest (in KL divergence) to the pretraining distribution тeo Concretely, for an input image we compute To (z) E R and binary indicator vector R E (0, 1 encoding which labels are correct given the digit's parity. The oracle distribution q" the solution to:
qº' arg min DKL (Toll)))g
q R
Since KL is convex and the constraint linear, we can calculate a closed-form solution for every image. We then sample from q" to produce SFT annotations. Note that q" is the rejection sampling distribution as aforementioned in Appendix B
Hyperparameter sweep. For each method we trained models across a sweep of 15 learning rates logarithmically spaced between 3e - 6 and le - 3, using either a constant mostaat-witt-waammue or cosine- wtth-warmup scheduler, and training for or 2 epochs. Including mid-training checkpoints, this produced approximately 500 runs per method.
C.4 CENTERED KERNEL ALIGNMENT
Centered Kernel Alignment (CKA) Kornblith et al. 2019
Given representations X,Y E $1 be the centering matrix.
Raxd define kemnlss K XXT. L YY Let H
The centered kernels are
K = HKH, L= HLH.
23

---

CKA is then computed
(K, D)r |K |
CKA(K,L)
where p), ttrAA
CKA with k-NN Alignment (CKNNA) (Huh et al. 2024) Let ((i.j) E {0, }} indicate whether i are mutual k-nearest neighbors in both X and Y. Define the masked inner product
(A, B)
Σ
(l. Big-
CKNNA then given by
(K, L)a V(K, (L, L)a
CKNNA(K,
When (i. for all * CKNNA reduces standard CKA.
D ADDITIONAL RESULTS
D.1 REPRESENTATION PRESERVATION
While benchmark accuracy provides external measure of forgetting, it may conflate genuine loss of capability with superficial effects such formatting mismatch between tasks. To assess whether fine-tuning alters the model more fundamentally, analyzed changes to the model's representa- tions.
Experimental Setup. To study how representations change between models, compare their embeddings on a shared dataset. Following prior work, we compare the relative geometry of the embeddings- that is, how different inputs relate each other. This geometry be summarized by kernel (similarity) matrix, which encodes pairwise relationships among input embeddings. Centered Kernel Alignment (CKA) (Kornblith al. 2019) is a standard measure for comparing such kernels, providing a way to quantify representational similarity between models.
For this analysis, constructed kernels from random Wikipedia paragraphs, ensuring that the probe data are unrelated the fine-tuning tasks. We then compared the kernels of the base model and its fine-tuned variants using CKNNA (Huh et al.] 2004)), ant of CKA (see Appendix C.4 for details). Comparisons were made between SFT and RL models that achieved similar final accuracy the task, isolating repre- sentational differences due to training method rather than task performance.
(CKA) Score
Centered tome Algrmert (CKA)
0.8
nogl--eeghhorrhoo vari-
0.7
100
150
200
250
300
360
400
Results. Figure D.I shows that RL-trained models re- tain high representational similarity (CKNNA=0.94) the base model, with CKNNA scores remaining close one even after fine-tuning on the new task. In con- trast, SFT-trained models exhibit substantial representa- tional drift (CKNNA=0.56). These results indicate that RL fine-tuning integrates new abilities while leaving the overall representation space largely intact, whereas SFT alters the geometry more extensively. Together with the benchmark results, this suggests that RL is able to inte- grate abilities without disturbing the underlying rep- resentational structure, while SET incurs representational shifts that manifest as catastrophic forget- ting.
Gradient Staps
Figure 8: CKA similarity the base model during training. Although SFT and RL achieve comparable task per- formance, SFT models diverge substan- tially in their representations, whereas RL models remain more closely aligned with the base model.
24

---

D.2 SCALING AND FORGETTING
Prior work has suggested that catastrophic forgetting diminishes as model size increases (Ramasesh
al. 2021 Luo al. 2023 Cossu al. 2024). To evaluate this claim in our setting, we repeated the SFT experiments from Section 3 using Qwen 2.5 models with 3B, 7B, and 14B parameters on the Science Q&A task.
The results, shown in Figure P demonstrate that although larger models start with better general capabilities, the trade-off between new-task performance and prior-task retention remains unchanged: across all model sizes, SFT improves new-task accuracy at the expense of forgetting. In partieular, to reach high accuracy on the Science Q&A task, substantial degradation occurs in performance prior benchmarks regardless of model scale.
0.68
0.666
0.64
Prvious
0.60-
0.56
SFT
38
2.54
D.3 OPTIMIZATION DYNAMICS
SFT 148
0.52
0.3
0.4
0.5
0.6
To examine the link between parameter updates and for- getting, we analyzed the optimization trajectory at the level of individual training steps. For each update, we computed two quantities:
New Taik Accuracy
Figure 9: Pareto frontiers for SFT
Qwen 2.5 Instruct models of size 3B, 7B, and 14B on the Science Q&A task. All sizes exhibit the same fundamental trade-off -fan-nifnn-- on the new task require forgetting prior capabilities.
1. Forgetting direction. Using the FashionMNIST eval- uation set, we calculated gradient of loss with respect to model parameters. We then measured the cosine similarity between this gradient and the actual parameter update from the training step. A positive cosine indicates that the update increases FashionM- NIST loss (catastrophic forgetting), while negative cosine indicates update that reduces it.
2. KL shift. We measured the change in KL divergence between the model's output distributions on the Pari- tyMNIST test set before and after the update.
Plotting per-step KL change against the cosine similarity (Figure revealed a strong correlation: steps producing larger KL shifts tended to align more with the forgetting gradient. This analysis demonstrates that at the level of optimization dynamics, catastrophic forgetting is driven by updates that induce larger distributional shifts on the new task.
Tacker
80-
Fastioo MMS11 Acouresy
4
WNIST Accuracy
Figure 10: SFT distillation from an RL teacher. Accuracy trade-off between the new task (MNIST) and the prior task (FashionMNIST), Sweeping student hyperparameters shows that SFT can match the teacher within noise on both tasks. This suggests that what matters is not the opti- mization path, but the distribution of the final model.
25

---

Gradient Simil sit (Paeity NNISTI
Gredtent Similarity (twshion wwwiioonnNNIISSSS MWNIST)
-0.25
20.00
-0.2%
Im2el
-.00
6:22 KL Change (mnist)
8:55
Change (mniss))
Figure 11: Gradient similarity versus KL change. (Left) On the new training task (ParityMNIST), gradient cosine similarity and KL change per step remain uncorrelated. (Right) On the prior task (FashionMNISTT), the gradient similarity is more correlated with the KL change per step on the training task (PIr)) Together, these plots show that taking larger step the current task induces gradients that are more similar direction that forgets the most.
0.58
Avg. Score on Previous Tōsks
0.52
0.50
0.48
0.44
0.44
RL
0.5
1.0
2.0
2.5
3.0
Figure 12: We plot the KL divergence between the base and fine-tuned model on the task, alongside the corresponding forgetting performance across methods.
8
6
Old KL
4
2
2
6
New task KL
Figure 13: KL between the base and trained policy task is highly correlated to the KL on old
lask.
26


Reviews:
[{"review_id": "y2vBghMG8z", "reviewer": "Reviewer_Qf7W", "summary": "This paper shows the effectiveness of using RL in post training in terms of the catastrophic forgetting. Especially, compared to using RL, post-training the model using SFT suffers from forgetting when the model tries to adapt to new tasks. Through extensive experiments, the authors show that using on-policy RL makes the model converge to much closer point from the base model, and using SFT can converge to the points far from the base model. In the experiments on analyzing the contributions of sampling distribution and negative examples to forgetting, the interesting point is that on-policy sampling mainly contributes to preventing the model from being apart from the base model.", "questions": "1. Does this phenomenon also occurs in multi-trun RL? (e.g interaction with environments in agentic RL)\n\n2. Does the robustness come from the on-policy sampling or clipping in PPO-like on-policy methods?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The main contribution on analyzing the RL in terms of the catastrophic forgetting is quite novel. Most of the experiment results are well-aligned to support the claim on the effectiveness of RL, and proposing the quantitative metric on measuring the forgetting has significant contribution.", "weaknesses": "I think this paper is well-written, and cannot find the significant weaknesses.\n\n1. It would be better to specify that all of the RL approaches (e.g. RLVR or RLHF) are robust to the forgetting, and such robustness is still occurs when the model hacks the reward during RL training.\n\n2. It would be better to show the results in terms of the model sizes. I know training much larger LLMs takes enormous resources even though using LoRA tuning, but I still wonder the larger model is also robust to the forgetting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rnUXcZUp3v", "reviewer": "Reviewer_8HAS", "summary": "The paper proposes the hypothesis that RL finetuning, compared to supervised finetuning leads to a minimal change (in the KL sense) in the model distribution, therefore minimizes forgetting. They partially attribute this behavior to the success of RLHF for large LLM finetuning. \nThe authors provide both empirical evidence on LLMs (Math reasoning, Science Q&A, Tool use) and a robotic task (Pick and Place) as well as on a toy task build out of MNIST and FashionMNIST that they call parityMNIST. Finally they formalize and provide some theoretical justification of their hypothesis. \n\nOverall I think the hypothesis is interesting. On the other hand I feel the paper is quite liberal in the claims that it makes and potentially (and most likely not intentionally) misinterprets some of the previous work.", "questions": "1. Can you justify why not the KL constraint embedding in RL objective to stay close to the previous policy is not the main reason why these methods converge to a solution that has a minimal KL change? E.g. jumping to the appendix (lines 805-809) you say that for SFT equals the minimum-KL optimum policy. Why is that the case? What is the technical argument? In my view it should lead to a minimum -KL policy if that is representable by pi for any beta, no? Actually where is the proof of theorem A.1? \n\n2. There is no justification of why the KL computed on the new task data is sufficient to measure forgetness. Can you say more about this (particularly as I feel that implicitly in the later part of the paper you are implicitly making the claim that the fact that that CL algorithms look at the KL on the old data is somewhat problematic -- if that is not the intent, could you have a discussion of this in the paper and make your view explicit). I think it is easy to provide an argument of why KL on new data does not have to correlate with forgetness. E.g. if the old task and the new task have different supports, this metric will say very little about how much you forget (at least under a standard definition of forgetting). So it seems is more of an instance that the old task and new task are sufficiently similar (for an unknown definition of similar). Please also see the connection of this choice and algorithms like Learning without forgetting: https://arxiv.org/abs/1606.09282 which is not cited in current work.  In LwF paper the choice of computing on new data is justified from being cheaper (requiring not to store data) but no claim is being made that is more justified. But I think the predominant view is that the KL computed in the old data tracks the quantity we care, so if the paper tries to argue against this, it should be made more explicit and there should be more evidence for the it.\n\n3. I would rephrase a bit section 6. Actually similar claims are made in the intro. E.g. Kirkpatrick et al do not aim at minimizing change in parameter space. The objective of EWC is explicitly in functional space and talks about the KL term similar to the one you mention (except computed on the old data, so if the claim is that the particular choice of data is wrong please clarify and explain why). Most of CL works do this.  The approximation used to instantiate the algorithm leads to a parameter level regularization due to taking a second order taylor expansion of the KL term (and then approximating the Fisher by a diagonal). But in spirit most of CL algorithms explicitly target minimizing the change in functional space. It is just that minimizing this quantity it needs to be approximated by something cheaper. The framing currently sounds as if these works are minimizing the change in parameters hoping that will lead to less forgetting without connecting to the KL term or acknowledging it.  I would also argue that the claim that these metrics correlate weakly with the KL change is a bit misleading. They are direct approximations of the KL, and for sufficiently large models (where second order Taylor holds), they correlate quite well. Please provide evidence otherwise (and extend the technical description of what exactly is being run for e.g. for table 1). \n\n4. Can you actually have comparisons with at least some basic CL methods? At least replay, which is sometimes used with SFT. Or with SFT + KL regularization?\n\n5. Can you ablate further the RL algorithm used (e.g. PPO, reinforce) or rather remove terms of the GRPO objective to ablate what part of the algorithm empirically leads to small KL term (is the on-policyness or sampling from the model distribution, is it the clipping or KL regularization).  \n\n6.  Can you provide some comments regarding the theory part (outlined above)? I.e. discussing the tabular case, or at least discussing the role of the function approximator in this.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "To me the main strength of the work is the hypothesis proposed. I think it could be the basis of a potentially a novel perspective, and provides an angle on the discussion between RLFT and SFT that I have not seen yet. I'm not aware of work looking at whether on-policyness affects catastrophic forgetting.  The paper provides some decent arguments in favor of their perspective, both empirically and theoretically, though I'm not sure if these arguments are sufficient.", "weaknesses": "My main concern is probably with the generality of the claims (and potentially the lack of ablation to be sufficiently convincing). I would have preferred everything to be a bit more contextualised. Maybe the RL Razor hypothesis (not sure if it is the best name for the hypothesis) is fine -- which I understand it as the source of less forgetting being on-policy behaviour of RL compared to SFT. \n\nBut the empirical results are considerably more specific (e.g. it is about comparing GRPO in the RL context with AdamW for the SL context). Generally I think the main body of the paper is light on technical details (pushing many of these to the appendix), to a point that is harming the scholarly understanding of the work, brushing away some of the nuance of the results. I also think that the paper has not done sufficient work to isolate or ablate in order to show that it is the on-policyness that leads to this behaviour rather than other aspects (e.g. the regularization that is part of the GRPO objective which are also typical for certain optimization algorithms in SL).   \n\nIn terms of generality of the claims:  the choice of optimizer will greatly affect how far away you travel from the starting point. Is not clear to me that a different choice (e.g. natural gradient that includes a KL constraint not that different from GRPO/PPO but enforced more properly) would not lead to a similar minimal change in policy. Of course the approximation made to make the algorithm run efficiently can drastically impact things, and I can not tell if it would be equivalent or not. But that is the problem. The paper makes a broad argument about supervised learning, implying that this will be true for any choice of optimizer.  The same on the RL side, change in objective, change on the optimizer used in the RL setting will of course drastically change how much the model will traverse in functional space making it impossible to predict if the two settings will rank the way the authors are saying. So please be specific in claims.\n\nIn terms of ablations: GRPO contains a regularization as well as a clipping component (beside maximizing the likelihood of the action that gives bigger reward). Is not clear to me to what extent these components are the one driving the observed behaviour, compared to the argument made by the authors that it is because things are on policy. Specifically we know that PPO for example can diverge in RL settings, which to me is the opposite of doing minimal change in the policy in the KL sense. Actually a lot of RL algorithms suffer from such instabilities. How are these behaviours integrated in this hypothesis? \n\nTheory: While I understand the argument made for the RL algorithm, I'm not sure enough work has been put in the counterpart argument, that SFT does not reach a minimal change in the KL sense. Am I misunderstanding this? There does not seem to be any Lemma in the main paper saying that SFT does not do the same as RL, and is not clear to me that bar the optimizer used, SFT would not do the same as RLFT.\n\nI have two main issues with the theory: \n (1) It does not take into account the function approximator (i.e. the neural network) that I think can easily invalidate the argument. I understand that the neural network makes everything non-linear and hence any explicit/concrete mathematical argument is hard to make. But at least acknowledge this. Obviously due to shared parametrization, when the loss is minimized on a particular set of observation, this can change in arbitrary ways the behaviour of the model on not observed data (which is the main point of catastrophic forgetting). So while I appreciate the attempt of adding some theoretical results, I'm not sure to what extent they speak in favor of the hypothesis. \n\nAs a said note, a lot of continual learning work argues that is specifically the neural network (and its non-linear nature) that makes catastrophic forgetting interesting. \n\n(2) Maybe related to the above point. If you consider the tabular case. I do not see who being on policy vs not (in the case where the optimal policy is achievable) leads to different solutions. Maybe there is something to the argument that I'm missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8UI9UrshH", "reviewer": "Reviewer_muvc", "summary": "The paper investigates why reinforcement learning (RL) fine-tuning tends to forget less than supervised fine-tuning (SFT) when adapting foundation models to new tasks. Across LLM and robotics benchmarks (and a controlled ParityMNIST toy experiment), the authors show an empirical forgetting law: the degree of catastrophic forgetting is well predicted by the KL shift measured on the new-task inputs, $\\mathbb{E}_{x\\sim\\tau}\\big[ \\mathrm{KL}(\\pi_0\\|\\pi)\\big]$. They argue (and empirically validate) that on-policy RL is implicitly biased toward KL-minimal solutions among all policies that solve the new task — because RL samples from and reweights the model’s own outputs — while SFT can converge to arbitrarily distant solutions given the annotator distribution.\n\nThe paper’s contributions are threefold: (1) empirical demonstration that RL fine-tuning preserves prior capabilities better than SFT at matched new-task performance, (2) the KL–forgetting law showing forward (and backward) KL on the new-task distribution predicts forgetting, and (3) theoretical and experimental evidence for “RL’s Razor” — a proof that policy-gradient on-policy methods converge to the optimal representable policy closest in KL to initialization, plus ablations (oracle SFT, KL regularization, different objectives) that validate the mechanism.", "questions": "One clarity remark - I am not sure how dist. 1 and dist. 2 differ - it's not easy to find in the text or in the Figure caption.\nAdditionally, I would like to know whether the authors plan to conduct any further experiments with VLA, as the paper currently only presents a single task: Pick and Place. I'm wondering whether the trend of catastrophic forgetting with SFT is more severe across some family of manipulation tasks and maybe less severe in some others? Can authors provide any intuition about that?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper presents solid evidence for a novel and significant claim that online RL finetuning preserves the base model's knowledge better than SFT finetuning. This claim has far-reaching consequences for fine-tuning pipelines in foundational models, both in LLMs and Robotics (VLAs and Behavioral Foundation Models) space. In addition to an extensive empirical evaluation of SFT and RL finetuning on Qwen 2.5 models and OpenVLA, the paper also presents the forgetting phenomenon during fine-tuning from a theoretical perspective. I find the provided toy experiment on ParityMNIST to be illustrative and instructive for gaining intuition. The soundness of the experiment section is high-level, and the presentation is clear and of high quality.", "weaknesses": "The paper does not present any obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k25khJD4ho", "reviewer": "Reviewer_xH9f", "summary": "This paper finds that fine-tuning base model by RL on new tasks can better resist catastrophic forgetting compared to fine-tuning by SFT. The authors find that the KL divergence between fine-tuned model and base model can be a strong predictor of catastrophic forgetting. RL forgets less because on-policy sampling inherently biases the model toward KL-minimal solutionns, while SFT may drift arbitrarily far to adapt to new tasks. Their empirical evidence and theoretical justifications support the claims.", "questions": "1. In the toy experiment, forgetting is significant, but in LLM tasks, the absolute value of forgetting is small. Why is there such a big gap between the two? Is it due to differences in model size, optimization robustness, task overlap, or KL sensitivity across models of different sizes?\n1. The compression in Fig. 2 and Fig. 4 visually makes forgetting look more severe than the actual numerical value.  Would the author be willing to provide an absolute difference or a version with a uniform scale?\n1. The authors test alternative indicators in distribution change, weight change, and activation change in Sec. 6. Have the authors tested other distribution-level indicators, like JS divergence and Wasserstein distnace? Will they perform better?\n1. Is this finding only suitable for on-policy RL, what about off-policy RL?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The cor insight is simple but efficient: the extent of catastrophic forgetting is determined by KL divergence. That RL does better than SFT is because the on-policy nature which minimizes the KL divergence. This insight can bring more insights of RLHF and continual post-training research\n1. The takeaway messages are consice and clear, directly conveying the most important conclusions.\n1. The experiments including LLM and robotics control are comprehensive to illustrate the effectiveness and generality of the KL-based explanation, demonstrating consistent forgetting behavior and RL's advantage.", "weaknesses": "1. The analysis of KL divergence in catastrophic forgetting is still high level:\n    1. The sampling distribution and KL divergence is a correlation analysis but not a causality explanation. A mathematics proof should be more helpful to support this finding\n    1. The paper does not provide a deeper examination of how KL drift translates into the loss on prior tasks. For instance, it is unclear whether the degradation arises from representational collapse on past tasks, or from changes in critical neurons that significantly affect previously acquired skills. A more fine-grained analysis would greatly strengthen the causal argument behind the proposed explanation.\n1. As KL divergence is an indicator to forgetting, it would be meaningful to add KL-regularized online and offline methods as baselines, e.g.,[1,2,3], which also extends the scope if minimizing KL divergence directly can mitigate forgetting.\n1. The experiments in the paper indicate that the forgetting is substantial in toy experiments, but noticeably milder in LLM-based tasks. However,Fig. 2 and Fig. 4 exaggerate the performance loss by compressing the y-axis. This plotting is misleading and give readers an inflated perception of catastrophic forgetting, overstating the empirical impact of the proposed phenomenon.\n\n[1] Azar, Mohammad Gheshlaghi, et al. \"A general theoretical paradigm to understand learning from human preferences.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\n[2] Ethayarajh, Kawin, et al. \"Kto: Model alignment as prospect theoretic optimization, 2024.\" URL https://arxiv. org/abs/2402.01306.\n\n[3] Wu, Yifan, George Tucker, and Ofir Nachum. \"Behavior regularized offline reinforcement learning.\" arXiv preprint arXiv:1911.11361 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)