Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training

## Abstract

Abstract Continual post-training (CPT) is a popular and effective technique for adapting foundation models like multimodal large language models to specific and ever-evolving downstream tasks. While existing research has primarily concentrated on methods like data replay, model expansion, or parameter regularization, the fundamental role of the learning paradigm within CPT remains largely unexplored. This paper presents a comparative analysis of two core post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), investigating their respective impacts on knowledge retention during CPT. Our experiments are conducted on a benchmark comprising seven diverse multimodal tasks, utilizing Qwen2.5-VL-7B-Instruct as the base model for continual post-training. The investigation yields two significant findings: (1) When continuously learning on downstream tasks, SFT leads to catastrophic forgetting of previously learned tasks. In contrast, RFT inherently preserves prior knowledge and achieve performance comparable to multi-task training. (2) RFT successfully protects and even enhances the model’s general knowledge on standard benchmarks (e.g., MMMU and MMLU-Pro). Conversely, SFT degrades general model capabilities severely. Further analysis reveals that this stability is not primarily due to explicit mechanisms like KL penalty or chain-of-thought reasoning. Instead, we identify an implicit regularization mechanism inherent to RFT as a key contributing factor. Our theoretical analysis suggests that RFT’s gradient updates are naturally scaled by the reward variance, acting as a data-dependent regularizer that inherently protects previously acquired knowledge. Finally, we propose a rollout-based instance filtering algorithm to enhance the stability and efficiency of RFT. Our comprehensive study demonstrates the superiority of RFT as a robust paradigm for continual post-training. 1 1 1 Our code is provided in the supplementary material. An anonymous link for review is: https://anonymous.4open.science/r/RFTvsSFT-A999

## 1 Introduction

Recent advancements in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in complex world understanding (Achiam et al., 2023; Liu et al., 2024; Wang et al., 2024a). To align with the demands of real-world deployment, MLLMs must adapt to a stream of data and evolving user requirements, incorporating new skills and domain knowledge over time (Zhu et al., 2024). This calls for an efficient and scalable continual post-training (CPT) paradigm. A key challenge in CPT is the well-known phenomenon of catastrophic forgetting (McCloskey & Cohen, 1989), where adapting to a new task leads to a severe degradation of performance on previously learned tasks. To reduce forgetting, recent studies (Guo et al., 2025c) focus on data replay (Maharana et al., 2025; Lee et al., 2025; Wang et al., 2025), model expansion (Zhao et al., 2025; Guo et al., 2025b; Zeng et al., 2024), and explicit regularization (Liu et al., 2025a).
Nevertheless, existing methods typically leverage the supervised fine-tuning (SFT) paradigm by default, and the role of the fundamental fine-tuning paradigm in CPT has been overlooked.

Recently, reinforcement fine-tuning (RFT), which optimizes models based on feedback from generated outputs, has significantly advanced foundation model post-training (Chu et al., 2025; Shao et al., 2024; Guo et al., 2025a). To the best of our knowledge, this work presents the first direct comparative investigation into whether SFT or RFT is the more suitable paradigm for CPT, focusing on knowledge preservation for both specific downstream tasks and general capabilities. Experimentally, we continually fine-tune the Qwen2.5-VL-7B-Instruct model (Bai et al., 2025b) on a benchmark comprising diverse multimodal tasks covering various domains. To fully reflect the knowledge preservation ability, we evaluate forgetting on both learned specific tasks and general benchmarks such as MMMU (Yue et al., 2024), MMLU-Pro (Wang et al., 2024b), and POPE (Li et al., 2023a).

The empirical investigation yields two notable findings: (1) As shown in Figure [1](https://arxiv.org/html/2507.05386v3#S0.F1), when continuously learning on downstream tasks, SFT leads to catastrophic forgetting of previously learned tasks, which is consistent with existing studies (Guo et al., 2025c). In contrast, RFT can inherently protect prior knowledge, maintaining strong performance on old tasks after being adapted to new tasks. Surprisingly, without any data replay, continual post-training with RFT can achieve comparable performance with that of multi-task training, which is not achievable even when equipping SFT with continual learning strategies. (2) As demonstrated in Figure [2](https://arxiv.org/html/2507.05386v3#S1.F2), continual training on downstream tasks with SFT severely degrades general model capabilities, which is known as base model degradation (Liu et al., 2025a). For example, the performance drops from 52.1% to 40.1% on MMMU. Fortunately, RFT protects the general performance and enhances the model’s general knowledge (52.1% → 54.2%). These observations highlight the knowledge preservation capability of RFT.

Figure: Figure 2: General capability preservation after continual post-training. We evaluate models at the end of learning all downstream tasks on general benchmarks using both CoT and direct prompting. Compared to the base model, SFT (shown in light colors) causes degradation while RFT (shown in darker colors) preserves and even enhances general capabilities.
Refer to caption: x2.png

To understand how RFT mitigates forgetting during CPT, we conduct additional experiments with the popular and representative group relative policy optimization (GRPO) framework (Shao et al., 2024). We analyze the impact of KL divergence penalty and chain-of-thought (CoT) reasoning (Wei et al., 2022) on forgetting mitigation. Particularly, the KL divergence penalty prevents the policy from changing too drastically, similar to the well-known knowledge distillation in continual learning (Li & Hoiem, 2017). However, our analysis indicates that these explicit mechanisms are not the primary drivers of forgetting mitigation. We instead attribute this phenomenon to an implicit regularization effect within RFT. We offer a theoretical perspective suggesting that RFT’s updates are inherently more conservative in parameter subspaces sensitive to prior tasks. This conservatism is naturally scaled by the variance of the reward signal, creating a data-dependent regularization that dampens updates on uncertain samples, thus protecting established knowledge. Last but not least, we observe that the learning process of RFT can be highly inefficient. Thus, we introduce a rollout-based instance filtering algorithm that enhances the stability of GRPO while still being an excellent knowledge protector.

Our main contributions are threefold:

- 1.
We present the first comprehensive analysis of the forgetting mitigation effects of SFT and RFT during continual post-training of MLLMs, demonstrating that RFT naturally preserves not only the performance of learned downstream tasks but also general model capabilities.
- 2.
Based on in-depth analyses, we reveal that the implicit regularization introduced by RFT significantly contributes to the forgetting mitigation, being more important than KL regularization and CoT reasoning.
- 3.
We propose a rollout-based instance filtering algorithm that enhances the stability and efficiency of RFT while still maintaining previous learned knowledge.

## 2 Related Works

#### Continual Post-Training in MLLMs.

Continual learning aims to enable models to learn from a stream of tasks without catastrophically forgetting previously acquired knowledge (Van de Ven et al., 2022). For MLLMs, this capability is particularly important for adapting these powerful models to a diverse range of downstream multimodal tasks. Existing CPT research in MLLMs (Guo et al., 2025c) has focused on adapting traditional forgetting mitigation strategies such as regularization, data replay, and model expansion, within an SFT paradigm. Regarding benchmark, Chen et al. (2024) introduced a continual instruction tuning benchmark including several specific multimodal datasets. Zhao et al. (2025) introduces two settings named domain continual learning and ability continual learning, providing a realistic evaluation for continual post-training of MLLMs. In addition to these methods, recent efforts to mitigate catastrophic forgetting in MLLMs primarily focus on parameter-efficient learning and dynamic data selection. For instance, HiDe-LLaVA (Guo et al., 2025b) employs a hierarchical decoupling framework for task-specific LoRA expansion and general knowledge fusion. MRLoRA (Zhao et al., 2025) leverages architectural decoupling and a multimodal routing mechanism to selectively activate specialized parameters. In terms of data management, Adapt-$\infty$ (Maharana et al., 2025) dynamically selects high-impact samples based on gradient representations and prunes redundant data. These diverse strategies collectively aim to enhance the ability of MLLMs to continually learn new tasks while preserving previously acquired knowledge. Recently, Liu et al. (2025a) developed LLaVA-c, which is a simple yet effective CPT framework for MLLMs, addressing task balancing and catastrophic forgetting through spectral-aware consolidation and unsupervised inquiry regularization.

#### Post-Training of Foundation Models.

Post-training is a critical stage for refining the capabilities of pre-trained foundation models (Shao et al., 2024; Chu et al., 2025; Achiam et al., 2023). SFT on task-specific or instruction-formatted datasets is a common approach to adapt models to downstream applications (Chung et al., 2024; Zhou et al., 2023). For example, Chung et al. (2024) demonstrated that by scaling the number of tasks and model size, and incorporating CoT data, SFT significantly enhances the performance and generalization of various large language models across diverse benchmarks.
Recently, RFT has gained prominence for aligning models with human preferences or improving performance on specific objectives (Liu et al., 2025c; Zhai et al., 2024; Shao et al., 2024; Luong et al., 2024; Li et al., 2025; 2023c; Ahmadian et al., 2024). Particularly, GRPO (Shao et al., 2024) largely enhances mathematical reasoning and optimizes memory usage, being a popular method for post-training of large language models. Liu et al. (2025b) revealed inherent biases in the GRPO algorithm, then introduces an unbiased optimization method that improves token efficiency while
maintaining reasoning performance. Visual-RFT (Liu et al., 2025c) boosts MLLMs by using reinforcement learning with rule-based visual rewards, making them more data-efficient and better at various visual tasks than traditional SFT. Recently, Chu et al. (2025) demonstrated that reinforcement learning significantly enhances the generalization capabilities of foundation models, while SFT primarily leads to memorization.
In this work, we study the comparative effect of SFT and RFT on knowledge retention in MLLMs continual post-training.

## 3 Preliminaries

Post-training is a critical phase following large-scale pre-training that adapts foundation models to specific downstream tasks or align them with human preferences (Ouyang et al., 2022; Kumar et al., 2025). We model the MLLM with parameters $\theta$ as a policy $\pi_{\theta}$. This policy defines a conditional probability distribution $\pi_{\theta}(a|x)$ over possible text responses $a$ given a multimodal input prompt $x$, which consists of text and images. We also assume a scalar reward function $r(x,a)\in\mathbb{R}$ that evaluates the quality of a response. Post-training aims to update the parameters $\theta$ of a pre-trained base model $\pi_{\theta_{\text{base}}}$ to improve its performance on a downstream task using a training dataset $\mathcal{D}$, which can be achieved by SFT (Ouyang et al., 2022) or RFT (Lee et al., 2023).

#### SFT.

Given training dataset $\mathcal{D}=\{(x_{i},a_{i}^{*})\}_{i=1}^{N}$ consisting of prompts $x_{i}$ and their corresponding ground-truth responses $a_{i}^{*}$, SFT maximizes the likelihood of generating the ground-truth responses. This is typically achieved by minimizing the negative log-likelihood loss:

$$ $\mathcal{L}_{\text{SFT}}(\theta)=-\mathbb{E}_{(x,a^{*})\sim\mathcal{D}}[\log\pi_{\theta}(a^{*}|x)]=-\mathbb{E}_{(x,a^{*})\sim\mathcal{D}}\left[\sum_{t=1}^{|a^{*}|}\log\pi_{\theta}(a_{t}^{*}|x,a_{<t}^{*})\right].$ (1) $$

#### RFT.

In RFT, the model $\pi_{\theta}$ is treated as a policy, and generates one or more candidate responses for a given prompt $x$. The optimization objective is to maximize the expected reward:

$$ $\mathcal{J}_{\text{RFT}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\mathbb{E}_{a\sim\pi_{\theta}(\cdot|x)}[r(x,a)].$ (2) $$

The gradient of this objective is typically estimated using policy gradient methods. The most basic form is the REINFORCE (Williams, 1992) estimator, which, unfortunately, has high gradient variance. Recent RFT algorithms (Shao et al., 2024; Li et al., 2023c; Ahmadian et al., 2024) address this issue by designing more stable advantage estimators and baselines. We introduce some of the representative methods used in our study below.

For a prompt $x$, *GRPO* (Shao et al., 2024) generates a group of $n$ responses $\{a_{1},\dots,a_{n}\}$ and computes their rewards $\{r_{1},\dots,r_{n}\}$. The advantage for a response $a_{i}$ is its normalized reward relative to the group mean: $A(a_{i})=(r_{i}-\bar{r})/\sigma_{r}$, where $\bar{r}$ and $\sigma_{r}$ are the mean and standard deviation of the rewards. The objective is to maximize the expected advantage-weighted log-probability, often with a KL-divergence penalty against a reference policy $\pi_{\text{ref}}$ to stabilize training:

$$ $\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{x,\{a_{i}\}}\left[\sum_{i=1}^{n}A(a_{i})\log\pi_{\theta}(a_{i}|x)\right]-\beta D_{\text{KL}}(\pi_{\theta}(\cdot|x)||\pi_{\text{ref}}(\cdot|x)),$ (3) $$

where $\beta>0$. *ReMax* (Li et al., 2023c) use the reward of a greedy decoding response $a$ as a baseline. For a single sampled response $a$, the objective is to maximize:

$$ $\mathcal{J}_{\text{ReMax}}(\theta)=\mathbb{E}_{x,a\sim\pi_{\theta}}\left[(r(x,a)-r(x,a))\log\pi_{\theta}(a|x)\right].$ (4) $$

This adaptive baseline helps to normalize rewards and reduce gradient variance.
To further reduce variance, *RLOO* (Ahmadian et al., 2024) generates $n$ samples $\{a_{1},\dots,a_{n}\}$ and uses the average reward of the other $n-1$ samples as a baseline for sample $a_{i}$:

$$ $\mathcal{J}_{\text{RLOO}}(\theta)=\mathbb{E}_{x,\{a_{i}\}}\left[\frac{1}{n}\sum_{i=1}^{n}\left(r(x,a_{i})-\frac{1}{n-1}\sum_{j\neq i}r(x,a_{j})\right)\log\pi_{\theta}(a_{i}|x)\right].$ (5) $$

#### Continual Post-Training Formulation.

In CPT, the model learns from a sequence of $T$ tasks with datasets $\{\mathcal{D}_{1},\dots,\mathcal{D}_{T}\}$. The core challenge is catastrophic forgetting, i.e., a significant drop in performance on previously learned tasks. Following the general continual learning framework, CPT can be formulated as a constrained optimization problem. When learning task $t$, the objective is:

$$ $\theta^{t}=\arg\min_{\theta}\mathcal{L}(\theta;\mathcal{D}_{t})\quad\text{s.t. }\mathcal{L}(\theta;\mathcal{D}_{i})\leq\mathcal{L}(\theta^{i};\mathcal{D}_{i}),\quad\forall i\in[1,t-1]$ (6) $$

where $\mathcal{L}(\theta;\mathcal{D}_{i})$ is the training objective (e.g., negative log-likelihood for SFT or negative expected reward for RFT) on task $i$, and $\theta^{i}$ are parameters after learning task $i$.

## 4 Reinforcement Fine-Tuning Mitigates Forgetting in CPT

This section presents our comparative results comparing RFT and SFT in a continual post-training scenario. We detail our experimental setup and then present the main findings that highlight the superiority of RFT for knowledge preservation.

Figure: Figure 3: Illustrative examples of continual post-training benchmark.
Refer to caption: x3.png

### 4.1 Experimental Setup

#### Continual Post-Training Model & Datasets.

We adopt the open-source Qwen2.5-VL-7B-Instruct (Bai et al., 2025a) as our base model, primarily due to its demonstrated superiority in vision-language comprehension and its favorable resource footprint, which is crucial for practical deployment.
We continually fine-tune the model on diverse vision-language datasets (ScienceQA (Saikh et al., 2022), TextVQA (Singh et al., 2019), VizWiz (Gurari et al., 2018), GQA (Hudson & Manning, 2019), Geometry3K (Lu et al., 2021), PathVQA (He et al., 2020), Super-CLEVR (Li et al., 2023b)), covering a wide range of common downstream applications. After the end of CPT, evaluation is performed on the test sets of all previously encountered tasks.
Additionally, to fully assess the knowledge preservation ability, we evaluate the model on diverse, general benchmarks at the end of learning all downstream tasks. Specifically, we evaluate the model on three specialized benchmarks: MMMU (Yue et al., 2024), MMLU-Pro (Wang et al., 2024b), and POPE (Li et al., 2023a). Particularly, we include POPE to systematically assess whether CPT induces object hallucination in MLLMs. A detailed description of those datasets is provided in the Appendix.

#### Learning Algorithms & Reward.

Our experiments encompass a range of fine-tuning algorithms, including standard SFT (Zheng et al., 2024) and several representative RFT algorithms, i.e., GPRO (Shao et al., 2024), ReMax (Li et al., 2023c), and RLOO (Ahmadian et al., 2024). For both SFT and RFT, model outputs are normalized by disregarding extraneous whitespace (e.g., spaces, indentations, newlines) and ignoring case sensitivity to ensure precise assessment. For GRPO, the overall reward $r_{\text{overall}}$ is designed with a weighted sum of accuracy reward and format reward:

$$ $r_{\text{overall}}=0.9r_{\text{acc}}+0.1r_{\text{format}}.$ (7) $$

Specifically, the accuracy reward $r_{\text{acc}}$ assesses the semantic correctness of the generated content, which yields a reward of 1 if the generated answer $a$ matches the ground truth answer $a^{*}$, and 0 otherwise. The format reward assesses adherence to the expected output structure. It utilized regular expressions to verify the correct presence and formatting of the CoT reasoning block, delineated by <think> and </think> tags, and the final answer encapsulated within a \boxed{} environment. A perfect format match resulted in a score of 1, otherwise 0.

#### Prompt Template.

Our base model, Qwen-VL-7B-Instruct, utilizes two kinds of input prompt templates, as illustrated in the Appendix.
NoCoT (non-chain-of-thought) prompt template adheres to a basic question-answering format, where the question text is presented directly, and the model is expected to provide the final answer without intermediate steps. Differently, in CoT prompt template, the query’s question text is directly incorporated into the prompt, followed by an instruction for the model to first engage in a reasoning process. This CoT reasoning is then generated within a dedicated <think> and </think> block. The final answer is explicitly distinguished and encapsulated within a \boxed{} environment.

#### Evaluation Metrics.

To quantify the model’s performance during CPT, we adopt two standard metrics. Let $P_{t,j}$ denote the test accuracy on task $j$ after learning task $t$.
We measure the final overall performance using average accuracy (AvgAcc), which is the average accuracy across all tasks after training on the final task $T$. To measure knowledge retention, we use the forgetting measure (FM), which calculates the average difference between the final accuracy of a task and the best accuracy achieved for that task throughout the training sequence. Let $P_{i}^{*}=\max_{k\in\{i,\dots,T\}}P_{k,i}$ be the best performance for task $i$. The above two metrics are defined as:

$$ $AvgAcc=\frac{1}{T}\sum_{i=1}^{T}P_{T,i},\qquad FM=\frac{1}{T}\sum_{i=1}^{T}(P_{T,i}-P_{i}^{*}).$ (8) $$

A higher AvgAcc indicates better overall performance, while an FM closer to zero signifies less forgetting and better knowledge preservation.

#### Implementation Details.

All experiments employ full-parameter fine-tuning for both SFT and RFT to ensure comprehensive capability assessment. Experiments of SFT are conducted using the llamafactory (Zheng et al., 2024) framework, with a learning rate of $1e-5$ and a batch size of 24. RFT methods (GRPO, ReMax, and RLOO) are implemented using the easyR1 (Zheng et al., 2025) framework, building upon Verl (Sheng et al., 2024).
A consistent configuration is applied across RFT methods to ensure an equitable comparison: a learning rate of $1e-6$, a rollout batch size of 512, a sampling temperature of 1.0, with KL-divergence coefficient $\beta=0.01$. Specifically, GRPO is implemented adhering to its foundational methodology, with a group size set to 8. ReMax followed its core algorithm, and RLOO adopted the official Hugging Face algorithm.

Table: Table 1: Final performance comparison on all tasks after the entire continual learning sequence. The best and second-best results are highlighted. “-” indicates that the metric is not applicable.

Table: Table 2: General capabilities evaluation on MMMU, MMLU-Pro, and POPE benchmarks after training on downstream tasks. The best and second-best results are highlighted.

### 4.2 Finding 1: RFT Inherently Resists Catastrophic Forgetting

Our primary investigation focuses on the knowledge retention capabilities of SFT and RFT within a continual learning sequence. The results, summarized in Table [1](https://arxiv.org/html/2507.05386v3#S4.T1), reveal a contrast between the two paradigms.

#### SFT suffers from catastrophic forgetting.

We observe that sequential SFT leads to a severe degradation of performance on previously learned tasks with a forgetting measure (FM) of -10.4%. For instance, performance on ScienceQA drops dramatically (95.2% $\rightarrow$ 76.1%) after completing the entire task sequence. The final average accuracy (AvgAcc) of 54.0% is substantially lower than the multi-task learning of SFT, which is the upper bound of 62.9%, confirming that SFT is highly susceptible to forgetting.

#### RFT preserves task knowledge and achieves MTL performance.

In contrast, all RFT methods demonstrate remarkable resilience against forgetting. As shown in Table [1](https://arxiv.org/html/2507.05386v3#S4.T1), RFT methods exhibit very low forgetting measures, with GRPO achieving an FM of -2.3%. For example, GRPO maintains ScienceQA performance at 93.0% after learning all tasks, compared to its peak performance of 95.6%, which is a minimal drop compared to SFT. Among RFT methods, GRPO performs best, achieving a final AvgAcc of 60.0%, which is close to the upper bound of 62.9%. The model achieves this high performance without any explicit continual learning strategies, suggesting that the RFT paradigm is inherently robust for CPT.

### 4.3 Finding 2: RFT Protects and Enhances General Capabilities

Beyond task-specific knowledge, an ideal CPT process also requires preserving the model’s foundational, general-purpose abilities. We evaluated the models on general benchmarks to measure this effect. The results, presented in Table [2](https://arxiv.org/html/2507.05386v3#S4.T2), highlight another critical advantage of RFT.

#### SFT harms general capabilities in both CL and MTL.

Our experiments reveal that SFT causes significant base model degradation (Liu et al., 2025a). SFT induces a severe performance drop of $\downarrow$16.9% on the challenging MMLU-Pro benchmark (47.5% $\rightarrow$ 30.6%). Crucially, this is not merely an artifact of sequential learning; even multi-task SFT (MTL (SFT)), which trains on all data simultaneously, still causes a severe drop of $\downarrow$14.3% on the same benchmark. A similar trend is evident on MMMU, where SFT and MTL (SFT) cause performance to decline by $\downarrow$12.0% and $\downarrow$4.3% respectively. This demonstrates that the SFT paradigm itself appears harmful to the model’s foundational capabilities.

#### RFT preserves and enhances general capabilities.

In contrast to the capability decay observed across all SFT methods, the RFT paradigm effectively safeguards the model’s general abilities. GRPO, in particular, often enhances these abilities. For instance, GRPO improves performance on MMMU by $\uparrow$2.1% (52.1% $\rightarrow$ 54.2%). Crucially, RFT also improves model general capabilities, with GRPO improving the POPE score by $\uparrow$1.9% (86.6% $\rightarrow$ 88.5%) and reducing the tendency for hallucination. This clear difference highlights that RFT is a more robust paradigm for continual post-training.

## 5 How Does RFT Mitigate Forgetting?

To investigate the mechanisms behind RFT’s remarkable stability, this section presents a series of ablation studies based on the popular and representative GRPO algorithm Shao et al. (2024).

Table: Table 3: Downstream task performance for ablation models. We investigate the role of the KL term and CoT through variants of GRPO. ^† indicates that the training process is unstable and requires multiple restarts from a previous checkpoint to achieve convergence.

Table: Table 4: General capabilities evaluation for ablation models. Each benchmark is evaluated with and without CoT prompts to provide a comprehensive view.

### 5.1 The Roles of CoT and KL Penalty

We test two primary hypotheses: (1) The KL-divergence penalty, by regularizing policy updates, acts as a form of knowledge distillation (Li & Hoiem, 2017) that preserves past knowledge. (2) The complex reasoning structure of CoT builds more abstract and resilient knowledge representations, protecting them from being overwritten. Thus, we evaluate three GRPO variants against the SFT baseline: GRPO w/o KL: trained with CoT prompts but without the KL penalty term. GRPO w/o CoT: trained without CoT prompts, using direct question-answering format but retaining the KL penalty.

#### KL penalty is not the primary factor for preserving task-specific knowledge.

As shown in Table [3](https://arxiv.org/html/2507.05386v3#S5.T3), removing the KL penalty (GRPO w/o KL) causes no degradation in performance on the continual learning sequence. The final average accuracy remains, demonstrating that the KL penalty is not the primary mechanism preventing task-specific catastrophic forgetting. However, it is crucial to note that the training process without the KL penalty exhibits significant instability in the later stages of the task sequence. These results are obtained after multiple attempts, re-initializing from the previous task’s checkpoint to achieve a convergent outcome, which suggests KL penalty plays a critical role in stabilizing the RFT process.

Figure: Figure 4: Loss landscapes illustration of SFT and RFT based continual post-training.
Refer to caption: x4.png

#### CoT is a performance booster, not a forgetting mitigator.

Our second hypothesis is also not supported by the data. The model trained without CoT (GRPO w/o CoT) still strongly resists forgetting, maintaining a high average accuracy across the task sequence (Table [3](https://arxiv.org/html/2507.05386v3#S5.T3)). In fact, it outperforms GRPO on VizWiz ( 63.8% vs. 51.8%). The general capabilities evaluation in Table [4](https://arxiv.org/html/2507.05386v3#S5.T4) further confirms this conclusion. The GRPO w/o CoT model remains robust, and it achieves the highest score on the POPE benchmark (88.7%) when tested in non-CoT format evaluation. This demonstrates that while CoT can enhance performance on certain types of tasks, it is not the mechanism responsible for RFT’s resistance to catastrophic forgetting. Besides, as shown in Table [3](https://arxiv.org/html/2507.05386v3#S5.T3), we observe that for GRPO w/o KL, using CoT during inference would lead to notable hallucination.

### 5.2 Implicit Regularization from Reward Variance

To build intuition for the empirical resilience of RFT, we analyze its gradient dynamics in the context of continual learning. Our analysis suggests that RFT’s forgetting mitigation stems from an implicit regularization mechanism, where the learning signal itself modulates the update strength. To explore this intuition, we adopt the concept of forgetting risk from continual learning theory (Kirkpatrick et al., 2017), using the Fisher Information Matrix (FIM) as a tool to quantify parameter sensitivity to past tasks. This allows us to conceptually link the structure of RFT’s gradients to knowledge retention.

###### Definition 5.1 (Forgetting Risk) .

Let $\mathcal{D}_{1:k-1}$ be the data from all previously learned tasks. The FIM is defined as $F_{k-1}\triangleq\mathbb{E}_{(x,a^{*})\sim\mathcal{D}_{1:k-1}}[\nabla_{\theta}\log\pi_{\theta}(a^{*}|x)(\nabla_{\theta}\log\pi_{\theta}(a^{*}|x))^{\top}]$. The *forgetting risk* of a gradient update $g$ for the current task $k$ is defined as its squared Mahalanobis norm with respect to the FIM of past tasks:

$$ $\mathcal{R}(g)\triangleq g^{\top}F_{k-1}g.$ (9) $$

This risk measures the update’s magnitude in parameter subspaces critical for prior knowledge. Note that $F_{k-1}$ is a theoretical construct for our analysis and is not computed in practice.

For a single data point $(x_{k},a_{k}^{*})\in\mathcal{D}_{k}$, the SFT loss gradient is $g_{\text{SFT}}=-\nabla_{\theta}\log\pi_{\theta}(a_{k}^{*}|x_{k})$. In contrast, the RFT policy gradient for a sampled response $a\sim\pi_{\theta}(\cdot|x_{k})$ is $g_{\text{RFT}}(a)=A(x_{k},a)\nabla_{\theta}\log\pi_{\theta}(a|x_{k})$, where $A(x_{k},a)$ is an advantage function ( $r(x_{k},a)-b(x_{k})$).

The following proposition establishes a conceptual link between the expected forgetting risk of an RFT update and that of an SFT update, highlighting the central role of reward variance.

###### Proposition 5.2 (RFT’s Implicit Regularization Effect) .

Consider a single update on task $k$ at parameters $\theta_{k-1}$. Let the rewards be normalized, $r(x_{k},a)\in[0,1]$. Under the technical assumptions specified in Appendix [B](https://arxiv.org/html/2507.05386v3#A2), the expected forgetting risk of an RFT update is related to the SFT risk by:

$$ $\mathbb{E}_{a\sim\pi_{\theta_{k-1}}}[\mathcal{R}(g_{\text{RFT}}(a))]\approx\text{Var}_{a\sim\pi_{\theta_{k-1}}}[r(x_{k},a)]\cdot\mathcal{R}(g_{\text{SFT}}),$ (10) $$

where the approximation holds when an error term $\mathcal{E}$, capturing second-order effects, is small. The term $\text{Var}[r(x_{k},a)]$ is bounded by $1/4$ for normalized rewards.

The full proof is provided in Appendix [B](https://arxiv.org/html/2507.05386v3#A2). Proposition [5.2](https://arxiv.org/html/2507.05386v3#S5.Thmtheorem2) offers an intuition: the expected impact of an RFT update on prior knowledge is not fixed but is dynamically scaled by the reward variance. For an uncertain sample where the model generates diverse responses with high reward variance, the update magnitude in sensitive directions is naturally dampened, thus protecting established knowledge. Conversely, for samples where the model produces consistently high-reward responses (low variance), the update is more aggressive. This inherent, data-dependent regularization mechanism contrasts sharply with SFT’s uniform, high-variance gradients, offering a compelling explanation for the stability observed in our experiments and illustrated in Figure [4](https://arxiv.org/html/2507.05386v3#S5.F4).

###### Definition 5.1 (Forgetting Risk) .

###### Proposition 5.2 (RFT’s Implicit Regularization Effect) .

### 5.3 RIF-RFT: Enhancing Stability and Efficiency of RFT

Our analysis in Section [5.2](https://arxiv.org/html/2507.05386v3#S5.SS2) reveals that RFT’s resilience to forgetting is based in a reward-variance-scaled regularization. However, this mechanism’s effectiveness relies on the model’s ability to generate responses that produce a meaningful reward signal. We identify a critical failure mode when the model is faced with incompetent samples: training instances for which the current policy $\pi_{\theta}$ consistently fails to produce non-zero rewarded outputs. For such samples, the advantage estimates $A(x,a)$ are dominated by noise, leading to high-variance gradient estimates.

To address this challenge, we propose a simple yet effective method: Rollout-based Instance Filtering for RFT (RIF-RFT). The motivation, illustrated in Figure [4](https://arxiv.org/html/2507.05386v3#S5.F4), is to prune the training data by identifying and discarding these incompetent samples before the RFT training. By filtering them out, RFT focuses its capacity on instances where it can receive a productive learning signal, stabilizing the regularization effect and improving efficiency.

The mechanism is formalized in Algorithm [1](https://arxiv.org/html/2507.05386v3#alg1) in Appendix [C](https://arxiv.org/html/2507.05386v3#A3). For each instance in a new task’s dataset $\mathcal{D}_{k}$, we perform a small number of policy rollouts. If at least one of these rollouts produces a response with a reward greater than a minimal threshold $\tau$, we classify the instance and retain it in $\mathcal{D}_{k}^{\text{filt}}$.
As shown in Table [5](https://arxiv.org/html/2507.05386v3#S5.T5), RIF-RFT achieves competitive performance (57.5% of AvgAcc) and strong knowledge retention (-4.5% of FM), which is superior to SFT with a FM of -10.4%. Meanwhile, RIF-RFT use only 48.4% of training data. This demonstrates a compelling trade-off between efficiency and robustness.

Table: Table 5: Performance and data efficiency comparison of our proposed RIF-RFT.

## 6 Conclusion

This work presents a comprehensive investigation into the role of the fundamental learning paradigm in continual post-training for MLLMs. Our central finding is that RFT naturally mitigates the catastrophic forgetting that plagues the standard SFT. Through extensive experiments, we demonstrate that while SFT leads to severe degradation of both previously learned task-specific skills and general capabilities, RFT paradigms inherently preserve those knowledge, achieving performance comparable to an offline multi-task learning setting. Our analysis suggests this superiority stems not from explicit mechanisms like CoT or KL regularization, but from an implicit regularization effect inherent to RFT. We provide a theoretical perspective that attributes this stability to reward-variance-scaled updates, which naturally protect previously acquired knowledge by moderating learning on uncertain samples. Finally, we introduce RIF-RFT, an efficient instance filtering method that improves the stability and sample efficiency of RFT without compromising its robustness. This research suggests that RFT is not merely an alternative but a fundamentally more suitable paradigm for the continual and lifelong adaptation of foundation models.

## Ethics Statement

This research focuses on the fundamental learning paradigms for continual post-training of Multimodal Large Language Models. All experiments were conducted on publicly available and well-established academic benchmarks. Our work did not involve human subjects, private data, or generation of personally identifiable information.

## Reproducibility Statement

To ensure the reproducibility of our findings, we provide comprehensive details throughout the paper and in the appendix. The source code is available in the supplementary material. Our experiments are based on the publicly available Qwen2.5-VL-7B-Instruct model. All implementation details are documented in Section [4](https://arxiv.org/html/2507.05386v3#S4) and the Appendix. We use standard public datasets, with detailed descriptions provided in Section [4](https://arxiv.org/html/2507.05386v3#S4) and the Appendix.

## References

- Achiam et al. (2023)
Jacob Achiam, Benjamin Anthrop, Prafulla Bajaj, Andy Baumli, Tong Cai, Bowen Cheng, Debajyoti Chowdhury, Sreeram DasSarma, Emily Dinan, Jack Gao, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- Ahmadian et al. (2024)
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker.
Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.
*arXiv preprint arXiv:2402.14740*, 2024.
- Bai et al. (2025a)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report, 2025a.
URL [https://arxiv.org/abs/2502.13923](https://arxiv.org/abs/2502.13923).
- Bai et al. (2025b)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al.
Qwen2. 5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025b.
- Chen et al. (2024)
Cheng Chen, Junchen Zhu, Xu Luo, Heng T Shen, Jingkuan Song, and Lianli Gao.
Coin: A benchmark of continual instruction tuning for multimodel large language models.
*Advances in Neural Information Processing Systems*, 37:57817–57840, 2024.
- Chu et al. (2025)
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma.
Sft memorizes, rl generalizes: A comparative study of foundation model post-training.
*arXiv preprint arXiv:2501.17161*, 2025.
- Chung et al. (2024)
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.
Scaling instruction-finetuned language models.
*Journal of Machine Learning Research*, 25(70):1–53, 2024.
- Guo et al. (2025a)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025a.
- Guo et al. (2025b)
Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, and Cheng-Lin Liu.
Hide-llava: Hierarchical decoupling for continual instruction tuning of multimodal large language model.
In *The 63rd Annual Meeting of the Association for Computational Linguistics*, 2025b.
- Guo et al. (2025c)
Haiyang Guo, Fanhu Zeng, Fei Zhu, Jiayi Wang, Xukai Wang, Jingang Zhou, Hongbo Zhao, Wenzhuo Liu, Shijie Ma, Xu-Yao Zhang, et al.
A comprehensive survey on continual learning in generative models.
*arXiv preprint arXiv:2506.13045*, 2025c.
- Gurari et al. (2018)
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham.
Vizwiz grand challenge: Answering visual questions from blind people.
In *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2018.
- He et al. (2020)
Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie.
Pathvqa: 30000+ questions for medical visual question answering.
*arXiv preprint arXiv:2003.10286*, 2020.
- Hudson & Manning (2019)
Drew A. Hudson and Christopher D. Manning.
Gqa: A new dataset for real-world visual reasoning and compositional question answering, 2019.
URL [https://arxiv.org/abs/1902.09506](https://arxiv.org/abs/1902.09506).
- Kirkpatrick et al. (2017)
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.
Overcoming catastrophic forgetting in neural networks.
In *Proceedings of the national academy of sciences*, volume 114(13), pp. 3521–3526. National Acad Sciences, 2017.
- Kumar et al. (2025)
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan.
Llm post-training: A deep dive into reasoning large language models.
*arXiv preprint arXiv:2502.21321*, 2025.
- Lee et al. (2023)
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al.
Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback.
*arXiv preprint arXiv:2309.00267*, 2023.
- Lee et al. (2025)
Minjae Lee, Minhyuk Seo, Tingyu Qu, Tinne Tuytelaars, and Jonghyun Choi.
Oasis: Online sample selection for continual visual instruction tuning.
*arXiv preprint arXiv:2506.02011*, 2025.
- Li et al. (2025)
Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang.
Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning.
*arXiv preprint arXiv:2504.06958*, 2025.
- Li et al. (2023a)
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
*arXiv preprint arXiv:2305.10355*, 2023a.
- Li & Hoiem (2017)
Zhizhong Li and Derek Hoiem.
Learning without forgetting.
*IEEE transactions on pattern analysis and machine intelligence*, 40(12):2935–2947, 2017.
- Li et al. (2023b)
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille.
Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 14963–14973, 2023b.
- Li et al. (2023c)
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo.
Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models.
*arXiv preprint arXiv:2310.10505*, 2023c.
- Liu et al. (2024)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 26296–26306, 2024.
- Liu et al. (2025a)
Wenzhuo Liu, Fei Zhu, Haiyang Guo, Longhui Wei, and Cheng-Lin Liu.
Llava-c: Continual improved visual instruction tuning.
*arXiv preprint arXiv:2506.08666*, 2025a.
- Liu et al. (2025b)
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin.
Understanding r1-zero-like training: A critical perspective.
*arXiv preprint arXiv:2503.20783*, 2025b.
- Liu et al. (2025c)
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang.
Visual-rft: Visual reinforcement fine-tuning.
*arXiv preprint arXiv:2503.01785*, 2025c.
- Lu et al. (2021)
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu.
Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning.
*arXiv preprint arXiv:2105.04165*, 2021.
- Lu et al. (2024)
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=KUNzEQMWU7](https://openreview.net/forum?id=KUNzEQMWU7).
- Luong et al. (2024)
Trung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, and Hang Li.
Reft: Reasoning with reinforced fine-tuning.
*arXiv preprint arXiv:2401.08967*, 3:2, 2024.
- Maharana et al. (2025)
Adyasha Maharana, Jaehong Yoon, Tianlong Chen, and Mohit Bansal.
Adapt-infty: Scalable continual multimodal instruction tuning via dynamic data selection.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- McCloskey & Cohen (1989)
Michael McCloskey and Neal J Cohen.
Catastrophic interference in connectionist networks: The sequential learning problem.
*The psychology of learning and motivation*, 24:109–165, 1989.
- Ouyang et al. (2022)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
Training language models to follow instructions with human feedback.
*Advances in Neural Information Processing Systems*, 35:27730–27744, 2022.
- Saikh et al. (2022)
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya.
Scienceqa: A novel resource for question answering on scholarly articles.
*International Journal on Digital Libraries*, 23(3):289–301, 2022.
- Shao et al. (2024)
Jing Shao, Kun Li, Wentao Dong, et al.
Deepseekmath: A multimodal multitask benchmark for mathematical reasoning.
*arXiv preprint arXiv:2406.01297*, 2024.
- Sheng et al. (2024)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu.
Hybridflow: A flexible and efficient rlhf framework.
*arXiv preprint arXiv: 2409.19256*, 2024.
- Singh et al. (2019)
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.
Towards vqa models that can read.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 8317–8326, 2019.
- Van de Ven et al. (2022)
Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias.
Three types of incremental learning.
*Nature Machine Intelligence*, 4(12):1185–1197, 2022.
- Wang et al. (2024a)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024a.
- Wang et al. (2025)
Xingjin Wang, Howe Tissue, Lu Wang, Linjing Li, and Daniel Dajun Zeng.
Learning dynamics in continual pre-training for large language models.
In *Forty-second International Conference on Machine Learning*, 2025.
- Wang et al. (2024b)
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen.
Mmlu-pro: A more robust and challenging multi-task language understanding benchmark, 2024b.
URL [https://arxiv.org/abs/2406.01574](https://arxiv.org/abs/2406.01574).
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Williams (1992)
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning.
*Machine learning*, 8:229–256, 1992.
- Yue et al. (2024)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
In *Proceedings of CVPR*, 2024.
- Zeng et al. (2024)
Fanhu Zeng, Fei Zhu, Haiyang Guo, Xu-Yao Zhang, and Cheng-Lin Liu.
Modalprompt: Dual-modality guided prompt for continual learning of large multimodal models.
*arXiv preprint arXiv:2410.05849*, 2024.
- Zhai et al. (2024)
Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, et al.
Fine-tuning large vision-language models as decision-making agents via reinforcement learning.
*Advances in neural information processing systems*, 37:110935–110971, 2024.
- Zhao et al. (2025)
Hongbo Zhao, Fei Zhu, Rundong Wang, Gaofeng Meng, and Zhaoxiang Zhang.
Mllm-cl: Continual learning for multimodal large language models.
*arXiv preprint arXiv:2506.05453*, 2025.
- Zheng et al. (2024)
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma.
Llamafactory: Unified efficient fine-tuning of 100+ language models, 2024.
URL [https://arxiv.org/abs/2403.13372](https://arxiv.org/abs/2403.13372).
- Zheng et al. (2025)
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong.
Easyr1: An efficient, scalable, multi-modality rl training framework.
[https://github.com/hiyouga/EasyR1](https://github.com/hiyouga/EasyR1), 2025.
- Zhou et al. (2023)
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.
Lima: Less is more for alignment.
*Advances in Neural Information Processing Systems*, 36:55006–55021, 2023.
- Zhu et al. (2024)
Fei Zhu, Shijie Ma, Zhen Cheng, Xu-Yao Zhang, Zhaoxiang Zhang, and Cheng-Lin Liu.
Open-world machine learning: A review and new outlooks.
*arXiv preprint arXiv:2403.01759*, 2024.

## Appendix A Dataset Information

Figure: Figure 5: Example prompt templates w/o and w/ CoT.

#### Multimodal Datasets for Continual Post-Training and Evaluation.

Our study utilized a diverse suite of vision-language datasets for both model training and comprehensive evaluation of various multimodal capabilities, along with specialized benchmarks to assess knowledge retention and nuanced multimodal challenges. Here is a brief introduction to these datasets:

*Multimodal Datasets for Continual Post-Training:*

- •
ScienceQA (Saikh et al., 2022) presents multimodal science questions requiring complex reasoning over diagrams, text, and general knowledge.
- •
TextVQA (Singh et al., 2019) focuses on questions that necessitate reading and inferring from text embedded within images.
- •
VizWiz (Gurari et al., 2018) comprises real-world image-based questions posed by visually impaired individuals, often involving ambiguity.
- •
GQA (Hudson & Manning, 2019) is designed for compositional question answering over real-world images with a strong emphasis on spatial understanding and object relationships.
- •
Geometry3K (Lu et al., 2021): This subset of MathVista (Lu et al., 2024) comprises multi-choice geometry problems equipped with dense annotations in formal language for both diagrams and text, specifically designed to evaluate complex geometric reasoning skills.
- •
PathVQA (He et al., 2020) provides medical visual question answering on pathology images that demand specialized domain knowledge.
- •
Super-CLEVR (Li et al., 2023b) is a synthetic dataset crafted to rigorously test complex relational and logical reasoning.

*Benchmarks for General Knowledge Evaluation:*

- •
MMMU (Yue et al., 2024) is comprehensive benchmark comprising 11.5K college-level, multi-discipline multimodal tasks with diverse image types, demanding deliberate reasoning.
- •
MMLU-Pro (Wang et al., 2024b) is an enhanced benchmark designed for more discriminative evaluation of large language models, featuring more challenging and reasoning-focused questions with ten multiple-choice options, sourced from various academic and STEM fields.
- •
POPE (Li et al., 2023a) is a benchmark introduced to systematically investigate and assess object hallucination in vision-language large models through an improved polling-based query method.

## Appendix B Proof and Technical Details for Proposition 5.2

We provide the detailed derivation for Proposition [5.2](https://arxiv.org/html/2507.05386v3#S5.Thmtheorem2), which establishes the relationship between the forgetting risks of RFT and SFT.

#### Proposition 5.2 .

Let the rewards be normalized, $r(x_{k},a)\in[0,1]$. Under Assumption [B.2](https://arxiv.org/html/2507.05386v3#A2.Thmtheorem2), the expected forgetting risk of an RFT update is related to the SFT risk by:

$$ $\mathbb{E}_{a\sim\pi_{\theta_{k-1}}}[\mathcal{R}(g_{\text{RFT}}(a))]=\text{Var}_{a\sim\pi_{\theta_{k-1}}}[r(x_{k},a)]\cdot\mathcal{R}(g_{\text{SFT}})+\mathcal{E}$ $$

where $\mathcal{E}$ is an error term characterized in the proof.

###### Definition B.1 (Importance-Weighted Score Norm (IWSN)) .

For a response $a$, we define its IWSN as the squared norm of its score function, weighted by the FIM of past tasks:

$$ $I(a)\triangleq(\nabla_{\theta}\log\pi_{\theta}(a|x_{k}))^{\top}F_{k-1}(\nabla_{\theta}\log\pi_{\theta}(a|x_{k}))$ $$

###### Assumption B.2 (Technical Assumptions) .

Our analysis relies on the following two technical assumptions for a given data point $x_{k}$ and parameters $\theta_{k-1}$:

- 1.
Bounded Covariance. The covariance between the squared advantage and the IWSN is bounded: $\text{Cov}(A(a)^{2},I(a))=\epsilon_{1}$, where $\epsilon_{1}$ is a small error term. This implies that the magnitude of an advantage signal is not strongly correlated with the gradient’s impact on prior tasks.
- 2.
Centered Policy Expectation. The expected IWSN under the current policy is close to the IWSN of the ground-truth response: $\mathbb{E}_{a\sim\pi_{\theta_{k-1}}}[I(a)]-I(a_{k}^{*})=\delta$, where $\delta$ is another small error term. This holds when the policy $\pi_{\theta_{k-1}}$ generates responses that, on average, have a similar gradient geometry to the ground-truth response with respect to past tasks.

###### Lemma B.3 (Variance of Advantage) .

For policy gradient methods, using the reward baseline $b(x_{k})=\mathbb{E}_{a\sim\pi_{\theta}}[r(x_{k},a)]$ minimizes the variance of the gradient estimator. With this optimal baseline, the expected squared advantage equals the reward variance:

$$ $\mathbb{E}_{a\sim\pi_{\theta}}[A(x_{k},a)^{2}]=\text{Var}_{a\sim\pi_{\theta}}[r(x_{k},a)]$ $$

###### Proof.

By definition, $A(x_{k},a)=r(x_{k},a)-b(x_{k})$. We have:

$$ $\displaystyle\mathbb{E}[A^{2}]$ $\displaystyle=\mathbb{E}[(r-b)^{2}]=\mathbb{E}[r^{2}]-2b\mathbb{E}[r]+b^{2}$ $$

Since $b=\mathbb{E}[r]$, this simplifies to $\mathbb{E}[r^{2}]-2(\mathbb{E}[r])^{2}+(\mathbb{E}[r])^{2}=\mathbb{E}[r^{2}]-(\mathbb{E}[r])^{2}=\text{Var}[r]$
∎

###### Proof of Proposition 5.2 .

Let us analyze the forgetting risks at parameters $\theta=\theta_{k-1}$ for a single data point $(x_{k},a_{k}^{*})$.

The SFT loss gradient is $g_{\text{SFT}}=-\nabla_{\theta}\log\pi_{\theta}(a_{k}^{*}|x_{k})$. Its forgetting risk is deterministic:

$$ $\mathcal{R}(g_{\text{SFT}})=g_{\text{SFT}}^{\top}F_{k-1}g_{\text{SFT}}=(\nabla_{\theta}\log\pi_{\theta}(a_{k}^{*}|x_{k}))^{\top}F_{k-1}(\nabla_{\theta}\log\pi_{\theta}(a_{k}^{*}|x_{k}))=I(a_{k}^{*})$ $$

The RFT gradient for a sampled response $a\sim\pi_{\theta}(\cdot|x_{k})$ is $g_{\text{RFT}}(a)=A(x_{k},a)\nabla_{\theta}\log\pi_{\theta}(a|x_{k})$. We compute the expectation of its forgetting risk:

$$ $\displaystyle\mathbb{E}[\mathcal{R}(g_{\text{RFT}})]$ $\displaystyle=\mathbb{E}_{a\sim\pi_{\theta}}\left[(g_{\text{RFT}}(a))^{\top}F_{k-1}(g_{\text{RFT}}(a))\right]$ $\displaystyle=\mathbb{E}_{a\sim\pi_{\theta}}\left[A(x_{k},a)^{2}\cdot(\nabla_{\theta}\log\pi_{\theta}(a|x_{k}))^{\top}F_{k-1}(\nabla_{\theta}\log\pi_{\theta}(a|x_{k}))\right]$ $\displaystyle=\mathbb{E}_{a\sim\pi_{\theta}}\left[A(x_{k},a)^{2}\cdot I(a)\right]$ (11) $$

We use the covariance identity $\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]+\text{Cov}(X,Y)$ to decompose Eq. [11](https://arxiv.org/html/2507.05386v3#A2.E11):

$$ $\displaystyle\mathbb{E}[\mathcal{R}(g_{\text{RFT}})]$ $\displaystyle=\mathbb{E}[A(a)^{2}]\cdot\mathbb{E}[I(a)]+\text{Cov}(A(a)^{2},I(a))$ $\displaystyle=\text{Var}[r(x_{k},a)]\cdot\mathbb{E}[I(a)]+\epsilon_{1}\quad(\text{from Lemma \ref{lemma:advantage_variance} and Assumption \ref{assump:technical}.1})$ $\displaystyle=\text{Var}[r(x_{k},a)]\cdot(I(a_{k}^{*})+\delta)+\epsilon_{1}\quad(\text{from Assumption \ref{assump:technical}.2})$ $\displaystyle=\text{Var}[r(x_{k},a)]\cdot I(a_{k}^{*})+\text{Var}[r(x_{k},a)]\delta+\epsilon_{1}$ $\displaystyle=\text{Var}[r(x_{k},a)]\cdot\mathcal{R}(g_{\text{SFT}})+\underbrace{\text{Var}[r(x_{k},a)]\delta+\epsilon_{1}}_{\mathcal{E}}$ $$

This completes the proof. The error term $\mathcal{E}=\text{Var}[r(x_{k},a)]\delta+\epsilon_{1}$ is small under reasonable conditions. Specifically, $\delta$ is small when the current policy is not drastically different from one that produces the ground-truth response, a condition met after initial task adaptation. $\epsilon_{1}$ is small if there is no systematic correlation between a response’s quality (reflected in its advantage) and its gradient’s impact on prior tasks, which is a mild assumption for complex, high-dimensional models. While this analysis provides an approximation rather than a strict bound, it formalizes the core intuition that reward variance acts as a natural, implicit regularizer, offering a strong theoretical motivation for the empirical stability of RFT in continual post-training.
∎

###### Definition B.1 (Importance-Weighted Score Norm (IWSN)) .

###### Assumption B.2 (Technical Assumptions) .

###### Lemma B.3 (Variance of Advantage) .

###### Proof.

###### Proof of Proposition 5.2 .

## Appendix C Pseudo Code of RIF-RFT

Figure: Algorithm 1 Rollout-based Instance Filtering for RFT (RIF-RFT)

## Appendix D Statement on the Use of LLM

We disclose that LLMs were used to assist with writing. Their role was limited to improving the grammar and overall readability of the manuscript. The core research ideas, experimental results, and scientific claims presented are the work of the authors.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: Generality to Other Settings: How well do you expect the advantages of RFT to transfer to other model types and domains? For example, have you considered or attempted applying RFT-based continual learning on a pure text large language model, or a smaller model with less prior knowledge? It would be insightful to know if RFT’s implicit regularization still shines in those cases, or if there are any domains where SFT might remain competitive. Reward Function Design: Could you clarify how the reward signals were defined for your tasks, and whether any reward shaping or learned reward models were needed? Since the tasks seem to have ground-truth answers (e.g. classification or QA), was the reward simply 1 for a correct answer and 0 for incorrect, or something more nuanced (partial credit, etc.)? Understanding this would help gauge how easily RFT can be applied to new tasks. Additionally, if a task had a more subjective or continuous evaluation metric, do you foresee any challenges in applying RFT there? RFT in Low-Performance Regimes: How does RFT perform when the model’s initial success on a new task is very low, and how effective is RIF in that scenario? The paper introduces RIF to handle “incompetent” samples – for a new task where the model almost never produces a correct output initially, RIF would filter out many samples. Does this risk filtering out too much data and hindering learning new knowledge? In other words, is there a point at which a task is so difficult that a bit of supervised signal might be needed to kickstart learning before RFT takes over? Any discussion on this or empirical observation (even anecdotal) would be appreciated to understand the limits of RFT’s applicability. Combining RFT with Other CL Techniques: Have you considered if combining RFT with traditional continual learning techniques (like a small experience replay buffer or weight regularization) would further improve performance, or is RFT alone already near-optimal? Your results suggest RFT alone is as good as multi-task training, which is remarkable. But for completeness, it would be interesting to know if adding even minor rehearsal of old tasks (or other CL tricks) on top of RFT yields any marginal gains or if it’s unnecessary. This could help practitioners decide how to allocate resources (e.g., focus purely on RFT or also maintain some memory of past data). (Feel free to address these during the rebuttal if time permits – they are mostly aimed at understanding the breadth of applicability and practical considerations of the proposed approach.)
soundness: 3
strengths: Originality & Importance: The paper addresses a previously overlooked question – whether reinforcement learning-based fine-tuning could fundamentally improve continual learning – making it a fresh and notable contribution. This exploration of the fine-tuning paradigm itself (as opposed to add-on techniques) is highly original. It targets the important problem of catastrophic forgetting in lifelong learning for foundation models, which has broad relevance for deployed AI systems. Empirical Results – Effectiveness of RFT: The experimental results are impressive and persuasive. RFT nearly eliminates catastrophic forgetting on a diverse set of seven tasks, matching the performance of multi-task training without using any memory or regularization tricks. It also preserves general knowledge and even slightly improves overall capabilities of the base model – a striking outcome since typical fine-tuning often degrades general performance. The magnitude of improvement of RFT over SFT (e.g. retaining old task accuracy ~95% vs dropping to ~70% for SFT, in some cases) demonstrates clear superiority. These strong empirical findings support the paper’s claims and indicate high practical utility. Theoretical Rigor and Insight: The authors provide a sound theoretical explanation for RFT’s advantages. They introduce a forgetting risk metric and prove a bound (Theorem 5.2) showing RFT’s expected forgetting risk is scaled down by reward variance compared to SFT. This insight reveals an implicit regularization effect: RFT’s stochastic policy updates naturally avoid large detrimental shifts in parameters important to past tasks. The theory is novel and aligns with the empirical observations, adding significant credibility and understanding to the results. Comprehensive Analysis and Ablations: The paper is thorough in its analysis. It evaluates multiple RFT algorithms (e.g. GRPO, ReMax, etc.) and ablates key factors: for instance, showing that removing the KL penalty or chain-of-thought does not erase RFT’s forgetting mitigation. This rules out alternative explanations and solidifies that the benefit comes from the core RFT mechanism. The authors also measure both task-specific retention and general knowledge retention, providing a holistic view of the model’s performance. The inclusion of RIF-RFT with results (Table 5) demonstrates the authors’ depth of understanding: they not only identified RFT’s strength but also tackled its weakness (instability on some samples) with a validated solution. Overall, the experimental section leaves little doubt about the conclusions due to its breadth and careful design. Clarity and Presentation: The paper is well-written and organized, making it easy to follow the complex subject matter. The motivation is clearly stated, related work is well-situated, and the contributions are clearly itemized. Figures and tables are effectively used to illustrate key points (e.g. a graph showing SFT vs RFT forgetting dynamics), which aids understanding. The clarity of presentation means the ideas and results are accessible to readers, increasing the impact of the work. Relevance and Impact: The work has strong relevance to multiple areas: continual learning, large-scale model fine-tuning, and reinforcement learning. Its findings are likely to influence future research and practice, as they suggest a paradigm shift (incorporating RL objectives for better retention). The method is also practical – e.g., RIF makes RFT feasible by reducing data needs – which could encourage real-world adoption. The paper thus excels not only academically (with insight and rigor) but also in potential real-world impact, aligning well with ICLR’s interests in advancing learnability of AI systems in dynamic settings.
summary: This paper presents a compelling study on continual post-training (CPT) of large foundation models, comparing two fine-tuning paradigms: supervised fine-tuning (SFT) versus reinforcement fine-tuning (RFT). The authors conduct extensive experiments on a sequence of seven diverse multimodal tasks using a 7B-parameter vision-language model (Qwen2.5-VL-7B-Instruct) as the base. The key finding is that RFT dramatically outperforms SFT in mitigating catastrophic forgetting: when tasks are learned one after another, SFT suffers severe forgetting of earlier tasks, whereas RFT inherently preserves prior task knowledge, achieving performance on old tasks comparable to an ideal multi-task training baseline. Remarkably, RFT even maintains or improves the model’s general knowledge (evaluated on broad benchmarks like MMMU and MMLU-Pro), whereas SFT significantly degrades general capabilities. The study identifies that this superior knowledge retention of RFT is not primarily due to common tricks like KL-divergence regularization or chain-of-thought prompting, but rather stems from an implicit regularization effect in the RFT paradigm itself. To address the practical challenge that vanilla RFT can be less sample-efficient and stable, the authors propose a simple yet effective addition: a Rollout-based Instance Filtering (RIF) algorithm. RIF prunes “incompetent” training instances (those where the current policy fails to produce any rewarding output) before applying RFT, which significantly improves training stability and efficiency without sacrificing performance. Overall, the paper’s contributions include: (1) the first comprehensive analysis demonstrating RFT’s natural advantage in preserving both task-specific and general knowledge during continual fine-tuning, (2) a theoretical insight that RFT’s use of reward signals induces an implicit, data-dependent regularization that mitigates forgetting better than explicit techniques, and (3) the introduction of the RIF-RFT algorithm to enhance RFT’s efficiency while retaining its robustness to forgetting
weaknesses: Computational Complexity: RFT requires rollouts and policy optimization, making it more computationally intensive and sensitive to hyperparameters than standard SFT. While RIF improves efficiency, real-world deployments may still face engineering challenges. Evaluation Scope: The study focuses on one 7B multimodal model and QA-style tasks. Broader validation across domains, tasks, and model scales would strengthen the generality claims. Reward Dependence: RFT assumes access to clear reward signals. In tasks lacking binary correctness (e.g., subjective or open-ended outputs), reward design could be non-trivial, reducing RFT’s immediate applicability. Initial Model Competence: RFT performs best when the base model has some proficiency on new tasks. In cold-start scenarios with zero initial success, the reward sparsity could limit RFT’s learning. RIF helps mitigate this but may discard too much data in such cases. Limited Baseline Comparisons: While the contrast to vanilla SFT is strong, the paper could benefit from including other continual learning baselines (e.g., SFT with replay or regularization) to further isolate the benefit of the RFT paradigm. These limitations are either practical considerations or natural directions for future research and do not detract from the core strength of the paper.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The related-work discussion can be expanded. For example, [1] also studies SFT vs. RFT and suggests that SFT "directly providing answers to new tasks, without linking them to the model’s existing perceptual abilities through reasoning trajectories, causes the output distribution to shift abruptly". This seems at odds with the paper’s claim that CoT does not mitigate catastrophic forgetting. Could the authors elaborate on the differences between [1] and their work, and clarify the potential contradiction between them?

2. Typo in Eq. (4): the second "$a$" should denote the greedily sampled response and should be a different symbol (e.g., "$\hat{a}$").


[1] Zhang, Z., Dong, Q., Zhang, Q., Zhao, J., Zhou, E., Xi, Z., Jin, S., Fan, X., Zhou, Y., Wu, M., Fu, Y., Ji, T., Gui, T., Huang, X., & Chen, K. (2025, June 30). Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
soundness: 2
strengths: 1. This paper presents an interesting theoretical claim: RFT mitigates catastrophic forgetting via implicit gradient regularization driven by reward variance.

2. It offers solid experiments supporting that RFT reduces forgetting, and argues the effect is not attributable to KL constraints or chain-of-thought (CoT).

3. The paper is clear writing and overall easy to follow.
summary: This paper investigates forgetting in two mainstream post-training paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). The authors report that SFT can induce catastrophic forgetting, whereas RFT tends to preserve -- and sometimes even enhance -- general knowledge. They further propose that RFT benefits from an implicit regularization effect in which gradient updates are scaled by reward variance. Building on this, they introduce a rollout-based instance filtering algorithm to improve the stability and efficiency of RFT.
weaknesses: 1. My primary concern is, the central theorem lacks convincing empirical support. The proposed rollout-based instance filtering method (RIF-RFT), motivated by the theorem, should improve anti-forgetting via lower reward variance under the theory, yet it underperforms the baseline RFT. The authors note that filtering reduces training data, but this raises a fairness concern: why not match the effective data budget across RIF-RFT and baseline RFT to isolate the effect of variance reduction?

2. The efficiency claim for the filtering method needs stronger evidence. While filtering may yield higher-quality data for subsequent RFT, it introduces a nontrivial overhead from sampling answers from the base LLM -- often a bottleneck for methods like GRPO. We need more solid proof of this method’s efficiency.

I am willing to increase my score if the authors can address the concerns mentioned above.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. There is a correlation between continual learning performance and task order; can the model's forgetting resistance effect be verified under different task orders?
2. Only models of the 7B scale are verified in the paper; can the impact of model parameter scale on the forgetting mitigation effect of RFT be explored?
3. The experiment focuses on multimodal models: what is the degree of protection provided by RFT for the knowledge of different modalities, and is there any preference?
soundness: 3
strengths: 1. This paper reveals the inherent forgetting mitigation property in the reinforcement fine-tuning (RFT) paradigm, which endows RFT with superiority in post-training.
2. The mechanism analysis of forgetting mitigation under RFT is in-depth and supported by theories.
summary: This paper focuses on the catastrophic forgetting problem in the continual post-training (CPT) of multimodal large language models (MLLMs) and conducts a systematic comparison of the knowledge retention capabilities between the two paradigms: supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Through extensive experiments, it verifies that RFT has an inherent forgetting mitigation advantage, reveals the implicit regularization mechanism driven by reward variance, and finally proposes the Rollout-based Instance Filtering for RFT (RIF-RFT) algorithm to enhance the stability and efficiency of RFT.
weaknesses: 1. Experiments are only conducted based on Qwen2.5-VL-7B-Instruct, making it impossible to confirm whether the forgetting mitigation advantage of RFT is independent of the architecture.
2. There is a lack of direct comparisons with mainstream continual learning methods in terms of knowledge retention performance, storage overhead, and computational efficiency.
3. The selection method of the filtering threshold in RIF-RFT is not provided, and relevant ablation experiments are lacking.