Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
Authors: Zhihao Zhang
Qiaole Dong
Qi Zhang
Enyu Zhou
Jun Zhao
Zhiheng Xi
Senjie Jin
Xiaoran Fan
Yuhao Zhou
Mingqi Wu
Yanwei Fu
Tao Ji
Tao Gui
Xuanjing Huang
Kai Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Fine-Tuning
Catastrophic Forgetting
Data Distribution
Learning Dynamics
Primary area: interpretability and explainable AI

Abstract: Post-training algorithms such as Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) are widely used to adapt (multimodal) large language models to downstream tasks. While effective at task adaptation, their impact on retaining prior knowledge remains unclear. In this paper, we introduce jigsaw puzzles as a novel task absent from existing pretraining corpora and systematically study the behavior of SFT and RFT on the open-source Qwen2.5-VL series. Our experiments reveal a sharp trade-off: SFT enables rapid task acquisition but leads to catastrophic forgetting, whereas RFT learns more slowly but better maintains prior knowledge. We study this phenomenon through learning dynamics by examining both the magnitude and direction of how training data influence prior knowledge. Our analysis shows that RFT mainly reinforces correct samples naturally aligned with the base model’s probability landscape, leading to weaker interference with prior knowledge. Moreover, training on RFT-simulated rollouts, which exert a smaller magnitude of influence and are better aligned in direction to prior knowledge, allows SFT to preserve prior knowledge better while rapidly learning new tasks. We further validate our framework on Qwen2.5 post-training in math and scientific QA, observing consistent forgetting and learning-dynamics trends. These findings suggest that the distribution of post-training data, rather than algorithmic differences alone, plays a central role in forgetting, and highlight RFT as a promising ingredient for stable continual post-training.

Published as a conference paper at ICLR 2026
WHY REINFORCEMENT FINE-TUNING PRESERVES PRIOR KNOWLEDGE BETTER: A DATA PERSPECTIVE
Zhihao Zhang Qiaole Dong' Qi Zhang
Enyu Zhou', Jun Zhao', Zhiheng Xi², Senjie Jin', Xiaoran Fan", Yuhao Zhou' Mingqi Wu', Yanwei Fu', Tao Ji Tao Guil.,, Xuanjing Huangl.3i Kai Chen?!
Fudan University Shanghai Artificial Intelligence Laboratory
3 Shanghai Collaborative Innovation Center of Intelligent Visual Computing (zhangzhihaol,, gldong18, qz ] @udann. edu. en
ABSTRACT
Post-training algorithms such as Supervised Fine-Tuning (SFT) and Reinforce- ment Fine-Tuunig (RFT) are widely used to adapt (multimodal) large language models to downstream tasks. While effective at task adaptation, their impact on retaining prior knowledge remains unclear. In this paper, we introduce jigsaw puzzles as a novel task absent from existing pretraining corpora and systemati- cally study the behavior of SFT and RFT on the open-source Qwen2.5-VL. series. Our experiments reveal a sharp trade-off: SFT enables rapid task acquisition but leads catastrophic forgetting, whereas RFT learns more slowly but better main- tains prior knowledge. We study this phenomenon through learning dynamics by examining both the magnitude and direction of how training data influence prior knowledge. Our analysis shows that RFT mainly reinforces correct samples naturally aligned with the base model's probability landscape, leading to weaker interference with prior knowledge. Moreover, training on RFT-simulated rollouts, which exert a smaller magnitude of influence and are better aligned in direction to prior knowledge, allows SFT to preserve prior knowledge better while rapidly learning new tasks. We further validate our framework on Qwen2.5 post-training in math and scientific QA, observing consistent forgetting and barming-dynamics trends. These findings suggest that the distribution of post- post-training data, rather than algorithmic differences alone, plays central role in forgetting, and highlight RFT as a promising ingredient for stable continual post-training.
INTRODUCTION
In the era of large models, two primary post-training methods. i.e., Supervised Fine-Tuning (SFT) (Wei et al. 202T) and Reinforcement Fine-Tuning (RFT) (DeepSeek-Al et al. 2025 Ouyang et al. 2022), have emerged for enhancing model performance on domain-specife tasks. These meth- ods have been pivotal in enabling (multimodal) large language models to learn specific downstream tasks, follow human instructions, and acquire reasoning capabilities, yielding impressive results. However, current practices primarily focus on performance improvement for specific downstream tasks, while overlooking potential impact of fine-tuning algorithms on model's pre-existing knowl- edge. This oversight raises concerns about the model's 's ability to retain and apply prior knowledge.
To this end, this paper investigates how post-training algorithms, specifically SFT and RFT, affect the retention of prior knowledge when large models are trained to learn entirely novel knowledge or tasks that were absent during pretraining. To establish a challenging and genuinely novel task for testing, we introduce jigsaw puzzles as the target task for learning, as in Fig. [ Through pre- liminary experiments, we observe that existing state-of-the-art MI.MMs including GPT-4o (Hurst et al., 2024). fail to solve even simple 3x3 jigsaw puzzles, indicating that this task represents a novel problem not covered by current pretraining corpora. Thus, jigsaw puzzles can serve as fair and meaningful task for evaluating impact of post-training algorithms on prior knowledge.
Equal contributions.
Corresponding author.

---

Published as a conference paper at ICLR 2026
Jigsaw Grounding
No Thinking:
6, 8,
Split
Shuffle
T Jigsaw
Grounding
de Thinking <ithink> Canswers
SubPicture: 0: <image <image 8> Please arrange SubPicture: 8: (numbered 0 Provide the correct sequence of the sub-images in order from left to right and top to bottom.
12,1,,,,,,444,,,
sub-images complete image.
<lanswer>
thinkk>
T Jigsaw
Grounding
GRPO Thinking -hinle
(Optionaf) First output the thinking process in <think> <ithink> tags, then output answer <ansmer> slanwerr tags.
Sanswers
12 121,6,4,,,
Figure 1: Overview of jigsaw puzzles in the context of MLLMs. We split the original image into 9 patches and randomly shuffle the order of the patches. During SFT, MLLMs are supervised ei- ther with Non-Reasoning data directly or GPT-40-geeerattee catastrophic forgetting. In contrast, RFT generates reasoning trajectories and answers by itself, re- inforces the correct outputs, and avoids severe forgetting.
reasoning trajectories, while both incur
We conduct systematic fine-tuning experiments using both standard SFT and RFT, i.e. GRPO (Shao ct al. 2024), on open-sourced Qwen2.5-Ve VL [Bai et al., 2025) series. Interestingly, we find that SFT master novel tasks with solely hundreds or thousands of training steps, while RFT requires several tens of thousands of training steps to successfully solve jigsaw puzzles and achieves similar accuracy as SFT. This finding suggests that large-scale RFT can teach model solve tasks that base model is completely unable to handle. In sense, this shows that RFT can push the model beyond its original capability boundary. In addition performance on novel tasks, we observe that SFT incurs severe forgetting of previous knowledge, with substantial performance drops across diverse benchmarks, especially on tasks with similar output formats as jigsaw puzzles. In contrast, RFT, by leveraging reward-drig -drvven credit assignments for the simulated rollouts, can master the novel jigsaw puzzles while maintaining decent performance on prior tasks.
The distinct phenomenon between SFT and RFT naturally raises a question: Why SFT incurs catas- trophic forgetting while RFT does not? While both algorithms increase likelihood of correct re- sponses, RFT adaptively reweights likelihood of rollout with reward, whereas SFT uniformly in- creases likelihood of static human annotations. Inspired by this, we collect correct rollouts during RFT and use them as supervised data for SFT. Surprisingly, SFT trained these correct rollouts not only acquires novel knowledge quickly but also preserves prior knowledge better. This suggests that construction of fine-tuning data, rather than training algorithm, is key factor in knowledge forgetting.
Furthermore, we provide a new perspective based on learning dynamics (Ren & Sutherland 2024), which links the likelihood change of prior knowledge H. the gradient induced by an individual training example Ta, on understanding this distinct forgetting behavior by analyzing the magnitude and direction of how training data influence prior knowledge. We first observe that SFT data without reasoning trajectories usually interfere more with prior knowledge, as verified with a much larger norm of empirical neural tangent kernel (cNTK) between SFT data and prior knowledge. While datasets with reasoning trajectories, such as reasoming trajectories generated by GPT-4o and col- lected during RFT. usually exhibit smaller norm of eNTK and less forgetting of prior knowledge, implying that introducing explicit reasoning can help alleviate knowledge forgetting.
As for the direction of interference, we find that data with reasoning trajectories generated by GPT- 4o typically belong to high-perplexity regions of the base model. In contrast, data collected during RFT are naturally generated from regions where the base model already assigns a moderate proba- bility. This suggests that pretraining has already shaped certain linguistic regions by accident that are alll-aligned with novel tasks, while remaining compatible with prior knowledge. Importantly, according learning dynamics, the influence of training of another example x, is symmetric: increasing the likelihood of x, has the same marginal effect on F. increasing H. has F.. Therefore, when RFT discovers and reinforces such hidden lin-
example Va the likelihood
2

---

Published as a conference paper at ICLR 2026
guistic regions Tu shaped during pre-training, it degrades less the likelihood of prior knowledge Ty. Crucially, such regions are difficult to identify during the stage of dataset construction for SFT, but accessible through RFT's active exploration within linguistic space. This highlights RFT as an effective algorithm for stable novel knowledge acquisition without suffering from catastrophic forgetting. Further, consistent forgetting and learning-dynamin results. Formally, our contributions are three-fold:
experiments Qwen2.5 post-training in math and scientific QA show
We show that RFT can solve unseen tasks while preserving prior competencies. Moreover, SFT trained on RFT-generated rollouts can match RFT's performance while markedly reducing catastrophic forgetting, underscoring the central role of data construction in post-training.
We propose a influence prior knowledge into its magnitude and direction, providing a principled view of inter- ference and informing fine-tuning design.
interpretation of forgetting that decomposes how training data
Building on this interpretation, we conduct extensive experiments demonstrating that RL- sampled corpora strike favorable magnituee-direction trade-off, offering strong empirical sup- port for the stability of RL algorithms.
2 RELATED WORKS
Jigsaw Puzzles. Jigsaw puzzles has long been popular self-supervised task in the computer vision community, aimed at learning visual representations (Noroozi Favaro, 2016, Carlucci al. 2019) by spatial reasoning and part-whole understanding. Recently, this task has been repurposed for probing weak spot of MLLMs: Lyu et al. (2025) shows that leading MLLMs perform far behind than human performance. The contemporary work Jigsaw-R1 (Wang al. 2025 solves jigsaw puzzles with RFT, achieving much better performance. Collectively, these works mainly treat jigsaw puzzles pretext task for representation learning or test benchmark for MILMs. However, we employ jigsaw puzzles to investigate how post-training algorithms affect the forgetting behavior of MLLMs.
Reinforcement Fine-Tunine in MLLMs. Inspired by the success of RFT in large language mod- els (DeepSeek-AA et al.) 2025; Ouyang et al. 2022), recent work has applied RFT to MILMs. Among them, [Meng et (2025) finds that RFT can achieve better out-of-distribution generaliza- tion performance than SFT. Meanwhile, RFT is also employed for perception-centre tasks (Liu al. 2025c Shen et al. 2025 Liu al. 20256), still conferring notable gains in generalization and robustness. Concurrently, Jigsaw-RI (Wang et al. 2025) introduced RFT to the novel task of jigsaw puzzles but achieved limited accuracy. Building on this direction, we extend RFT training tens of thousands of steps to enable deeper exploration, yielding substantial gains on jigsaw puzzles.
Catastrophic Forgetting. Early work (McCloskey Cohen 1989 Ratclifi 1990) showed that even minimal sequential training on disjoint data can cause rapid "catastrophic forgetting" (CF). Existing strategies to mitigate CF fall into three categories: (i) Regularization-bal
methods (Kirkpatrick cet al. 2016 Zenke et al. 2017: Li & Hoiem 2017) constrain updates protect old tasks but often limit new learning. (ii) Memory-replay strategies (Shin et al. 2017 Rebuff et al., 2017 Chaudhry let al.) 2019) interleaves past and current data, yet pretraining data of modern open-soures MLLMs 1s usually unavailable for post-training. (iii) Architecture- based techniques (Rusu et al., 2016 Serrà
al. 2018) assign task-specific modules, but their parameter overhead makes them impractical for large MLLMs. Fortunately, with the rise of RFT algorithms such as GRPO (Shao et al. 2024). recent
studies (Liu al. 2025b Wang al.) 2025 Lai al.) 2025) have shown that RFT can significantly reduce CF in MLLMs, although their analysis and explanations are still limited. Recently, RL's
Razor (Shenfeld et al. 2025) argues that online RI. mitigates CF because it is implicitly biased toward KL-minimal solutions. Aligning with this discovery, we further analyze why online sampling distribution inherently reduces forgetting from a data-centric perspective with learning dynamics theory. Besides, would like to clarify that our paper isn 'n't aim to design better algorithm than classical methods for continuous learning.
3 DEFINITION AND BACKGROUND
This section details the format of jigsaw puzzles tailored for MLLMs. For Reinforcement Fine- TTuning (RFT), we propose several rule-based rewards to learn jigsaw puzzles.
3

---

Published as a conference paper at ICLR 2026
3.1 PUZZLES GENERATION
Image Slicing. Puzzle creation begins with source image, which is divided into m x regular grid; adjusting m and n directly controls the difficulty of task. If the image height is not divisible by the width by the excess pixels are cropped from the bottom right edge, respectively, so that the resulting pixels are exact multiples of the grid cell size. The aligned grid is then used to slice the image into x patches, whose order is randomly permuted produce the puzzle instance.
Index Assignment and Objective. To uniquely identify each patch's original position, we assign row-major indices from (top-left) to
(bottom-right). The model receives this permuted sequence of patches as input and must output the indices in canonical top-left to bottom-right order, thereby reconstructing the image. In this study, we adopt 3 x configuration. Empirical results show that statc-of-the thr-r-rrtttttttt multimodal large language models perform at the chance level on this task.
3.2 RULE-BASED REWARDS AND RFT
The objective of RFT driven by rule-based reward that comprises three components: the hit reward Rhit, accuracy reward Races and format reward Rimt-
Hit Reward. This term measures partial correctness by computing the fraction of position indices that are predicted accurately:
# correct indices
REt
E [0. 1].
m X
Accuracy Reward. A binary bonus that evaluates whether the entire configuration is correct. The model receives Race = 1 only when every index perfectly placed; otherwise Race 0.
Format Reward. The output must satisfy formatting rules: the reasoning process is wrapped in
</answer> tags, with each tag appearing exactly once and in the correct order. The final answer must be non-repeating sequence of digits 0-S within '[J'. If all requirements are met, Rimt 1; otherwise, Rim = 0.
<think> </think> tags and the final answer in <answer>
RFT Algorithm. We adopt Group Relative Policy Optimization (GRPO) (Shao al. 2024 our RFT algorithm. Formally, we maximize the following objective:
lodl
no (4 ALt 33DL. (moll (1)
JGRPO = E. g. ao (110 T
0g
Tea (0i, 44, 0i,
where is the problem, r = (ri.... ral is reward for model outputs for,-.. ... oa.. Ait = (r. me)n(r)) (r))/st((r) is the advantage for each token. Besides, Te = TT(.) ) in our experiments, so we omit the original clip term here for simplicity.
EXPERIMENTAL SETUP
Dataset Construction. Jigsaw puzzles are built upon COCO 2014 (Lin et al. 2014) image dataset. For training, we sample around 22k images from the COCO training set to generate jigsaw puzzles. For testing, we sample 100 images from the COCO test set. For the SFT dataset, we provide two data formats, iee. Kon-Reasoning data and Reasoning data: the first directly provides the ground-truth answers without reasoning processes, while the second additionally consists of reasoning trajectories generated by GPT-4o with both the question and the answer as input, dubbed as Rea-4o-Rollout.
MLLMs. We employ Qwen2.5-VL-33 (Bai et al. 2025) and Qwen2.5-VL-7B our MLLMs due to their strong performance on vision-language understanding and support of native resolution input.
Evaluation. We not only evaluate the post-trained model on novel tasks, ie., jigsaw puzzles, but also on representative capability axes of prior knowledge:
Grounding, RefCOCO/+/g (Kazemzadeh et al. 2014 Mao et al. 2016) test referring-expressoon comprehension, requiring the model to localize objects described by the free-form text.
4

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison across post-trained models of Qwen2.5-VL-3B and Qwen2.5-
VL-7B. Numbers in parentheses denote the change w.r.t. each scale's 's base model.
Qwen2.5-VI.-tt
Qwen2.5-V1L-22
SFT-Rea-
SFT-Ren- 4o-Rollout
SFT-Rea-
Base
RFT
SPT-Non-Rea
4o- Ao-Rollout
GRPO-Rollout
Base
RFT
SPI-Non-Rea
GRPO-Rollout
Jigaaw Parzles (lest)
Training steps 3x3 puzzles
2,670
3,000
0.0
66 (166)
53.0 (153)
70.0 (170)
(1
0.0
75 (175)
80 (180)
78 (178)
(181)
Grounding
RefCOCO.
88.8
88.4
6.1 (U828)
(J.46)
(L42))
90.0
89.4
(2.9())7.2)
52.5 (L.37.5
(1.4
Re1COCO+al
82.0
82.2 (10.2)
(177.7)
68.3 (413.)
77.6 (1444)
84.7
83.6 (11.1)
28.8 (155.9)
47.8 (1.36.9)
75.1 (19.6)
RefCOCOg.a
86.0
84.1 (11.9)
5.6 (U80,4)
71.3 ((14.7
80.8 (L5.2)
86.4
86.3 (10.1)
30.1 (156.4)
48.3 (L.38.1)
76.1 (1100.3)
Document & OCR
Doce
91.5 (11.3)
(1111)
(12.5)
89.8 (L3.1)
94.4 (10.0)
(127.4)
92.1 (12.3)
(J0.9)
74.3
73.1 (11.2)
62.6 62.6 (1111.7)
(12.8))
70.7 (L3.6)
80.1
79.1 (1110)
(4.65)35.5)
75.6 (L4.5)
77.3
OCRBench
79.3
77.1 (12.1)
65.9 (3133))
69.1
74.9 (L4,4)
83.4
834 (10.0)
(43.7)
80.5 (12.9)
(124
General VQA
MMEan
2140
2137 (13)
1631 (1.509)
1478 (1,62)
2132
2333
2325 (18)
479(11854)
2084 (1249)
2207 (L126)
MMS
56.2
55.8 (10:5)
49.2 (17.0)
51.7 (14.5)
(14.((
62.8
64.4 (11.)
0.D
59.1 (13.7)
60.4 (12.4)
OQA
(10.6)
54.7 (15.4)
(0.001)111)
(1661)
60.3 (10.1)
(1.7
$3.5 (U6.))
$7.0
Hollacination
POPE
86,9
86.5 (10.3)
85.9 (U1.0)
69.4 (117.5)
(L14)
86.2
86.0 (10.2)
(6.699)
74.1 (112.))
83.1 (43.D
Collego-tere Problems
MMMU,
(10.6)
43.4 (U3.5)
(13.9)
(12.6)
50.0 (11.3)
21.4(128.))
(L5.2)
(12.6)
OCR, chart & document understanding, Doc (Mathew et al. et al., (202)), and OCRBench (Liu al., 2024) probe the ability of MLLMS read and reason over scanned documents, forms, and scientific plots.
2021 InfoVQA (Mathew
General VQA. MME (Fu et al. 2023), MMStar (Chen et al. 2024), and GQA (Husson & Man- ning 2019) cover visual reasoning, spatial relations, and multimodal commonsense.
Hallucination. POPE (Li et al. 2023 measures tendency of generation not grounded in image.
College-level Problems. MMMU Yue et al. 2024 is college-level multimodal benchmark spanning six disciplines for wwwllede-grounddee visual reasoning.
Hyper-parrmettr setup. All experiments are conducted on 4 x NVIDIA-A800 80GB GPUs. For GRPO tuning the jigsaw puzzles, set the number of generations G=4 per prompt with sam- pling temperature of 1.0, batch size 4, learning rate 1 x 10-6, and KI. divergence penalty coefficient 8-0.04. For SFT training, set the batch size to 16 with learning rate of 1 x 10-5 by default.
RESULTS AND ANALYSIS
5.1 CAN RFT MASTER THE NOVEL JIGSAW PUZZLES?
We first test current wate-of-the-art multimodal large language models (MLLMss) on our test set of jigsaw puzzles in zero-shot manner. We find that both GPT-4o and Qwen-2.5-VL-72B obtain an accuracy of 0.0, and their hit rate of correct position indices is close to random chance 1/9, indicating that jigsaw puzzles are indeed a novel task for these models and are suitable for our research on forgetting during learning of new tasks.
We then examine whether reinforcement fine-tuning can enable the base model learn entirely new tasks or knowledge from scratch. Specifically, we apply GRPO to 0wwn-2.55VLL33B77 on the training set of jigsaw puzzles for 10 epochs, encouraging a comprehensive and sufficient exploration of the novel task. After convergence, the final models achieve an accuracy of 66%/77% on the held- out test set as shown in Tab. I dramatically outperforming the base model. Qualitative results on test examples, as in Fig. I1 of the Appendix, show that the model learns to generate meaningful reasoning processes before giving the final answers. Although prior work (Yue et al. (025) suggests that RFT fails to induce fundamentally new reasoning patterns in base models, show that with sufficiently long-term exploration, RFT can in fact enable the model to solve novel jigsaw puzzles from seratch.
5

---

Published as a conference paper at ICLR 2026
5.2 FORGETTING OF PRIOR KNOWLEDGE: RFT Vs. SFT
We also fine-tune approach with Non-Rea and Rea-4o-Rollout dataset. As shown in Table T due the property of teacher forcing, the model quickly picks up task-specific patterns under SFT, achieving performance comparable to RFT after just one epoch. To assess the impact of SFT and RFT on previously learned knowledge, we further evaluate both models on a set of prior benchmarks in Tab. [ While SFT achieves high accuracy with much less training time, leads to significantly more catastrophic forgetting than RFT, even though it is trained for many fewer steps. This forgetting is particularly evident on the Grounding, Document OCR, and General VQA. Besides, SFT on Non-Rea data incurs much more forgetting than on Rea-4o-Rollou data across several prior benchmarks.
on the jigsaw puzzle training set using the standard SFT
5.3 WHY DOES RFT AVOID CATASTROPHIC FORGETTING?
We start by analyzing the loss function of RFT and SFT. By carefully comparing the gradients of the RFT and SFT losses (Eq. 11 and Eq. 13, derivation be found in Appendix CD, find that both losses optimize the model's s likelihood. However, the difference lies in the fact that RFT optimizes on the dataset sampled by the model and uses adaptive weights for the likelihood objective, while SFT uniformly improves the model's likelihood on pre-constructed dataset.
Therefore, investigate whether the corpus sampled from the model itself enables the base model to Iearn jigsaw puzzles while retaining its performance on previous tasks with SFT. Specifically, we employ the GRPO-trained model generate responses on the training split of jigsaw puzzles, and filter the responses based on the correctness of the answer, leaving about 65% of training samples. We then use this filtered corpus to fine-tune the base model under the SFT paradigm. To rule out confounding factors, we adopt exactly the same hyperparameters as in previous SFT experiments.
As shown in Tab. D fine-muning on model-generated data (SFT- -ooloo)) achieves similar accuracy on jigsaw puzzles, while forgetting much less than SFT-Non- Rea and SFT-Rea-4o-Rolle across most benchmarks. Interestingly, we also find that training on Rea- Gee-GRROO-Rll Rollout learns much more slowly than Non-Rea data, necessitating more training steps to achieve a comparable performance on jigsaw puzzles. This may be because the long reasoning paths dilute the per-token learning signal. Overall, we find that it is not the adaptive weights but the training data that is the key factor why RFT does not suffer from catastrophic forgetting.
and Rea-4o-
5.4 LEARNING DYNAMICS- BASED ANALYSIS OF DATA DISTRIBUTION
Motivated by the observation that the distinct data distributions in the post-training phase lead to different forgetting behaviors, we take learning dynamics perspective to investigate and explain this phenomenon. Let's consider the SFT loss on different datasets:
min C(0) -iggo.t)- log To Oct),
where o is the ground-truth next token, conditioned on the prompt y and previous completion dgr-
Following Ren & Sutherland (2024), we employ learning dynamics to describe "how the change in parameter 0 induced by a step of gradient descent on single training example x,, _ {q"h o of)
impacts the probability of another example n. A {q" o% }" Here, we treat 2h GRPO- rollout sample or man-made SFT sample, and x. as sample of prior knowledge. We have
A (#w) A gt-1 0¹ 17 V log Tez (z.) 17 Ve log To. (o ,0=))
(3)
A log nT' (Ir,)aa, A log Tga (f.) ) - log Tog (x.).
(4)
And we want to specify the relationship between A and A log (zv)le..
Theorem 5.1. Let x (x) = (oftmax(=(=)) E [0, 1], where z x(x) = hg: (g. 0 E RV, V is the number of tokens within vocabulary. The one-step learning dynamics has the following format:
A (xx)" = ('(x)) 1C' (xx,,, xw) G (x.) ( O(n²)
1x1
1xV
VXV
Vxl
where A',x,) = Vz lo (x,), K' x.)) = (Voz(z,))a) (( T is the empirical neu- ral tangent kernel (eNTK) of the logit network z, and g' (Ix.) Va log To. (F.).
To.
6

---

Published as a conference paper at ICLR 2026
Non-Rea Qwen2.5-VL
Rea-GRPO-Rollout
Rea-4o-Rollod
Qwen2.5- VL
4
10
LBK.
2
5-
0
50
100
150
200
0
50
100
150
200
Steps
Steps
Figure Evolution of LBK,. during the SFT process on three different datasets.
Question:
Base: "json
Please provide the bounding box coordinate of the region this sentence describes: man in black.
"bbox 2d": 1289. 140, 396, 419,, "label": "man in black")
SFT: [0.43, 0.37, 0.61, 0.821
Figure 3:
Qualitative result on Grounding before and after training with SFT on Non-Reasoning data. Model finetuned on Non-Reasoning data often switches its output format Grounding, i.e., from the expected ISON format containing bbox_ 2d and label to a list of numbers.
Proof of the theorem and more discussion can be found in Appendix D The theorem shows that the effect of AO' (Ir.) on A log (a,)))l,, mainly determined by three factors: (1) the model's sensitivity to the old and new knowledge (A4()).) and (rr)))) and (2) the level of interference between them, captured by K (' T.). Since the gradients with respect to the logits (i.e., (x,,) and G' (Ix.) are typically bounded, this implies that the relative interference is the dominant factor driving forgetting. A larger ||]]]]] means more interference between 2u and Tu. Besides, our anal- ysis in this section also depends on the assumption of "the eNTK matrix K remains roughly stable over training". which is well-validated in Ren & Sutherland (2024) and our following experiments.
So we first measure the interference between the post-training dataset and prior knowledge during the training process of SFT. As it requires huge computation to calculate ||K]|] || directly, we estimate the Lower Bound of Kernel ||A*||| (LBK) as follows:
|| log n (Ir,) (r,))=,, || S 7/| (x.,) (x.))l= (Z,, a |= (sw) (1)) 110(1))
(6)
log n' (Ix, Ia..
LBKE LE
S (Io,
(7)
||A4 (x.) I
(I. (x.))
Specifically, we sample responses from base models on Grounding as our prior knowledge Zy, which follow similar answer format to jigsaw puzzles (i.e., with numbers enclosed within '[I') and ex- hibit the most severe forgetting as in Tab. I We then conduct SFT training on three different datasets I, and record the LBK between prior knowledge and training examples. As shown in Fig. 2 the LBK quickly stabilizes after only a few dozen training steps. Besides, the Non- aan-Reaooning data ex- hibit much larger LBK compared the Reasoning data, suggesting stronger interference with prior knowledge. Appendix Fig. TO further shows that introducing reasoning trajectories improves the model's confidence in answers. These suggest that directly providing answers to new tasks, with- out linking them to the model's existing perceptual abilities through reasoning trajectories, causes the output distribution to shift abruptly as in Fig. 3l which heavily disrupts prior knowledge and leads to catastrophic forgetting. In contrast, for Reasoning data, the LBK is smaller, meaning that interference with prior knowledge is weaker and forgetting progresses more slowly.
5.5 WHAT MAKES THE MODEL-GENERATED REASONING DATA DIFFERENT?
Next, we investigate why reasoning data generated by model itself (Rea-GRPO-RRllouu) and by GPT-4o (Rea-4o-Rollou still result in different forgetting behaviors. To do this, we use the per- plexity (PPL) of the base model as measure to compare how well each type of data aligns with the model's distribution. As shown in Fig. perplexity region of the base model's output distribution, whereas Roa--o-Rollout typically lies much higher than Rea-GRRP--olllut. This suggests that GRPO- is more compatible with the base model's prior knowledge when compared to the Rooloo.Rollout.
Gool-oooo tends to align with the lower-

---

Published as a conference paper at ICLR 2026
Base- G--oollutt (Box)
Rea-GRPO-Rollout
-coloo-Rollout
Qwen2.5.VV V
20.0
10.0
PPL
12.0
8.0
4.0
Qwen2.5 Owee2.5.VVL-77 V.
18.0
150
12.0
PPL
3.0
Question index
Figure 4: PPL of Rea-GRPO- GRPO-Rollout and 40-Rollou under the base model. Base- 16-Rollout (Box) denotes PPL range estimated from 16 rollouts generated by base model, serving as a reference.
But then, why does post-training strategy that focuses low cww-perplexity samples alleviate catas- trophic forgetting of prior knowledge? Fortunately, we can answer this using the following symme- try property from learning dynamics:
Theorem 5.2. The one-step learning dynamics has the property of symmetry:
A AlogT'S (z.)le. = A log T' (Iu)la, + O
(8)
According Theorem 5.2 (Proof in Ap- pendix DD. we find that the influence of learn- ing Tu on Ty is nearly the same as the influ-
(won2.5-VL-3B
Qwen2.5-VL-7
TigSlww (Ace)
60-
JigSaw (Acd)
70-
ence of learning To Ta- Additionally, since the eNTK matrix stabilizes (Ren & Suther-
50-
40-
50
land 2024) in the later stages of pretraining, the interactions between Ia and 2e remain consistent across the training step i, also ob- served in Fig. 2 Therefore, models pretrained with prior knowledge show lower perplexity for Sea-GRPO-Rollon with prior knowledge enhances these samples. During post-training, further training the Rea-GRPO-RRllout samples results in less interference with prior knowledge compared to other higher perplexity samples like Rea-4o-Rollout, As shown in Appendix Fig. g when training with Rea-GRPO GRPO--Rottll,,,,, perplexity of sentences representing prior knowledge continues to decrease on 3B model and remains low on 7B model. In contrast, under Rca-4o-Rollout perplexity steadily increases. As result, forgetting effect for Rea-GRPO- Rollout is less pronounced than for Rea-4o-Rollout. Moreover, reinforcement learning algorithms like GRPO, which naturally generate training samples through model rollouts, tend to produce sam- ples with lower perplexity under the base model. This explains why reinforcement learning methods less prone to catastrophic forgetting.
30.
SFT-4o
SPT-GRPO
20
75
81
56
Grounding (REC)
Grounding (REC)
Pareto front curves the jigsaw and Rec tasks for models fine-tuned on Rea-4o- Rollout and Rea-GRRP- data.
Figure 5:
indicating that training
5.6 MORE EXPERIMENTS ON REA-GRPO-ROIIS
AND REA-4O-ROLLLLOU
We further plot the Pareto front curves of accuracy on the Grounding and jigsaw tasks during training on the two datasets. As shown in Fig. 5 the Pareto front from Re---ee-GRPO-TT Rollout clearly better than that from SFT-Ree-44-Rolll much smaller performance variance on the Grounding task during the SFT process, indicating that Me--GROO-Rolooo interferes less with prior knowledge of base models. In contrast, SFT-Rea-4o- Rollout improves jigsaw performance at the cost of degrading Grounding performance. This result further highlights the importance of low wewppeeeexxtty under the base model illustrated in Sec. 5.5 Though Rea-4o-Rollon data generally has a smaller LBK on Qwen2.5 Swen2.55VLL-BB VLL3B as in Fig. it still forgets more due to its property of high-perplexity.
Moreover, models trained with Rea-GRPO-RRllon show
8

---

Published as a conference paper at ICLR 2026
Table Performance comparison across post-trained models of Qwen2.5-3B-Instruc and
Qwen2.5577-IIstrruc Numbers in parentheses denote the change w.r.t. each scale's 's base model.
Qwen25-3H-nnsttee
Qwen255-7--neiirua
Base
RFT
SFT-Now-Rea
SFT.Ren- 4o Rollont
GRPO Rollont
Base
RET
SFT-Non-Rea
SFT-Ren- 4o-Rellout GRPO-Roleen
Open-Reasoner-22 Zero (tert) (New Task)
Training steps
2,650
1,00
2.140
2,140
2,650
1,500
2.140
2,140
ORZ Test
21.3
(11377
2f2.11211
(1140))
37.8 (1164)
32.1
(017.))
30.3 (U1.8)
(f.0
53.4 (121.3)
Marh Recsnnnng (Old Tisks)
GSMBK
84.1
83.4 (10.7)
15.1 (169.0)
(4.2()
83.0 (1LL1)
90.1
90.2 (f0.))
21.8 (.0844
85.8 (14.3)
90.3 (10.2)
Math-500
42.4
55.2 55.2 (112.)
19.4 (422)
508 (18.)
54.4 (1120)
66.6
6488
26.4 (440.))
57.2 (49.4)
66.4 (10.2)
Instruction Following (Old Task)
IFEval
71.6
73.4 (11.)
64.0 (2.66)
68.0 (13.6)
72.2 (11.)
80.6
80.5 (10.1)
57.2 (123.4)
64.4 (L16.2)
80.0 (10.6)
Non-Rea
Rea-4o Rollout
Rea- GOPO-RRROOooe
Qwea2.5-LLM:
Qwen2.5-LLM-7B
40-
LBK..
50
20-
25
0
30
10
30
40
Steps
Steps
Figure 6: Evolution of LBK.v during the SFT process with three different datasets on math dataset.
5.7 LLM EXPERIMENTS ON MATH REASONING AND SCIENCEQA
We additionally provide experiments on the LLM Qwen2.5-Instruc (Yang al. 2024) here, show- ing its forgetting behavior during post-training on math reasoning, along with the corresponding results. We hope these extra experiments can further strengthen the generality and credibility of our theoretical analysis and conclusions. More detailed experiments setup can be found in Appendix F As summarized in Tab. 2 the math reasoning experiments exhibit a forgetting pattern highly con- sistent with our multimodal jigsaw setting: on both 3B and 7B scales, SFT-Non-Res achieves the largest performance drop on the old math (GSM8K, MATH-500) and cnntructtoo-follwwiitn (IFEval) benchmarks, while reasoning augmeentd SFT with external CoT (SFT-Rea-44) 4c----oooo))) forgets less but still substantially more than SFT-Rea-GREP GRPO-Rolloot...........ooooooo.. The latter attains strong gains on the new ORZ task while keeping the performance on old tasks close to the base models, indicating the same hierarchy of forgetting severity, i.e., Non-Rea > Rea-4o > Rea-GRPO.
To probe the underlying mechanism, compute LBK between post-training samples and prior math knowledge during SFT. As shown in Fig. 6 Non-Rea data consistently display much larger LBK values and Rea-4o also contains occasional high-IBBK outliers, providing further evidence for the generality of our learning- -enninngdyyamiccs analysis in Sec. [.4 Moreover, Fig. 7 shows that Rea- 4o-Rollout concentrated in the high-perplexity region of the base models, whereas Rea-GRPO- Rollout lies closer to the oowperplyyityy region, mirroring our findings on jigsaw puzzles and sup- porting the www-perlexxtyy training hypothesis in Sec. 5.5 that post-training on model-aligned (low- PPL) reasoning trajectories mitigates catastrophic forgetting. Besides, we also analyze the Pareto front curves (Fig. [3) and Perplexity on prior knowledge (Fig. [4] of different SFT datasets in Ap- pendix F it is consistent with our theory and prior analysis jigsaw puzzles.
Beyond math reasoning, we also conduct experiments on a scientific multiple-choice QA benchmark to test the robustness of our conclusions. Concretely, use the Sci-MCQ4 subset from SciKnow- Eval (Feng et al. 2024)); detailed settings and full results are provided in Appendix H As shown in Tab. and Fig. 17 we again observe the same hierarchy of forgetting severity on prior bench- achieves the best trade-off between performance gains on the new Sci-MCQ4 task and retention old tasks, further supporting the generality of our analysis.
marks, while
Direa-GRP--Ran
5.8 'COOPERATION' BETWEEN SFT AND RFT
In our previous experiments, we first generate reasoning data after RFT has achieved a high jig- saw accuracy. SFT training on such data can not only achieve high accuracy the new task, but

---

Published as a conference paper at ICLR 2026
Base- Rollout (Box)
Rea GRPO-Rollou
Rea -ollolo
Qwen2.5-1.8 5w..2.5-LLL.M33. 3B
5.-
4.0-
3.0
1.0
S-en2.5-11LM--11
4.-
121
12-
244
4b
100
Question index
Figure 7: PPL of Rea-GRPO-Rollowe and Rea-4o-Rollous math reasoning dataset under the base LLM (Qwen2.5-Instruct) Base-166 1o-Rollou (Box) denotes PPL range estimated from 16 rollouts generated by base model, serving reference.
also preserve old knowledge better than Rea-4o-Rollout. This phenomenon suggests that the data distribution is a key factor that determines whether the model forgets during post-training.
To further verify this, we show that if we only want to generate data aligned with the model's own distribution and capable of teaching the new task, does not require running RFT to very high ac- curacy, As shown in Tab. 5 run RFT for only one epoch (5,472 steps), during which the model's jigsaw accuracy stays below 5% (see Fig. | (Left) in the appendix). Even so, by collecting the model's rollout CoT and pairing with the correct answers, we can already construct effective Fine-tuning the base model on this dataset yields new-task ac-
SFT dataset Saa--Seff-GGerettee curacy comparable to RFT and SFT-Rea-GGRP-RRollon while its performance old tasks is also similar to them and much better than SFTT-Rea-4-Rollout.
DISCUSSION AND CONCLUSION
SFT is a widely used post-training method and is often employed as cold-start phase for RFT (DeepSeek-AI et al. 2025). helping the model acquire basic skills that support subsequent exploration. Besides, SFT also enables the base model to master novel tasks quickly. However, manually curated SFT corpora can lead to the forgetting of prior knowledge. In this work, we show that one can instead construct more stable SFT training data from the model's own reasoning trajec- tories produced by RFT. Even short RFT phase is sufficient to generate such self-consistent data (Sec. (5.8). and a subsequent SFT update on this corpus attains new-task performance comparable RFT while preserving prior knowledge better than SFT-Rea-4o-Rolllut. Therefore, developing an efficient and reliable interplay between SFT and RFT that combines their respective advantages remains a promising problem.
This paper provides a systematic investigation into how post-training algorithms affect knowledge retention in multimodal large language models. By introducing jigsaw puzzles genuinely novel task, we uncover a clear contrast between SFT and RFT: while SFT enables rapid task acquisition, it suffers from severe forgetting: contrast, RFT achieves stable learning without significantly degrading prior capabilities. Through empirical studies and theoretical analysis grounded in learning dynamics, we show that this difference arises not from the training algorithm itself, but from the distribution of training data. Specifically, introducing reasonable reasoning trajectories into the SFT process can help alleviate forgetting due to less interference with prior knowledge. Besides, RFT naturally discovers low-perplexity examples that are already partially aligned with the model's output space, making them less disruptive to previous knowledge. Furthermore, using RFT rollouts supervision enables SFT to forget less, underscoring the importance of fine-tuning data quality. These findings suggest that future post- training efforts should move beyond algorithmic choices and focus on data selection.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by Henan Province Major Industrial "Challenge-Based Innovation" (No. 251000210300), National Natural Science Foundation of China (No.62476061, 62376061, 62576106, 62521004), Shanghai Rising-Star Program (23QA1400200), 1400200), and Natural Science Foun- dation of Shanghai (23ZR 1403500). Supported by Shanghai Artificial Intelligence Laboratory.
ETHICS STATEMENT
From data distribution perspective, this research employs learning dynamics explain the ad- vantages of the sampling distribution induced by RI and why RL. training tends to yield reduced forgetting. We firmly state that this work is intended for ethical and constructive purposes. Users of this method bear the full responsibility for ensuring it is applied in safe, fair, and harmless manner. Any misuse of this method is strictly against the intent of the authors.
REPRODUCIBILITY STATEMENT
We have described our theory analysis and the experiment setup in Sec. and Sec. 5 reproducibility, we will open-source our code.
To support
REFERENCES
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov. and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang. Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mfing-Hsual Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502. 13923, 2025, doi: A44S550ARXI..252.215.
URL httph:tt///c// org/
10. 118550/arXi11 2502. 13923
Fabio Maria Carlucci, Antonio D' D'Innocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Domain generalization by solving jigsaw puzzles. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2229-2238 doi: 10.1109/CVPR 2019.00233. URL http:
Computer Vision Foundation / IEEE, 2019.
//openaccess. thecvf. com/ content CVPR 2019/btml/Carlaaec Generalization by "r_Soolving Jigsaw Puzzles CVPR 2019 paper. html
Domain.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, P Dokania, Torr, and M Ranzato. Continual learning with tiny episodic memories. Workshop on Multi- Task and Lifelong Reinforcement Learning, 2019.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evalu- ating large vision-language models?
Amir Globersons, Lester Mackey, Danielle Bel- grave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Informa- tion Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL
cc/paper files/ httlls/pappr/2224//// html
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemy Tworek. Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/2110..41ee 2021. URL
https://arrs wwtp:/wwwww.g///.//..ooo 14168

---

Published as a conference paper at ICLR 2026
DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangho Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao. Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang. Peng Zhang, Qiancheng Wang. Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepscek-rl: Incentivizing reasoning capability in ilms via reinforcement learning. CoRR, abs/2501. 12948, 2025. doi: 10. 25550/ARXIV.2501.1 12948. URLhttps:://do wwt.ooggo/o.. 48550/ arxiv 2501. 12948
Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang. Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen. Sciknoweval: Evaluating multi-level scientific knowledge of large language models. CoRR, abs/2406.09098,, 2024. doi: 10.48550/ARXIV.44660000 URL httph: www.:o..rgr11... 4850448550.00 www.4/////.x..4000000 09098
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive eval- uation benchmark for multimodal large language models. CoRR, abs/2306. 13394, 2023, doi: 10. 48550/ARXIV.2306. 11..ARIXIII.3011334433444 13394. BtRLhttts:/// //dtiss://di.... /doi. /00/10.0000. 48550/8550000 arwiv. www.w.wwww......... 2002.13333333
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL.
proceeed-poooedddggg...o. neur ps. cc 2201/ https: hatth//eBBa3ecc00077222222llaaa3 - round2.htm.
Abstract
Jingeheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open-reasoner--eroo An open source approach to scaling up reinforcement learning on the base model. CORR, abs/2503. 2025. doi: 10 08850/ARRIVV22503222233 URL. https://
doi www.lorrgl..... 48550/ 2503.24290
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6700-
2009//VPR.2019.0000 6709. Computer Vision Foundation / IEEE, 2019. doi: SWPR__0099/tmm/Hucsee _
URL
httttp//eeeaccss.hhheee thecvf. com/ waapa//oneeet____eeeeeeeee CVPR Reasoning_ wwwaanddddosiiiooooo
A New Dataset_ for__foooaaallaaaa-- Serall-orldd_aaeee CVPR SSFR_2019_paa 2019_paper.httm
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referitgame: Refer- ring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang. and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing. EMNLP 2014, October 25-29, 2014, Doha, Qatar. A meeting of SIGDAT, Special Interest Group of the ACL, pp. 79877798. ACL, 2014. doi: 10.3115/V1/DD 1086. URL https:://oo. www.//orroo... htt..//./..0.111111111111111111.
12

---

Published as a conference paper at ICLR 2026
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agmieszka Baroskka-ar,iskku, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. CoRR, a1./166,. 2016. URL http://arxxv..org/
1612.0072 00796
Song Lai, Haohan Zhao, Rong Feng, Changyi Ma, Wenzhuo Liu, Hongbo Zhao, Xi Lin, Dong Yi. Min Xie, Qingfu Zhang, al. Reinforcement fine-tuning naturally mitigates forgetting in continual post-training. arXiv preprint arXiv:2507.05386 2025.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evalu- ating object hallucination in large vision-language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 292-305. Associ- ation for Computational Linguistics, 2023, doi: 10.18653/V1/2223 2..666//1/2233MMMLPMAAA EMNLP-MAIN.200 URL http://d....r/11...8.6//////////
emnlp 20
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-294) 2017.
Tsung-V Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr
Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693 of Lecture Notes in Computer Science, pp. 740-755, Springer, 2014. doi: 1001 100. 978 31 3
10-3--19--1002--1 -.48. URL https: /tt::///oiiio wttp://///i..0//11111000000
48
Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong- Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models, 2025a. URL/htps:: httpp:s///////.//.///// http//axxiv..or/a/a/a// wwtp:o/////////ooe 2505. 24864
/
Yuliang Liu, Zhang Li, Mingxin Huang. Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of OCR in large multimodal models. Sei. China Inf. Sci., (72),, 2024. doi: 10. 00.100775514322-244433444 URL
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning- seasoning-chain chain guided segmentation via cognitive reinforcement. CoRR, abs/2503.06520,, 2025b. doi: 10.44550/AXXI.2553..05553
2503 06520
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0 2503.0 2025c.
Zesen Lyu, Dandan Zhang, Wei Ye, Fangdi Li, Zhihang Jiang, and Yao Yang. Jigsaw-puzzles: From seeing to understanding to reasoning in vision-language models, 2025. URL https:
wwtrii..rr//bb//2230 20728
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Mur- phy. Generation and comprehension of unambiguous object descriptions. In 2016 IEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 11-20. IEEE Computer Society, 2016. doi: 10. 10.11099CVPR.2016999
URL.
https: //oi//d.../o 110.//....11/////. R.0.P.R.016...
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for VQA on doc- ument images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, January 3-8, 2021, pp. 2199-2208. IEEE, 2021. doi: 10. 10.1109/WCC448600 2021.00225. ftR.pstts:://ott
org 1109 /ACCV486630 2021 00225
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawa- har. Infographievga. In IEFF/CVF Winter Conference on Applications of Computer Vision,
13

---

Published as a conference paper at ICLR 2026
WACV 2022, Waikoloa, HI, USA, January 3-8, 2022, pp. LI9//AACYS545::2222.00444 BRLllttps://db..co 10. 1109/NACV51458 220.15525.....2...... 00264
IEEE, 2022. doi: 10.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Bo- tian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring visual aha moment with rule-based large-scale rein- forcement learning. CoRR, abs/2503.07366 2025. doi: 1044550/ARXI.22000076666 URL
20t.///0.45550///// arxiv www....w................ 07365
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jig- saw puzzles. In Bastian Leibe, Jiri Matas, Nieu Sebe, and Max Welling (eds.), Computer Vi- sion ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, volume 9910 of Lecture Notes in Computer Science, pp. 69-84. Springer, 2016. doi: 10. 1..1007/978-3-31-4466666 978-3-319-464653
URL bttps://cci.org/ 10.107/
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, (7)2)228,, 1990.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl:
Incremental classifier and representation learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5533 5542. IEEE Computer Society, 2017. doi: 10.1109/VVP.2211..587
htRp:ttps: //t:: org//...........
11C9/CVPR, 2017. 587
Yi Ren and Danica Sutherland. Learning dynamics of Ilm finetuning. arXiv preprint arXiv:2407. 10490, 2024.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1606.04671 2016. URL http: arxiv. org/ abs/lbbs....... 04671
Joan Serrà, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In Jennifer G. Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsn, Stockholm, Sweden. July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pp. 4555-4564 PMLR, 2018. URL serra 8a. html
mlr ptwss//8C///////e wtt.pngs.s....p.ss.......
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseckmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint MXXx:2442200300 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang. Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. VLM-RI: A stable and generalizable rl-style large vision-language model. CoRR, abs/2504.07611 2025. doi: 10. 48550/ARXIVV2500.000
htthttt:: / 10110.85550 48550/ arxiv. 2504.07611
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RI's razor: Why online reinforcement learning forgets less. arXiv preprint a4X 2025.
14

---

Published as a conference paper at ICLR 2026
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan. and Roman Gamett (eds.), Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural In- formation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 2990-2999, 2017. URL
new..pp............ommmmm....... Abstract. Abstaatt html
Zifu Wang, Junyi Zhu, Bo Tang, Zhiyu Li, Feiyu Xiong, Jiaqian Yu, and Matthew B. Blaschko. Jigsaw-rl: A study of rule-based visual reinforcement learning with jigsaw puzzles, 2025, URL
https://aran //ppsi///axiv... org/ abs 2505.23590
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Lc. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0163: 096.00 2021.
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu. Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li. Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, 15s/2412.15115, URL https: //t:://ddoo /ooi www.or/./...00.....
2024. doi: 10.48550/ARXIV.24122.0
2412. 15115
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9556-9567, 2024.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does re- inforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13831 2025.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelli- gence. In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Con- ference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, vol- 70 of Proceedings Machine Learning Research, pp. 3987-3999 PMLR, 2017. URL .mw.diiinnmmm.......mmm.m..m...m mll. pwwss/v700000 www.////eeeea.a.ee...oo
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction- following evaluation for large language models. CoRR, abs/2311..09111 2023. doi: 10.49550/ARXII2331..00
Instruction-following
URL https: http:://dc..orr/o.....BB83
1611.07911
A LLM USAGE
This article employs large language models solely for polishing the sentence structures to better align with standard English writing conventions.
B LIMITATIONS
Due to resource limitations, our experiments are currently restricted to the Qwen-2.5-VL-33777 models. In future work, we plan to extend our analysis to larger multimodal models and large lan- guage models to assess the generality of our findings. Additionally, this study currently focuses only on the jigsaw puzzle task. Investigating forgetting behaviors across a broader range of multimodal tasks is an important direction we aim explore next.
15

---

Published as a conference paper at ICLR 2026
C CONNECTION BETWEEN GRPO AND SFT
This section follows the discussion DeepSeekMath (Shao al., 2024) on the unified paradigm of GRPO and SFT closely, And we include the derivation here for the completeness of the paper. We will first derive the gradient of GRPO loss. Specifically, we use the following unbiased estimator as KL divergence loss:
(o, 01,ct)
(1,0,,149 01,z1) log (or(os, 0s,<t)
DEL (To (Ter)
(9)
To(o 9 o1,c1)
By substituting the specific form of the KL divergence into Eq. get the following function:
lo:
no(0r, 01,ct)
(aRPO( (0) = Eg M G
Art
00
Toua 0s,21)
(10)
Tee 4, oi,c))
Tref /4, 0i,c))
3B
log
To
Therefore, the gradient of (0) is:
10g
(oto(on,l,,0ssc))
VoJeRPO(0) = ~ (10)
Ait
G
To 0(,ct)
1=1
Ther Tooloonel. 0i.ce)
log
Te(oi, 0i,ca)
T0 0i,c))
02
-E ma (1))
As, Vo log (o 01,0))
Oi
(II)
+ 3 (1
(o 0i,
Trer( 4 01, / (000(01.1100033) To To 0x,
( 0z,4z)
Frer(Oi, 12,
10:
EE,, ) ~ (+14) G
A
0
1=1
t=1
n Mer(onsla 01,0)) To(o,, 1q, (..0 0c,<()
-1)
B
Vo log To (o
Here, the second equal sign comes from the fact that Tew (-) TT(-) in our experiments.
In addition, the SFT objective is to maximize the following format:
JSPT (0) = E.goo
log To
(12)
So, the gradient of SFT objective is:
VoJ (0) = Eog.or-Dattset,,
Vo log (o(or)
(13)
Comparing Eq. and Eq. 13 we find that both gradients try to optimize the likelihood of the model. However, they are optimized different data sources with different gradient coefficients.
D PROOF OF LEARNING DYNAMICS RELATED THBOREM
Theorem 5.1. Let we. (x) (of) E [0, 1], where z(x) = hug (g. O22)) E RVV V is the number of lokens within vocabulary. The one-step learning dynamics has the followingformattt
A Alog x (xw))m 17 (xx.) K2 (x,, x.) G (((xx) +0(r2),
(5)
1xV
VxV
VXl
16

---

Published as a conference paper at ICLR 2026
where (z.) V= log To: (Ire). (Ze, = (Vez(z.))o (V T is the empirical neu- ral tangent kernel (eNTK) of the logit network z. and g (Iu) = V log Tg. (Tu).
Proof. We first apply first-order Taylor expansion approximate log m (Ir) within Eq.
4
log a (Ix) log mg (Ia) +
(Ve log mar (I.), A0" + A0' ( (I=))l
(14)
Then, substituting the gradient descent item (Eq. 3) into the leading term and applying the chain rule of calculus, we get
Vo log Tal (r.), Ad (I.))
V log Tay (r,)
(az(())) (n Vo log (1
1xd
1xd
1XV
Vxd
1xd
V x log Tor (x+)
Voz(Tu)lo: (n V log ner (Ixx
V (aw)orr )
IxV
Vxd
1xV
Vxd
V= log To (Fx)
Voz(xx))o (Voz(ru)lo:)
(Vz log Tgp (I.))
1XV
Vxd
dxV
Vx1
nA ".)
(15)
where d is the dimension of model parameters A.
For the remaining second-order term, we should notice that the trick of gradient clip is usually utilized to avoid too large gradients, we have
O11 (rr))l = On Vo log o (z,) =(1)).
(16)
Therefore, by reorganizing the terms in Eq. 14 we have
A log n' (s.)) nA' (3r )C' (Fe, ( + 011)).
The second term in this decomposition, 1C' (Ix,, z.). called the empirical neural tangent kernel and can evolve during training as the network adapts. For sufficiently wide networks initialized properly and trained with small learning rates, IC' stays nearly fixed throughout training the limiting kernel
in this case is referred to as the neural tangent kernel (Arora et al.) 2019 Jacot et al. 2018 Ren & Sutherland 2024). Additionally, Ren & Sutherland (2024) also validated a relaxed assumption for LLM fine-tuning: the relative influence of learning T. on other inputs 2.p remains roughly stable over training. Besides, the optimization steps during post-training of MLLMs in our paper are very less compared to the steps used in pre- training. So, the relative influence between v. and v, during the post-training remains similar to the influence during pre-training is a reasonable hypothesis.
Next, we prove the symmetry theorem of learning dynamics: Theorem 5.2. The one-step learning dynamics has the property of symmetry:
A log (zv)e. = A log n' (I.)),, + O(7²).
(8)
Proof. Following Theorem 5.1 we have
A Alog (rrr''(n))) (zw)lon
(A log ( T
1x1
1x1
nn V log To (Iv)
Voz(zx)lor No: (Vez(<u)lo=)
(V log To (IW)) + +
1xV
Vxd
dxV
Vxl
V log Toe (xw)
Voz (x (V (Voz(z))lo=)
(V log Tor (I.)) +0(??)
1xV
Vxd
dxv
Vxl
A log (zn)
011²) + O11²)
A log (#w)le. + On72)
This theorem points out that the influence of training on w, over another example x, is almost similar to the influence of v. on Tu.
17

---

Published as a conference paper at ICLR 2026
Algorithm 1: Construction Process of the 3 x 3 Jigsaw Puzzle Dataset
Input: Image dataset D (e.g.. COCO-2014); grid size = metadata file
Output: Tiles
foreach I E D do
(H, W) & size(T);
H' <- [//m] x m:
W' < [W/n]
if (H', W") & (H) then
I' +- bicubic bicubic.resize H', W'):
else
1' I: I:
(hate; Whil) (H' /m, W /n);
// Row-ma jar slicing into x tiles for to
for c n-
+ x
// Crop Tile & from Image
Th + crop(1', ,r,",rre, bate; Whk));
(mve
// Shuffle tiles
π f uniform_randee www.rraannddmraaeee pww.utationnn (O...., m x n 1)); metedaataa H, W, H', W m);
Table 3: Total training cost (in GPU-hours) for different model sizes and training recipes for the jigsaw puzzles (Qw3n2.5-VL-3BB7BB and math reasoning (Qwen2.5-3B//BB)
Method
Qwen2.5-VL-3B (jigaw)
Qwen2.5-VL-7B (jigsaw)
Qwen2.5-3B (math)
Qwen2.5-7B (math)
RFT
2200
96
SFT-Non
2.3
0.77
1.3
SFT-Rea-4o-Per Rea--o-RRolouut
6.4
11.5
1.3
2.3
SFT-Rea-GRPO-Roll
5.1
8.3
2.5
E
MORE RESULTS OF JIGSAW PUZZLES
Jigsaw Dataset Construction Details. We construct the 3 x 3 jigsaw dataset upon MS COCO images with the preprocessing pipeline in Algorithm For each image I, we obtain its original size (H, W) and compute the nearest resolution (H', W such that both H' and W are divisible by 3, applying bicubie resizing if needed to obtain l'. We then partition I' into a 3 3 grid with tile size (hate; Whit) HH' /3, W" /3) and assign row-major indices & {O...., 8). A uniform random permutation π of (O....,8 8} is used to get the shuffled indices. At training time, the model receives foltttto-bottom--ii indices.
the shuffled tiles and outputs the canonical
Training Cost. Table3 main experiment confiuurations of Jigsaw Puzzles and Math Reasoning. The largest configuration (Qwen2.5-VL.) LL (jigsaw) RFT) requires about 2200 GPU-hours, while the SFT is two orders of magnitude cheaper.
the total GPU-hours (Number of GPU x Training Hours) for the
Jigsaw Puzzles with Large Learning Rate. To better illustrate how different training corpora impact forgetting, we increase the learning rate of SFT 2 10-5 amplify the effect of forgetting. As shown in Tab.44 finetuning Rea-GRPO- Rollout not only masters the novel task jigsaw puzzles better and faster, but also preserves more prior knowledge than Rea-4o-Rollout, Specifically, fine- tuning on Rea-4o-Rollee e.g., accuracy on RefCOCO drops from 88.8 to 0.16 on Qwen2.5.VL-3B. GQA drops from 60.38 to 42.74 on Qwen2.5-VL-7B. In addition, training progresses, SFT-Rea- GRPO-Rolooo shows slight improvements on some benchmarks, while SFT-Rea-4o- Ro---oo-Rollout Rollout exhibits a consistent decline on previous benchmarks.
causes severe forgetting after just 300 steps under this larger learning rate,
18

---

across three Published as a conference paper at ICLR 2026
Table 4: Performance of various models on jigsaw puzzles, grounding, document QA, and general VQA benchmarks after SFT with learning rate 2 x 10-5.
Model
Training Stegs
Jigsaw-test
Grounding
Document & OCR
General VQA
Hallucination
3x3 puzzllss
Re(COCO.u
DocVQAue
MME,
MMSiar
GQA
POPE
385 Reu-fo--Rullout
7 (17)
0.1 (188.7)
90..612.6)
2010
26.2 (1,000
53.8(15-4)
82 (14.5)
315 PenoolttO-R
100
(123)
81.1 (17.7)
68.3(14.5)
2014 ((116)
44.2 (L12.0)
47.1 (313.)
86.5 (L0.4)
389 Ren-An-RRlloot
500
30 (130)
0.2 (138.6)
89.5 (13.3)
1975 (L165)
25.0 (L.1.2
50.3 (19.8)
79.0
3B Ren-GRPO-RRoleee
(164)
80.5
88.4()4.4)
2039
42.6
(110.4)
87.4
7B Base
0
90.0
94.4
2333
62.8
60,4
86.2
7B Ren-do-Rollont
100
(129)
67.0 (123.11)
93.1 (11.3)
2:75 (L159)
55.9 (.6.)
46.0 (114.4)
78.9 (17.3)
785 Rea-GRPO GRPO-Rollou
100
58 (158)
68.8 (_21.2)
93.1 (11.4)
2078 (1226)
(_7.4)
54.6 (15.8)
83.5 (12.7)
7B Reu-fo-RRllout
500
(165)
62.0 (128.0)
(1.9))
2063 (U272)
(L.5()
42.7 (117.6)
(L15()
7B Rea-GRPO-RRllee
(47474))
(.1999)
(11.3)
(L246)
(U9.3)
(3.94)64))
(12.1)
Qwen2.5-VL-33
Qwen2.5-VL-77
1.4041
1.4128
1.6752
1.6776
1.6857
1.3951
PPL (REC)
1.40
PPL (REC)
1.6
1,7277
1.4932
1.4689
1.35
1,3410
1.4
1.3145
1.3184
1.2983
J,2344
1.30
1.2
800
1,600
2,400
3,200
4,000
0
800
1,600
2,400
3,200
4,000
Training steps
Training steps
Rea-GRPO-Rollow
Rea-4o-Rollout
Figure 8: Perplexity versus SFT training steps for Grounding. We collect rollouts from the base model with question from Grounding dataset as our prior knowledge.
Perplexity on Prior knowledge during SFT. We further examine how the perplexity of sentences representing prior knowledge changes during SFT. As shown in Fig. 8 with SFT--Re-GGRPO- Rollout, perplexity decreases steadily on the 3B model and stabilizes (since -ee---RROOR-oooo- data have higher perplexity 7B than on 3B). In contrast, under SFT- Sca-4o-Rollour perplexity keeps increasing. highlighting the importance of low-perplexity training data for preserving prior knowledge.
low level on the model
GRPO with Periodic Restart. We observe that training with GRPO for too many epochs on a novel task can lead to instability. Periodic restarts essential for maintaining stable training. As shown in Fig. 99, prolonged training eventually causes the KL. divergence loss to explode, consistent with findings from Liu et al. (2025a). To address this, reset both the optimizer state and the reference policy every 2 epochs. Accordingly, when rule-based constraints or well-shaped reward signals are available, prolonged exploration stage helps the model generate higher-quality trajectories. This allows it to acquire new skills that are not present in the base model, leading to measurable improvements on the target tasks.
Answer-Only Perplexity with Reasoning Trajectories. For each question, we further compute token-level PPL only on the final answers (i.e., the J..nwerr datasets. As shown in Fig. TO Non-Reasoning data forms higher, more dispersed band, whereas Rea-GRPO-Rollout and Rea-oo-Rollout data cluster into a lower band across nearly all 100 ques- tions, indicating reasoning trajectories help systematically reducing the uncertainty in decoding swers. Both reasoning dataset exhibit tighter vertical spread than Non-Rez data, suggesting not only lower average PPL but also smaller variance across questions. The observation PPL across three datasets supports the claim that introducing thinking mitigates conflict from the novel jigsaw objective.
SFT with self-generated CoTs As shown in Tab. 5 using only one epoch of RFT (5,472 steps) self-generate CoT's and then running SFT ReF--Rea-Sefff-eneeratte already yields strong jigsaw
19

---

Published as a conference paper at ICLR 2026
0.6
wio Restart
6
Resturt
w/ Restart
w/ Restart
Divergence
Reward
0.4
0.2
2
0.0
0
5000
10000
15000
20000
5000
10000
15000
20000
Training Step
Training Step
Figure 9: Left: Accuracy reward across training steps. Right: KL divergence across training steps.
The dotted line indicates the point we restart the training GRPO with newly initialized model.
Non-Rea
Rea-GRPP-Rolloue Qwen2.5-VL-3B
PPL (Answer Only)
3.0
2.0
20
80
100
Qwen2.5-VL-7B 5--LL-BB
PPL (Answer Only)
2.0
1.0
20
60
80
100
Question index
Figure 10: Answer-only perplexity the numeric sequence (i.e., the J for 100 randomly sampled jigsaw items across three datasets. In the Non Roa-Rnesnning setting, insert an empty thinking tag (<thhnn>>> </think>> immediately before the answer.
20

---

Published as a conference paper at ICLR 2026
The images seem to represent different scenes, including people sitting canves, park setting with line of people, road, and trees. To arrange them logically from left right and top to bottom, should consider the progression of these different scenes.
Thinking:
The sequence of these images needs to follow the logical flow of a meal snack from one item another, starting with the background and moving to the food items.
Thinking:
The images depict scenes feuturing zebras in grassy field. The sequence likely shows various interactions between the zebras, moving around and interacting with the environment.
Thinking:
Thinking:
The given sub-images depict skier mid-run down snowy slope with red flags in the background. We'll arrange these images according to the logical sequence of skiing scene.
Thinking: The images depict a variety of outdoor scenes with objects such as statue, two individuals standing side by side, ivy-covered wall, a sculpture, pink-skinned hand and green hand, trees the background, and a suit jacket. We need to consider... Then, suggest that the last visible object might be part of the backdrop in more formal setting.
Figure 11: Qualitative Results on jigsaw puzzles (test) after training with RFT.
Table 5: Performance of various post-trained models on jigsaw puzzles, grounding, document QA, and general VQA benchmarks.
Model
RFT Steps
SFT Steps
Jigsaw-test
Grounding RefCOCO.t
Document & OCR
General VQA
Hallacitation
3x3 puzzles
DaeVQAuu
MMB
MMStar
GQA
POPE
Qwe.2.5.VL..38
Base
$8.8
92.8
2140
56,2
60,1
86.9
RFT
27,360
(6.1))166)
W8.4 (1004)
2137
(10.55
59.5
86.5
Rollout
4,100
70.0
(1662)
(14.5)
(9.17))
SFT-Rea-GRPO-RBN
27,360
2,670
(120)
84.6()4.)
(13.
2152 (18)
(14.0)
(16.1)
(11.4)
SFT Rea Sell-Geeeeteed
5,472
4,100
(1.0
84.3
90.3 (12.5)
2142 (12)
$2.4 (13.8)
(1574()5.4)
(11.3)
Owen225-V1L.38
Base
0.0
90.0
94.4
2333
62.8
60.4
86.2
RPT
27,360
(170
(9.4
94.4 (10.0)
2325 (18)
(11.7)
60.3 (1.3(
(6.0
SET Rea-do- Rea-do-Rollout
0
4,100
(776)
$2.5 (137.5)
92.1 (12.3)
2084 (1249)
59.1 (13.7)
53.5 (16.9)
54.1 (112.)
Bean--RPO---R Rollout
3,000
(1.01(191)
(1.4
2207
68.4 (12.4
83.1 (33.1)
SFT-Rea-SSe Rea--Slf-Geeerallee
86.0x(44.0)
(10.6)
2256
(12.7)
(13.6)
(21.))
performance and largely preserves old-task scores, while avoiding the severe degradation observed in Rea--4-RRollou
Qualitative jigsaw reconstructions. Fig. showcases representative test instances solved by the RFF-trained model. The predictions display strong global coherence and semantical consistency, indicating that the model finds globally plausible layouts successfully.
GPT-4o Prompt and Response. Fig. 12 shows the prompt elicit GPT-4o "thinking" for the 3 x 3 jigsaw task and representative response.
F MORE DETAILS OF LLM ON MATH REASONING
Math Reasoning Dataset. We use the curated math corpus released by Open-Reasoner-Zerro (Hu training data, and randomly split it into 90%
et al. 2025) as our large-scale reasoning-oriente training and 10% held-out test data. We refer to this held-out split ORZ Test, which serves new target task for post-training. Each example a competition-style math problem paired with a verifiable final answer, without any visual input.
21

---

Published as a conference paper at ICLR 2026
Prompt for generating thinking of jigsaw tasks.
You are g ven nine aub-images labeled 0-8 with encodings (
Sub-image 0: [ image]]
Sub-image 1: [ image]
Sub-image
[image]
Sub-image [ [image]
Sub-image 4: [image]
sub-image 5: [image]
sub-image 6: [ [image]
sub-image [image]
Sub-image [image]) and an optional weak reference
(Reference image (total): [globai.imaee). Write one concise paragraph (no more than 150 words) with planning notes to help human consider how these pieces might fit into 3x3 grid. Focus only on visual eues such as broad scene features (e.g., color bands, horizon-like divisions, sky/grouna contrasts), edge continuities (1ines, textures, colors that could extend between tiles), and plausible adjacency candidates with short justifications using cautious language (e.g., "likely," "possible," "candicate"). the final grid, coordinates, or an ordered list of indices. Do not reveal restate any hidden or original arrangement [[GROUND.TROTH.ORDER))
Do not output
provide descriptive details of any reference image as whole. Treat the optional reference
only as weak hint and prioritize direct, piece-to-pieco edge analysis. The goal is to provide helpful assembly guidance without committing
definitive solution. Keep your output to single paragraph under 150 words without final ordered lists.
Example model response.
To assemble the 3x3 grid, consider visual elements such as the wooden surface, objects on top, and edge alignments. Sub-image 0 features a yellow circular edge that could possibly align with Sub-image 8, given their similar yellow features and adjacent object edges. Sub-image 3, containing onions and chili peppers, seems to share continuity with Sub-image based on onion textures and pepper placements. Sub-image 7, with its mix of objects, could potentially sit near Sub-image due to similarities in scattered textures. The knife-like object in Sub-image 4 might extend toward Sub-image 6, as both show dark, linear edges plausible for alignment. Sub- image 5 has minimal features but might adjoin Sub-image 4 or Sub-image 8. their backgrounds transition smoothly. Evaluate adjacency by matching subtle overlaps between textures and objects, focusing on edges and angles that continue logically across tiles. Prioritize edge features over object-specific assumptions.
Figure 12: Details of employing GPT-4o generate reasoning trajectories for jigsaw puzzles.
22

---

Published as a conference paper at ICLR 2026
GSM8K
LLM-30,
LLM-33. Instruction Following
34-
ORZ-Test:ooee
33.0
-.5
32.5
30.0
ORZ -----ett
ORZ--Test
16-
35-
27.5
34-
25.0
Ros
Ree
22.
Rea-4o Bolout
33
Ree Ap Rullout
Res 4o Rolleat
10
12
54
5:
12
54
15
57
0.55
0.50
0.70
0.75
GSMSK Accuracy
Math Accurney
Instruction Following
LLM-7B, GSMAK
LLM-7B, Math-500
LLM-7B, listraction Following
50
-0
ORZ-Tast Tost Score
45-
R5
65
ONZZ-18 Test
DRZ DRZ- Toe-TTottte
40
Res- GRPO-Rullai
Rea-GiPPORRolle
Ros-to-Rlllott
Ro-ll--ull
Res-lo-Rlllat
GSMSK Accuracy 1
Math-500 Accurncy
Instruction Following
Figure 13: Pareto front curves on the ORZ.test and previous math tasks for models fine-tuned on Rea-4o-Rollout and Rea-GRPO-Rollout data.
LLMs and Evaluation. For math experiments, we use Qwen2.5-3BBIIstteee (Yang et al. 2024) and Qwen2.5-7B-Mnn Instruct as base LLMs. We treat ORZ Test new target task and report answer accuracy on it, and use GSM8K (Cobbe et al. 2021), MATH-500 (Hendrycks et al. (021), and IFEval (Zhou et al., 2023) as prior knowledge to monitor retention of prior math and instruction- following abilities, with their standard accuracy metrics.
Pareto Frontier of SRT- and SFT-Rea-4o-Rollout. We further sweep the learning rate over [1 x 10-6, 5 x 10-5,1 x 10 2 x 10-5} and, for each setting, plot the Pareto- optimal frontier between performance on the new ORZ task and the performance on old tasks (GSM8K, MATH-500, and IFEval) in Fig. 13 Across all learning rates and both 3B/7B scales, SFT training on Seca-GRPO-Ron
consistently achieves a strictly better Pareto frontier than on Rea-4o-Rollout Rollout, yielding either higher ORZ accuracy under similar level of forgetting, or better retention of prior knowledge at comparable ORZ performance.
Perplexity Prior knowledge during SFT. Fig. 14 also shows that training Rea-GRPO- Rollout maintains the perplexity of old math corpora much more stably than Rea-4o-Rollout, these results further corroborate our low-perplexity training hypothesis in Sec. 5.3 and demonstrate that the advantages of Rea-GRPP-Rollown are robust under different optimization hyperparameters.
GRPO Training Recipe. In our initial experiments, followed the default settings of the Hug- gingFace/trl framework, where num_iteration is set to (i.c., the GRPO parameter μ). This means that, by default, T.. and (-)-) are identical during training. In Tab. 6 we report RFT results on math reasoning with different GRPO training recipes. The experiments show that using the standard GRPO recipe and using our recipe leads only minimal differences in model performance on new or old tasks.
GPT-4o Math Prompt and Response. Fig. 13 shows the prompt we use to elicit GPT-4o "thinking" for single-step math problems and a representative response.
G MIXTURE OF REA AND NON-REA DATA FOR SET
We have experimented with mixing data of different styles for SFT. Specifically, we combine reason- ing and non-reasoning data in equal proportion and apply unified prompt template: non-reasoning samples produce an empty CoT followed by the answer, while reasoning samples output a CoT first and then the answer. We run this experiment on both the Jigsaw Puzzles and Math Reasoning
23

---

Published as a conference paper at ICLR 2026
Qwen2.5-LLM-33
Qwen2.5-LLMM7B
1.663
1.660
1.631
1.644
PPL. (MATH-5500)
1.6
1.570
1.602
PPI. (MATH-500)
1.6
1.59
1.61
1,582
1.546
1.5
1,478
1.477
1.385
1.4
1.354
1,356
1.364
1.4
200
400
600
800
1,000
200
400
600
800
1,000
Training steps
Training steps
Rea-GRPO-Rollout
Rea-4o-Rollout
Figure 14: Perplexity versus SFT training steps for MATH-500. We collect rollouts from the base model with question from MATH-500 dataset prior knowledge.
Table 6: Performance comparison of different GRPO training recipe.
Owen2.5-3B-Insieee
Qwen2.5-7B-Instee
Base
RFT (j)
RFT (a+
2)
RET (jc 4)
Base
RFT (a
1)
RFT (jc 2)
RFT (p) 4)
OecnneReasoner-Zeer (test) (New Task)
ORZ Test
21.32
35.00
35.70
34,10
32.11
49,29
48.
47.12
Math Reasoning (Old Tasks)
GSMBk
84,08
83,40
83 02
81.27 $2.80
90.18 14
90.22
90.29
89.69
Math-50h 500
42.40
55.20
55.40
66.60
64.80
65.20
64.00
Instruction Following (Old Tisk)
IFEval
71.58
73.38
74.82
73.98
80.58
80.46
8
80.46
24

---

Published as a conference paper at ICLR 2026
Table 7: Albation results of Rea and Non-Rea data mixture SFT on jigsaw puzzles.
Model
Training Steps
Jigsaw-estt
Grounding
Document &
General GIAA
Hallucination
puzzles
Re(0000.,
DoeVQAAu
MMF
Quen2.5-V1-38
Base
0
$8.8
92.8
2140
56.2
60.1
$6.9
SPT--oo--Be
200
53.0 (153)
6.1 (1828)
81.6 (111.3)
1631 (1,09)
49.2 (97.0))7.))
(15.4)
(5.9
SRTT-Rnn-o--Boe Rellont
74.2 (114.6)
90.3
(1662)
$1.7 (14-5)
50.0 (J10.1)
(117.5
SFT--ReaGGPP-RRal
(170)
(18.2)
89.8 (13.1)
(18)
52.2 (14.0)
(U6.1)
(514))
Mixture
1,307
70.0 (770)
74.0 (114.8)
88.6 (J4.2)
1557 (1588)
46.2 (110.0)
46.8 (113.3)
70.4 (116.5)
Quen2.5.VL-78
Base
0.0
90.0
94.4
2333
62.8
60.4
SFT-Non-Ree
(1577.2)
(127,44
(L1184)
(138.7)
(169,9)
SFT-Ren--o
(175)
52.5 (137,5)
92.1 (12.3)
(1249)
59.1 (15.7)
(16.9)
(112.1)
SYT-Ren-GRPO-Rdoll
3,000
$1.0 (781)
$1.4 (18.6)
93.5 (,0.9)
2207 (1122))
00.4 (12.4)
83.1 (13.1)
SFT-Mixture
1,367
34.0 (194)
26.5 (163.5)
93.2 (11.2)
1992 (1341)
55.7 (47.1)
51.4 (19.0)
75.4 (110.8)
Table 8: Albation results of Rea and Non-Rea data mixture SFT math reasoning.
Training Steps
Open-Reasoner- Ren-onensne-eer
Math Reasoning
Instruction Following
Model
ORZ Test
GSM8k
Math-500
IF
Qwestue--B-Instrucct
Base
21.32
84.08
42.4
71.58
SFT-Non-Rea
1,600
23.40 (12.08)
15.09 (J68.99)
19.4 (123)
64.03 (17.555
SoT-Ret--o-Rollout
2,140
35.36 (114.04)
79.91 (49.17))
50.8 (18.4)
67.99 (L3.59)
SFT-Rea-GRPO-Rolllut
2,140
37.76 (116.44)
83.02 (L1106)
54.4 (112.0)
72.66 (11.08)
SFT-Mixture
2,140
32.33 (111.01)
77.86 (1.6.22)
50.6 (18.2)
72.78 (11.2)
Qwen2.5-7BInntruct
Base
32.11
90.14
66.6
80.58
SFT-Non-Rea
1,600
30.32 (11.79)
(J68.38)
(J40.2)
$7.19 (123.39)
SFT-Rou-4o-Rollout
2,140
45.04 (112.93)
85.82 (J4.32)
57.2 (19.4)
64.39 (116.11)
SFT-Rea-GRPP-Roollatt
2,140
53.44 (121.33)
90.30 (10.16)
66.4 (1.0.2)
79.98 ((00.)
SFT-Mixture
1,070
40.55 (18.44)
72.71 (417.43)
54.2 (112.44
73.26 (L7.32)
datasets. The mixed-data fine-tuning method is denoted as SFT-Mixture in the Tab. 7 and Tab. g The results show that SFT-Mixture performs between SFT-Non Rea and -FT-Rca-4o-Rollout but remains far worse than SFT using model-generated rollout data PeTT-Rc--GROO-RRlA though the latter uses only single fixed reasoning format. This indicates that simply increasing the stylistic diversity of SFT data does not effectively mitigate catastrophic forgetting. The key is to obtain data that better matches the model's own distribution.
even
H LLM EXPERIMENTS ON SCIENTIFIC MULTIPLE- HULTIPLE-CHOICE QA
Scientific MCQ Dataset. We further investigate catastrophic forgetting on scientific multiple- choice questions using the Sci-MCQ4 subset from SciKnowEval (Feng et al.) 2024). From the original corpus, we randomly sample 8,500 examples and split them into 90% training and 10% held-out test data. Each instance is four-choice science question together with its correct option, covering multi-level scientific knowledge such as physies, chemistry, and biology. We refer to this held-out split as Sci-MCQ4 Test, which serves as a target task for post-training.
LLM and Evaluation. For scientific QA experiments, we reuse Srven2.5-3B-Instreee as the base LLM. We treat Sci-MCQ4 Test the new target task and report answer accuracy on it, and we reuse GSM8K, MATH-500, and IFEval as old tasks with their standard accuracy metrics to monitor abilities.
retention of prior math and instructionfollowwiin
Results. As summarized in Tab. g all post-training methods improve performance on the new Sci- MCQ4 task compared to the base model, with (+6.6 points). However, the methods exhibit different degrees of forgetting on old tasks: SFT-Non- Rea suffers the most severe degradation on GSM8K (-14.6 points) and also a drop on IFEval, while SFT with external CoT (SFT-Rea-4o-Rollonn) forgets less but still more than SFT- GEwOG-RPOORRoooo The latter achieves favorable trade-off, boosting Sci-MCQ4 accuracy while keeping GSM8K, MATH-500 and IFEval close to-or even slightly better than the base model. This again establishes a consistent hicrarchy of forgetting severity, iee, Non-Rea > Rea-4o Rea-GRPO, now a scientific QA benchmark.
GRPO-Rollout obtaining the largest gain
25

---

Published as a conference paper at ICLR 2026
Prompt for generating thinking of math tasks.
You will be given a single math problem and the correct final answer.
Problem:
[PROBLEM]
Correct answer:
(ANSWER)
Write one compact reasoning sketch (no more than 300 words) that shows plausible path from the problem statement to that answer. Focus on the main ideas: pattern spotting, substitutions, identifying pecurrences/monotoonic modular reductions, or comparing forms until the result becomes forced. Use cautious language ("Wwe can see,'* "it suggests,' "this leads to'') and keep the tone technical. Do NOT say that the answer was given to you, do NOT restate the answer at the end, and do NOT list alternative answers. The goal is to provide hidden working notes that make the final answer unsurprising.
Example model response.
We consider the nested expression
f(x) - a (a 1 (x+ 2) (r 3) ....
This suggests defining a recursive form f(x) / of(x + 1), so that
f(x)² = 1 xf 1).
We can test a simple polynomial candidate f(x) = x + 1. Then
+ The expressions agree, which indicates that f(x) = x x is consistent with the recurrnnce for all positive integers Since the nested radical is increasing and all terms are positive, this closed form is the stable solution of the recursion. Evaluating 2008 gives f(2008) 2009.
=(t)²=(z++) (z+)² 2x 1+xf(x + = 1=)
Figure 15: Details of employing GPT-4o to generate hidden chain-of-though reasoning trajectories for math problems.
26

---

Published as a conference paper at ICLR 2026
Table Performance comparison across post-trained models of Q the scien-
tific multiple-choice dataset Sci-MCQ4. Numbers in parentheses denote the change w.r.t. the base model.
Steps
Sci-MCQ4
GSM8K
MATH-500
IFEval
Base
65.1
84.1
42.4
71.6
RFT
960
70.8 (15.7)
82.9 (41.1)
45.2 (12.8)
71.8 (10.2)
SFT-Non-Rea
240
69.3 (14.2)
69.4 (J114.6)
43.2 (10.8)
70.3 (41.3)
SFT-Rea-4o-Rollout
214
67.0 (11.9)
74.1 (49.9)
40.2 (42.2)
73.9 (12.3)
S
214
71.7 (16.6)
81.3 (42.7)
40.8 (41.6)
74.5 (12.9)
Qwen2.5-LLMM33
100
Non-Rea
Rea-oo-Rollout
LBKav
Rea-GRPO-Rollee
50
0
0
10
20
30
40
Steps
Figure 16: Evolution of LBK,e during the SFT process with three different datasets the Sci- MCQ4 scientific QA dataset.
Analysis. To probe the underlying mechanism, recompute LBK between post-training samples and prior math knowledge during SFT on Sci-MCQ4. As shown in Fig. [6 Non-Rea data consis- tently exhibit the largest LBK values, whereas Rea-4o contains occasional high-LBK outliers and Rea-GRPO is concentrated in the low-LBK region, further supporting the learning-dynamii anal- ysis in Sec. 5.4 Moreover, Fig. 17 plots the perplexity of Mae-GROO-Rollou and Rea-4o-Rolla Rollout trajectories under the base LLM. Similar to our findings on math reasoning and multimodal jigsaw puzzles, Rea-4o-Rollout tends occupy higher-perplexity regions while Rea-GRPO-Rollout stays closer to the low werpeeeeexity band defined by base rollouts, providing additional evidence for the low-perplexity training hypothesis in Sec. 5.3
GPT-4o ScienceQA Prompt and Response. Fig. 18 shows the prompt we use to elicit GPT-4o "thinking" for scientific multiple-choice problems and representative response.
Qwen2.5 ---M----LLL-BB
6.0
4.5
3.0
1.5P
40
80
100
Question index
Base-16-Rae 16-Rollo (Box)
Rea-GRRFORollouu
Rea-fo-Roljout
Figure 17: PPL of Rea-GRPO-Rodlan and Rea-4o-Rollout trajectories the Sci-MCQ4 dataset under the base LLM. Base-16-BB Rolllout (Box) denotes the PPL range estimated from 16 rollouts generated by the base model, serving reference.
27

---

Published as a conference paper at ICLR 2026
Prompt for generating thinking of Sci-MCQ4 tasks.
You will be given a single math problem and the correct final answer.
Problem:
(PROBLEM}
Correct answer:
{ANSWER)
Write one compact reasoning sketch (no more than 300 words) that shows plausible path from the problem statement to that answer. Focus on the main ideas: pattern spotting, substitutions, identifying modular reductions, or comparing forms until the result becomes forced. Use cautious language (""we can see, " Vit suggests," "this leads to"?) and keep the
technical. Do NOT say that the answer was given you, NOT restate the answer at the end, and do NOT 1ist alternative answers.
The goal is to provide hidden working notes that make the final answer unsurprising.
Example model response.
The question focuses on the role of NADPH oxidase activation in cardiac dysfunc- tion, specifically its primary function. NADPH oxidase key enzyme involved in generating reactive oxygen species (ROS), which are implicated in oxidative stress. Oxidative stress well-established contributor to pathological cardiac remodel- ing and dysfunction. Excessive ROS production can damage cellular components, leading to apoptosis (programmed cell death) and impairing diastolic function by disrupting myocardial relaxation and increasing stiffness.
The activation NADPH oxidase suggests pathological mechanism rather than compensatory beneficial one, oxidative stress typically exacerbates cardiac injury rather than promoting hypertrophy or enhancing systolic function. While hypertrophy may
secondary response stress, the direct link between NADPH oxidase activation and apoptosis aligns with its role in facilitating cellular damage. This indicates that the enzyme's activation is consistent with promoting cardiac apoptosis and diastolic dysfunction, which are hallmarks of oxidative stress- induced cardiac pathology.
Figure 18: Details of employing GPT-4o to generate hidden chain-of-thoughh reasoning trajectories for Sci-MCQ4 problems.
28


Reviews:
[{"review_id": "U7OPVXOQYi", "reviewer": "Reviewer_Ssap", "summary": "This paper investigates how Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) affect prior knowledge retention in Multimodal Large Language Models (MLLMs), using jigsaw puzzles as a novel task. It finds that SFT causes severe catastrophic forgetting despite rapid task learning, whereas RFT learns more slowly but preserves prior knowledge effectively. The key insight is that the data distribution, not the algorithm itself, drives forgetting; RFT naturally generates low-perplexity, model-aligned training samples that interfere less with existing knowledge.", "questions": "- In eq 1, why the author set $\\pi_{\\theta_{\\mathrm{old}}}(\\cdot)=\\pi_\\theta(\\cdot)$, instead of using the standard GRPO recipe? Can you provide the experiment result using the standard GRPO recipe?\n- In the rationale shown in Figure 7, I noticed that the model’s reasoning process differs from how humans typically solve a jigsaw puzzle. As a human, I do begin by identifying the overall scene depicted in the image—similar to the reasoning shown in Figure 7—by observing the misaligned image tiles. However, my step-by-step reasoning primarily involves progressively determining the position of each tile. Once a tile’s position is confirmed, humans usually infer the positions of adjacent tiles based on the edge pixels of the confirmed piece. In contrast, the rationale presented in Figure 7 does not capture this gradual confirmation process. Please analyze the underlying reason for this discrepancy.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The forgetting is less expored in multimodal setting.\n2. Provide theorectical analysis.", "weaknesses": "1. Limited Novelty of Core Insight\nGiven that [1] has already demonstrated that reinforcement learning can effectively mitigate catastrophic forgetting, the contribution of this work appears limited. In fact, [1] also conducted experiments showing that it is the on-policy training paradigm (e.g., RFT) that helps overcome catastrophic forgetting. The key distinction between the data used in SFT and RFT lies in whether they are on-policy or offline. Beyond the discussion in L153–L157, the authors should provide a more detailed comparison between their study and [1]. To me, this paper merely provides a possible explanation for why using on-policy data can help overcome catastrophic forgetting, rather than being the one that discovers this phenomenon.\n- In L55, the paper claims that RFT can “discover novel and useful problem-solving strategies.” However, I find it difficult to draw this conclusion from the preceding discussion about RFT, which merely states that “RFT requires several tens of thousands of training steps to successfully solve jigsaw puzzles.”\n- The paper lacks a detailed description of the construction process of the jigsaw puzzle task, such as the method used to shuffle the image tiles, and it also does not report the detailed training cost of the experiments. This omission makes it difficult for other researchers to reproduce the results.\n\n[1] Rl’s razor: Why online reinforcement learning forgets less\n\n2. Unjustified Focus on Multimodality\nWhile the study is conducted using multimodal models (MLLMs), the analysis and conclusions are not inherently tied to multimodal reasoning. The mechanisms explored—such as data distribution and perplexity—are equally applicable to unimodal language models, raising questions about the necessity of the multimodal framing.\n\n3. Opaque Theoretical Framework\nThe theoretical analysis, based on learning dynamics and neural tangent kernels, is presented with limited intuitive explanation or clear connection to the empirical findings. This obscures how the theory substantiates the main claims and reduces its accessibility to a broader audience.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hp2CxJXL5i", "reviewer": "Reviewer_sRUL", "summary": "This paper systematically investigate the impact of supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) on prior knowledge retention in multimodal large language models (MLLMS) by introducing a jigsaw puzzle task as a novel learning scenario.    This paper present an interesting and important finding: RFT outperforms SFT in avoiding catastrophic forgetting, and further provide a theoretical explanation in terms of data distribution and learning dynamics.", "questions": "1. RFT requires ~ 27k training steps while SFT only takes a few hundred, which may lead to an unfair comparison (different amount of training)?\n2. The paper assumes that eNTK remains stable during training, but it is questionable whether this holds in MLLM fine-tuning?\n3. Is the generation prompt of the \"Rea-4o-Rollout\" data in Section 4 consistent with the inference process generated by RFT?  If not, does it affect the fairness of the comparison?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1、This paper explicitly address a problem that is crucial at the intersection of Continual Learning (CL) and Reinforcement Learning (RL) : how to avoid catastrophic forgetting when adapting to a new task.\n2、Based on the theoretical analysis of Learning Dynamics and eNTK, this paper explain the impact of different data distributions on forgetting from two dimensions of magnitude and direction, which provides a new perspective for understanding the advantages of RFT.", "weaknesses": "1. Although the paper studies \"catastrophic forgetting\", the discussion of existing continuous learning (CL) methods is relatively superficial, and the classical CL methods (e.g., EWC, LwF, ER, etc.) are not fully compared.\n2. While the authors note that resource constraints prevented experimentation with more multimodal models and large language models, experimenting on more methods is key to improving the contribution of this work.\n3. The learning dynamic analysis formulation in Section 5.4 is dense and not friendly enough for readers with a non-theoretical background.\n4. The calculation method of A_(i,t) is not defined in formula (1).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SsBCJ9D0yf", "reviewer": "Reviewer_DcMQ", "summary": "This paper investigates the differences between Supervised Fine-Tuning (SFT) and Reinforcement Fine-Tuning (RFT) regarding catastrophic forgetting. The authors find that RFT learns new knowledge more slowly but significantly reduces catastrophic forgetting, while SFT learns faster but forgets old knowledge more quickly. To study this, the paper introduces a jigsaw puzzle task, which is novel to existing models. The authors analyze how RFT and SFT perform on this task and conclude as above. They also find that using reasoning trajectories generated by the fine-tuned model itself (e.g., Qwen2.5-VL) for SFT—rather than those generated by other models like GPT-4o—greatly mitigates catastrophic forgetting, while also retaining higher training efficiency than RFT. This leads to a balanced approach between learning speed and knowledge retention.", "questions": "- Have the authors experimented with other models or tasks beyond jigsaw puzzles? Even partial results on different setups could strengthen the generality and credibility of the conclusions.\n- Have the authors examined whether using SFT data with more stylistic diversity (e.g., mixing reasoning-based, multiple-choice, direct-answer, or natural-language descriptions of the answer) could also alleviate catastrophic forgetting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a comprehensive comparison between RFT and SFT in both learning efficiency and resistance to catastrophic forgetting, yielding valuable insights.\n- It proposes a method that uses model-generated reasoning trajectories for SFT, achieving both faster learning and stronger retention of prior knowledge.", "weaknesses": "- All conclusions are drawn from experiments on the jigsaw puzzle task using only the Qwen2.5-VL family. This setting is rather narrow and makes it difficult to claim general applicability. Therefore, the current experimental scope may not fully guarantee that the conclusions hold across other models or tasks.\n- Although Rea-GRPO-Rollout converges faster than standard RFT during training, it requires an additional stage of RFT training and reasoning trajectory generation beforehand. Hence, the overall time cost (RFT training + data preparation + SFT) is likely higher, which may reduce its practical appeal.\n- The poor performance of SFT-Non-Rea might partly stem from the overly uniform format of its answers, rather than SFT itself. If the training data included more diverse question and answer styles, the extent of catastrophic forgetting might be reduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)