SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
Authors: Tianzhe Chu
Yuexiang Zhai
Jihan Yang
Shengbang Tong
Saining Xie
Dale Schuurmans
Quoc V Le
Sergey Levine
Yi Ma
Venue: ICML 2025 poster (2025)
Keywords: Generalization
large language model
large vision-language model
post-training
Primary area: deep_learning->foundation_models
TL;DR: SFT Memorizes, RL Generalizes

Abstract: Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and  visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
Tianzhe Chu
Yuexiang Zhai ** Jihan Yang* Shengbang Tong
Saining Xie Dale Schuurmans +
Quoe V. Le*
Sergey Levine" Yi Ma
Abstract
1. Introduction
Although SFT and RL are both widely used for foundation model training (OpenAI, 2023b; Google, 2023; Jaech et al., 2024; DeepSeekAT et al., 2025); their distinct effects on generalization (Bousquet & Elisseeff, 2000; Zhang et al., 2021) remain unclear, making it challenging to build reli- able and robust AI systems. key challenge analyz- ing the generalizability of foundation models (Bommasani et al., 2021; Brown al., 2020) is to separate data mem- orization' from the acquisition of transferable principles. Thus, we investigate the key question whether SFT or RL primarily memorize training data (Allen-Zhu & Li, 2023a; Ye et al., 2024; Kang et al., 2024), or whether they learn generalizable rules that can adapt to novel task variants.
Supervised fine-tuning (SFT) and reinforcement learning (RL. are widely used post-training tech- niques for foundation models. However, their respective role in enhancing model generaliza- tion in rule-based reasoning tasks remains clear. This paper studies the comparative effect of SFT and RL on generalization and memoriza- tion, focusing on text-based and visual reason- ing tasks. We introduce GeneralPointe arithmetic reasoning card game, and also con- sider V-TRI, a real-world navigation environ- ment, to assess how models trained with SFT and RL generalize to unseen variants in both novel textual rules and visual domains. We show that RL, especially when trained with an outcome- based reward, generalizes in both the rule-based textual and visual environments. SFT, in con- trast, tends to memorize the training data and struggles to generalize out-of-distributio in ei- ther scenario. Further analysis reveals that RL improves the model's underlying visual recog- nition capabilities, contributing to its enhanced generalization in visual domains. Despite RL's superior generalization, we show that SFT is still helpful for effective RL. training: SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These find- ings demonstrate the advantage of RL for acquir- ing generalizable knowledge in complex, multi- modal tasks.
an
To address this question, we focus on two aspects of gener- alization: textual rule-based generalization and visual gen- eralization. For textual rules, study the ability of model to apply learned rules (given text instructions) to variants of these rules (Zhu et al., 2023; Yao et al., 2024; Ye et al., 2024). For vision-language models (VLMs), visual generalization measures the consistency of perfor- mance with variations in visual input, such as color and spatial layout, within given task. For studying text-based and visual generalization, we investigate two different tasks that embody rule-based and visual variants. Our first task is GeneralPoints, an original card game task similar to Points22 4 of RL4VLM (Zhai et al., 2024a), which is designed to evaluate a model's arithmetic reasoning ca- pabilities. The model receives four cards (presented as a text description or an image), and is required to compute target number (24 by default) using each card's numer- ical value exactly once. Second, we adopt V- I-IRL (Yang et al., 2024a), real-world navigation task that focuses on the model's 's spatial reasoning capabilities.
Equal contribution HKKU "UC Berkeley, Google DeepMind, *NYU, University of Alberta. All experi- ments are conducted outside of Google. Project page: ttttss//ttanzhechie com SFTvsLL. Correspondence to: Tianzhe Chu <tianzhechué @ gom,, Yuexiang Zhai <si- monzhai @ berkeley.edu>.
We adopt multi-step RL framework similar to Zhai et al. (2024a), by instantiating RL after running SET on
We use "memorization" " the refer a model's 's capacity to gener- ate near-exact copies of training examples when prompted based information present in the training dataset. This definition plicitly excludes bitwise codewise replication of training data within the model itself.
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SFT Memorizes, RL Generalizes
2024a). SFT adapts pre-trained models to downstream tasks by training them on task-specific, often instruction- formatted datasets. Previous work, such as FLAN (Wei et al., 2022a), demonstrates that fine-tuning on diverse instruction-tuning datasets significantly enhances zero-shot performance on unseen tasks. Furthermore, LIMA (Zhou et al., 2024a) shows that supervised fine-tuning acts as a "format teacher" effectively adapting the model's re- sponses to desired format while leveraging the capabil- ities of pre-trained LLMs. In contrast, RL (Ziegler et al., 2019; Ouyang et al., 2022; Sun et al., 2024; Ramamurthy et al., 2023; Abdulhai et al., 2023; Zhou et al., 2024b; Zhai et al., 2024a) has been primarily used to align mod- els with human preferences or training the foundational model to solve specific task (Abdulhai et al., 2023: Zhou et al., 2024b; Zhai et al., 2024a; Chen et al., 2024b). Our work differs from prior studies, as aim to comparatively analyze the generalization and memorization of SFT and RL on both LLM and VLM, while previous studies have focused primarily on only one of these two post-training methods (or only study LLM or VLM) or on only one post- training method.
100
Success Rate (%)
80
60
In- Distribulion
Ousto--Distrbution
SFT
RL.
20
0.0
0.5
1.0
1.5
2.0 2.5
Training Computation (GFLOPS)
1e9
Figure 1: A comparative study of RL and SFT on the vi- sual navigation environment V- IRL (Yang et al., 2024a) for OOD generalization. OOD curves represent perfor- mance on the same task, using a different textual action space. See detailed descriptions of the task in Section 5.1.
the backbone model (Dubey et al., 2024), using the se- quential revision formulation (Snell et al., 2024). In both GeneralPoints and V-IRL, we observe that RL learns generalizable rules (expressed in text), where in-distribution performance gains also transfer to unseen rules. In contrast, SFT appears to memorize the training rules and does not generalize (see Figure for an example). Beyond textual rule-based generalization, we further inves- tigate generalization in the visual domain and observe that RL also generalizes to visual OOD tasks, whereas SFT con- tinues to struggle. As by-product of the visual OOD gen- cralization capability, our multi-turn RL approach achieves state-of-the-ard performance on the V-IRL mini bench- mark, by +33.8% (47.0%)77.88) (Yang et al., 2024a), highlighting the generalization capability of RL. To un- derstand how RL affects the visual abilities of model, we conducted additional analysis on GeneralPoints. revealing that training RL with an outcome-based reward function (Cobbe et al., 2021) improves visual recognition capabilities. Although RL. exhibits superior generalization compared to SFT, we show that SFT is still necessary to stabilize the model's 's output format, enabling RL. to achieve its performance gains. Last but not least, we observe that scaling up the inference time compute by increasing the number of maximal steps leads to better generalization.
Memorization and generalization in LLM/VLM. Sev- eral studies have examined the interplay between memo- rization and generalization in neural networks (Han et al., 2022; Carlini et al., 2022; Yang et al., 2023). In LLMs, memorization can manifest as the model memorizing the training data (Carlini et al., 2022; Jiang et al., 2024; Kang et al., 2024), while generalization reflects the di- vergence between the model's 's output distribution and the pre-training data distribution (Zhang et al., 2023). Prior studies suggest that LLMs exhibit more overfitting on sim- pler, knowledge-intenste
tasks and greater generalization on more complex, reasoning-intent intensive ones (Wang et al., 2024; Qi et al., 2024). For example, recent studies (Ye et al., 2024; Allen-Zhu, 2024; Allen-Zhu & Li, 2023a;b; 2024; Tong et al., 2024b) have demonstrated that LLMs develop reasoning skill sets beyond their training data by pre-computing reasoning graphs before autoregressive gen- eration, which provides compelling evidence of generaliza- tion. Our study takes a different approach by investigating the role of different post-training paradigms on memoriza- tion versus generalization in the context of textual ruled- based and visual variants. We conduct comparative studies in both unimodal (LLM) and multimodal (VLM) settings, and demonstrate that RL leads to better generalization per- formance than SFT.
2. Related Works
Post-training. Post-training is crucial for enhancing model performance (Zhang et al., 2022: Hoffmann et al.,
2023; OpenAI, 2023b; Google, 2023; Touvron et al., 2023). This stage commonly utilizes large-scale supervised fine- tuning (SFT) (Radford et al., 2018; Brown et al., 2020; Radford et al., 2021; Wei et al., 2022a; Chung et al., 2022; Zhou et al., 2024a) and/or reinforcement learning (RL) (Ziegler et al., 2019; Ouyang et al., 2022; Sun et al., 2024; Abdulhai al., 2023; Zhou et al., 2024b; Zhai et al.,
Scaling up inference-time compute. Recent research has increasingly focused on scaling up inference- time com- putation to improve model performance (Wei et al., 2022b; Yao et al., 2024; Snell et al., 2024; Jacch et al., 2024). Early studies (Wei et al., 2022b; Yao al., 2024) prompted
2

---

SFT Memorizes, RL Generalizes
models to generate intermediate reasoning steps and extend the responses before producing final answer. Subsequent work (Zelikman et al., 2022; Feng et al., 2023; Tian et al., 2024; Chen et al., 2024a; Snell et al., 2024) has demon- strated that fine-tuning verifiers during inference improves model accuracy, effectively utilizing test-time computation. Notably, recent findings (Jaech et al., 2024; DeepSeekAl et al., 2025) reveal "scaling laws" for inference-time com- pute, highlighting significant performance gains with in- creased computational resources. Our work builds upon these findings in two ways. First, we integrate insights from inference-time verification into a multi-turn RL formula- tion that allows the model to identify and correct its errors. Second, we examine the impact of inference-time verifica- tion on RL generalization, demonstrating that scaling up inference-time verification (in terms of the maximum num- ber of verification steps) is key for RL to generalize.
m and n are the maximum token length of the input se- quence en and output sequence your For models requiring visual inputs (VLM), we define O as the space of all RGB images. The state space. denoted by S, is defined as S := VM x O for VL.,, and S == V for L.L.M The action space A is defined as A == VR We use VER: 1 - IR x V* to denote a verifier, which evaluates the outcome of y and generates an outcome- reward function (Cobbe al., 2021; Hosseini et al., 2024; Snell et al., 2024; Setlur al., 2024) along with textual information ye Mathemati- cally, at time I, VER(" - (rt,, e\²)). Similar to Zhai et al. (2024a), we treat the model with parameter 0 as our policy network Ty S - V", and adopt PPO (Schulman et al., 2017) as the backbone RL algorithm for updating тT.
Sequential revision. For modeling the state-action tran- sition, we adopt the sequential revision formulation (Snell et al., 2024). Specifically, at time step = O the initial input vo consists of the system prompt. For subsequent time steps (t 2 1). the input prompt w comprises the sys- tem prompt concatenated with all prior model and verifier outputs, denoted by [ugu Uk k==' An illustration of the sequential revision is provided in Figure 2 (also see Figure 5 of Snell et al. (2024)), and an example of the state-action transition is shown in Figure 3.
Improving visual capability in VLMs. While VLMs have demonstrated remarkable skill across a wide range of challenging tasks, such as solving advanced college exam questions (Lu et al., 2023; Yue et al., 2024a;b) and spatial understanding tasks (Yang et al., 2024a;b), they also ex- hibit limitations in visual perception (Zhai et al., 2024a;b; Tong et al., 2024c;d; Rahmanzadehgervi et al., 2024). Prior efforts to enhance VLMs' visual perception include com- bining multiple visual encoders (Tong et al., 2024d; Kar al., 2025; Tong et al., 2024a), curating high-quality SFT data (Chen et al., 2023; Liu et al., 2024; Tong et al., 2024a), and improving the SFT training recipe by unfreezing the vi- sual backbone (Liu et al., 2023; Tong et al., 2024a). While these prior works primarily focus on experiments during the SFT stage, our work demonstrates that RL can also im- prove visual perception.
1-1 b
4. Evaluation Tasks
To evaluate the generalization of different post-training methods, we select two tasks that each offer rule and vi- sual variations. The first task, GeneralPoints, is a new environment we have designed that allows assessment of arithmetic reasoning abilities (Section 4.1). The second task, V-IRL (Yang et al., 2024a), is chosen to examine the model's reasoning capabilities in an open-world visual nav- igation domain (Section 4.2).
3. Preliminaries
Standard RL terminology. We consider finite horizon decision making, and adopt standard notation from the classical RL literature (Sutton & Barto, 2018; Agarwal et al., 2019), where denotes the state space, A denotes the action space, r A + IR denotes the reward function, and T denotes the maximum number of steps per episode. The goal is to learn a policy S + A that maximizes the overall return max E, ET=rr Ti where r, denotes r(se, at). Without loss of generality, we use (aas)) E [0. to denote probability of x choosing a at s.
4.1. The General Points Environment
Our original GeneralPoint environment, instantiated on top of the Point s24 environment (Zhai et al., 2024a), is designed to evaluate generalization of arithmetic reason- ing. Each state s of the environment contains 4 cards, de- scribed as text (in the GP-L variant) or presented as an im- age (in the GP-VL variant); see Figure 2 left for a visual example of GeneralPointe equation that equals a target number (24 by default) us- ing all 4 numbers from the cards exactly once. Detailed examples of the state-action transitions are provided in Ap- pendix A.2. Note that when input from Genera Points is presented in an image (GP-VI.), it naturally introduces additional visual challenges requiring the VLM to recog- nize all cards before solving the equation.
The goal is to produce an
Adapting RL terminology to LLM/VLM with a verifier. We adopt a multi-turn RL setting for foundation model training (Zhai et al., 2024a). Let V represent the discrete and finite vocabulary (token) space. The input and output text spaces are denoted by ym and n respectively, where
3

---

SFT Memorizes, RL Generalizes
Q: Compute 24 using these four cards: [5, 4, 10,
Verifier Info:
wrong calcullation
illegal number used
correct answer
Reward:-1 -1
Reward: -5
Reward: +10
(V)LM
10+7+4+5
(7-4)~10-6
(7-5)
Figure 2: An example of the sequential revision formulation with a verifier. The model generate the next answer U++11 conditioned on all previous answers and information (v,"), 1/t". i S f) from the verifier.
System Prompt (u(")
[Task Description] You are an expert in (task name), you are observing (purely languageevissoo-llangagg imputs + <image>). You are currently [state related info). Please follow [tasks rules).
[Output] Your response should be valid json file in the following format: (task related information and answer)
Appending previous model and verifier outputs to obtain u!"
100 v0 vith ; " 1-11 01
0 concat (v [ug okk )==)))
Model output (witl) and Verifier Output (xtt)
(Task related json outputs), (You success/fail).
in = concat(v 0% 2t
An template of prompt update for constructing v' The brown parts marks the task and related information, and the purple parts denote state (s.) specific info. The blue and red describe the output from model and verifier, respectively.
Figure 3:
Rule variations. To study whether the model learns arithmetic operations or simply memorizes the post-training data, we introduce rule variations in GeneralPoints. These variations consist of interpret- ing the symbols 'J, 'Q', and 'K' either as '11', '12', and '13', respectively, all as the same number '10', These variations ensure a rigorous evaluation of the model's ability to generalize arithmetic reasoning across diverse settings. Each rule is specified as text the input prompt, see the (tasks rules) part in Figure 3, For studying ruled based generalization, we post-train the model using one rule, then evaluate using a different rule.
uses realistic visual input. As in GeneralPoint we consider two versions of the environment, one (V-IRL-L) that consists of pure language deseriptions," and another (V-IRL-VE VL) that includes vision- languanlanguage input. The ma- jor visual challenge in V-IRL involves recognizing differ- ent landmarks from the visual observation' before taking an action. The goal is to navigate to target location by following set of instructions that contain spatial informa- tion. A detailed example of one environment step is shown in Appendix B.2.
Rule variations. To evaluate whether the model pos- sesses spatial knowledge or simply memorizes post- training data, we consider two distinct action space con- figurations. The first variant utilizes an absolute orienta- tion action space, which includes 'southeast', 'south', 'southwest', 'west', 'northwest'). The second variant employs relative orientation action space, containing ( ('left', 'right,, 'slightly left', 'slightly right'). This relative configuration adjusts the current orientation by 90 degrees or 45 degrees to the left or right, respectively. An
Visual variations. The CeneralPoine environment can also be naturally customized evaluate generalization across visual variants. Since the major visual challenge is recognize the number of each card, agnostic the the color of the cards, consider the cards with different col- ors as visual variants of the task. In the visual generaliza- tion setting, we train the model using cards of one color, then test OOD performance using the other color.
{ ('north', 'northeast', 'east',
"The visual input can be parsed into pure text description, see more details in Yang al. (2024a) and an illustration of pure text version Figure
4.2. The V-IRL Environment
While the GeneralPoints environment is designed to assess arithmetic reasoning abilities, we further utilize the V-IRL environment (Yang et al., 2024a) to study spatial reasoning ability an open-wore world navigation domain that
'See Figure 4, the model needs to recognize landmarks like The Dutch, Lola Taverna, and Shuka from the visual observa- tion, and relate these landmarks with the textual instructions for taking the right action.
4

---

SFT Memorizes, RL Generalizes
SSILN
[OBSERVATION] Shuka my right." [ACTION]
Shuka Mediterranea
"Stop."
Lola Taverna Greek
The Dutch
American restaurant
Rince s<
[OBSERVATION]
"See The Dutch on my right." [ACTION]
"Teff turn to northwest."
[OBSERVATION]
*See Lola Taverna on my right." [ACTION]
"Left turn to northwest."
[OBSERVATION]
"Start!"
5th AVe
[ACTION]
"Turn to northeast"
S First, turn slightly right towards the northeast and walk short distance until you reach the next intersection, where you'll see The Dutch on your right. Next, make sharp left turn to head northwest. Continue for a while until you reach the next intersection, where Lola Taverna will be your right. Finally, turn slightly right face northcast and walk short distance until you reach your destination, Shuka, which will be your right.
Figure 4: Demonstration
navigation task in V-IRI. Agent navigates from place to place following the given linguistic navigation instructions V-IRI. navigation procedure shown the top, the navigation instructions displayed below. Visual observation-related information is highlighted in green, while action-related information is marked in orange.
overview of navigation task in V-IRL provided in Fig- ure 4, and detailed state-action transition in V-IRL is provided in Figure 13 (in Appendix B.2).
ing questions. Section 5.1: how does SFT or RL affect the model's generalization different rules? Section 5.2: when the model contains visual component, how does RL/SFT affect its generalization to different visual vari- ants? Section 5.3: how does RL/SFT affect visual recogni- tion capability a VLM? Section 5.4: what role does SFT play RI. training? Section 5.5: how does the number of verification iterations affect generalization?
Visual variations. The key visual challenge V- V-IRL is recognize landmarks from the visual observations (e.g., the green parts in Figure 4). Since the V-IRL environ- ment contains visual observations from different cities, can assess visual generalization in V- IRL by training the model to navigate in one location and then evaluate its per- formance different locations.
5.1. Generalization across Rules
We evaluate the performance of different post-training methods on GeneralPoints and V-IRL, each of which has pure language (-L) and vision-language (-VL) vari- ant, and each encompassing rule variations. For each task, we separately scale the training compute for RI and SFT on single rule. We consider the results on the trained rule as in-distribution (ID) performance, whereas results on the unseen rules measures out-of-distribution (OOD) general- ization. In GeneralPoints the ID case treats all J., 'Q', 'K' 10, and the OOD cases interprets them as 11, 12, and 13. As for V-IRL, the ID case adopts the absolute orienta-
5. Results
In this section, present experiments that investigate the generalization abilities induced by post-training with (Dubey
RL and SFT. We adopt 1aama-3.--Vision-11 et al., 2024) as the backbone model. Following the standard pipelines
RLHF (Ouyang et al., 2022) and RL4VLM (Zhai et al., 2024a), we initialize the model with SFT before running RI. We specifically study the follow-
5

---

SFT Memorizes, RL Generalizes
GP-L
V-IRL-L
GP-VL
V-RRL-L
100
100
55
93
In-distribution
52
80
28
77
14
69
0.0
1.0
1e10
1e9
1e10
1e10
iittof-ddstributii
65
14
37
6-
33
19
0.0
0.5
1.0
1e10
1e10
1e10
Training Computation (GFLOPs) Initialization
pure language
SFT
with vision
Figure 5: Success rate (%) - GFLOPs trendlines for RL and SFT on GeneralPoints and V-IRL. The top row shows in-distribution performance, while the bottom row shows out-of-distribution performance. Results are presented for both pure language (-L) and vision-language (-VL) variants of each task. For GeneralPoints,, episode success rate, while for V-IRL, report per-step accuracy with overall success rate in Figures and 18. Detailed evaluation setups (and curve smoothing) are provided in Appendix C.3.
report the
tion coordinate system and the OOD case uses the relative orientation action space. Other details and additional ex- perimental setup can be found in Appendix C.
mance on the red suits )) For V- IRL, we train the model on routes collected in New York City and evalu- ate on the original V-IRL VLN mini benchmark (Yang et al., 2024a) containing routes from various cities world- wide (see Appendix B.1 for details). Note that the rules remain consistent across experiments in this section.
RL generalizes, SFT memorizes. As illustrated in Fig- ure 5, RI. consistently improves OOD performance on all tasks, including both unimodal (LLM) and multi-
RL generalizes in visual OOD tasks. As shown in Fig- ure 7, we observe that RI still generalizes in visual OOD tasks, while SFT continues to suffer. Specifically, in GP-VL and VIRL-VI, RL achieves performance improve- ments of +17.6% (23.6% + 41.2%), +61.1 % (16.7% + 77.8%), whereas SFT suffers from performance decreases of -9.9% (23.6% - 13.7%) and -5.6% (16.7% + 11.1%). As byproduct of this visual OOD study, we also show that our multi-turn RL formulation improves the state-of- the-art results (see Table 5 of Yang et al. (2024a)) on the V-IRL mini benchmark by +33.8% 8% (44.0% + 77.8%). Notably, unlike the previous state-of-the-art approach re- ported in V-IRL, which relies on stage VLM-LLM collaboration technique and tailored prompt engineering closed-sourced model (OpenAI, 2023a), our end-to-end RL approach enables an open-sourced model (Dubey et al., 2024) to reach superior performance.
modal (VLM). Specifically, Figure demonstrates that
RL achieves an increase of +3.5% on GP-L (11.5% 15.0%) and +11.0% V-IRL-L (80.8% 91.8%).
Even with the additional challenge of visual recognition in the VLM, RL maintains consistent performance improve- + 14.2%) on GP-VL and +9.3% %
ments of +3.0% % (11.2% (35.7% + 45.0%) on V-IRL-VL, respectively. In con- trast, SFT consistently exhibits performance degradation across all OOD evaluations on all tasks: -8.1% on GP-L (11.5% 3.4%), -79.5% V-IRL- (80.8% + 1.3%). -5.6% % (11.2% + 5.6%) GP-VL, and -33.2% (35.7%-. 2.5%) on V-IRL-VI.
5.2. Generalization in Visual Out-of-Distribution Tasks
Section 5.1 demonstrates that RL yields generalization across rule variations, whereas SFT exhibits the opposite trend. Since VLMs also incorporate a visual modality, we next study the effects of visual variation in OOD general- ization. For GeneralPoint s, we train the VLM using the black suits and test duttif-diitribution perfor-
5.3. RL Improves Visual Capabilities
Building upon the above observation that VLMs trained with RL generalize to visual OOD tasks (Section 5.2), we
6

---

SFT Memorizes, RL Generalizes
GP-L
VIRL-L
GP-VL
V-IRL-VL
RL
15.0%
91.8%
14.2%
45.0%
Init
11.5%
80.8%
11.2%
35.7%
SFT
3.4%
1.3%
5.8%
2.5%
0
3
6
9
12
15
0
20
40
60
80
100
0
8
12
16
20
10
20
30
40
50
OOD success rate (%)
Comparison of out-of-distribution performance under rule variants. We report the success rate for GeneralPoint and per-step-aceuracy for V-IRL. For each subplot, RL and SFT are trained with equal computa- tion, and their shared initial checkpoint (marked as Init) is set as baseline. Detailed setups are provided in Appendix C.3.
Figure 6:
GP-VL
V-IRL-VL
V-OOD Success Rate (%)
41.2%
80
80
77.8%
40
60
60
30
30
23.6%
20
20
13.7%
20
16.7%
10
10
11.1%
2
SFT
Init
RL
2.5
5.0
7.5
SFT
Init
RL
GFLOPs
1e10
GFLOPs
1e10
Initializztion
SFT
Previous SOTA
Figure 7: Comparison of outto--dist performance under visual variants. Similar to Figures and 6, we present both the performance dynamics (shown as lines) and final performance (shown bars) for visual out-of-distribution evaluations. The previous state-of-the-art on V-IRL VLN mini benchmark (Yang al., 2024a) is marked in orange. Detailed evaluation setups (and curve smoothing) are provided in Appendix C.3.
consider a natural follow-up question: How does RL af- fect VLMs' visual capabilities? To study this question, we conducted additional ablation studies in the GP-VG EV-VL envi-
5.4. The Role of SFT for RL Training
Despite the superiority of RL in generalizing the model's reasoning and visual capabilities, as discussed previously, the experimental pipeline still instantiates RL after SFT. In this subsection, we focus on another key question: Is SFT necessary for RL training? To answer this question, we conduct additional experiments that directly apply end- to-end RL to post-train the base model Llama3.2 using General P nts s in the purely language case (Figure 9).
ronment to investigate the OOD performance of RL and SFT, along with the model's visual recognition accuracy, in terms of recognizing the 4 cards from the input image. In particular, we study how scaling post-training compute via RL/SFT both affects generalization in uuc-bbsed OOD (Figure left), and visual recognition accuracy and visual OOD (Figure right).
SFT is necessary for RL training when the backbone model does not follow instructions. Figure 9 shows that without SFT, all end-to-end RL runs fail to improve. More specifically, we observe that without SFT, the base model suffers from poor instruction following capability. A detailed failure case is provided in Figure 20 (in Ap- pendix D.3), revealing that the base Llama-3.2 Vision- model tends to generate long, tangential, and unstructured responses. This issue makes it impossible retrieve task- related information and rewards for RL training. Note that due to the difference in backbone model, our results do not contradict with DeepSeekA1 et al. (2025), which suggests that SFT is unnecessary for downstream RL training.
Scaling RL improves visual recognition accuracy in VLM training. As shown Figure 8, we observe that the VLM's visual recognition accuracy largely affects the overall performance, which was similarly observed in Zhong et al. (2024). In addition, scaling up RL compute also improves visual recognition accuracy, as byproduct of its generalization capability, while scaling SFT deteri- orates both visual recognition accuracy and overall per- formance. Additional experimental results are provided in Figures 16 and of Appendix D.I.
7

---

SFT Memorizes, RL Generalizes
Rule Variants
Visual Variants
Recognition Accuracy (%)
Computation (GFLOPs)
80
90
8e10
Te10
70
70
6e10
50
50
4610
30
20 30
40
30
50
GP-VL Success Rate (%)
Out-of-dietibbtioon
SFT
RL
In-distribution
Figure 8: Recognition success rate for RL and SFT under different variants in GP-VI We report both in- distribution (red) and OOD (blue) performance of recognition (y-axis) and episode success rate (x-axis). We denote the training compute of cach data point via transparency (color bar) while connected (x-o) pairs are evaluated using same checkpoints. As scaling up post-training compute, RL. improves both recognition and overall accuracy, while SFT shows opposite effect.
0.4
Success Rate (%)
1
In-distribution Growth (%)
12
Computation (GFLOPs)
0.3
2
5
10-
0.2
0.0
0.0
0.2
0.6
0.8
1.2
Computation (GFLOPs)
1e10
3e9
Figure 9: RL experiments GP-L without SFT initial- ization. All trials fail due to poor instruction following capability of the base model.
Out-of-distriuttion Growth (%)
Figure 10: In-distribution OOD performance growth GP-L. We record RL experiments with different num- ber of verification iterations (VIter) scaling up training compute (color transparency).
5.5. Role of Verification Iterations
Verification serves as another crucial component in our multi-step training and evaluation pipeline (see Figures 2 and 3). To validate its necessity and better understand its effect, we conduct RL experiments with different verifica- tion iterations 11,3,5 5, 10} using GP-L (Figure 10).
experiments on the GeneralPoints and V-IRL tasks, demonstrated that RL exhibits superior performance in learning generalizable knowledge, while SFT tends to merely memorize the training data, across both the rule and visual variations. This phenomenon consistently oc- curs across multimodal arithmetic and spatial reasoming ca- pabilities. addition, we studied effect of RL on vi- sual recognition, the role of SFT, and the role of verification steps. During our study, two challenges were resolved.
Scaling up verification improves generalization. In Figure 10, we observe that RL generalizes better with more verification steps. More specifically, under the same computational budget across all experiments, we observe improvements of +2.15% (3 steps), +2.99% (5 steps), +5.99% (10 steps).
case with one verification step, we only observe marginal improvement of +0.48% % in OOD performance improvement.
contrast,
Failure of SFT on GP-VL. In Figure 5 for VL. observe that SFT fails to achieve comparable in- distribution performance with RI. To mitigate the vari- ance introduced by hyperparameter choices, we addition- ally conduct 10 more experiments with different learning rates and tunable components (Figure 16), none of which exhibits strong increasing trend like RL (Figure 17).
6. Conclusion, Discussion, and Limitations
In this paper, we present a comprehensive analysis of the generalization effects of foundation model post-training techniques, specifically RL and SFT. Through extensive
8

---

SFT Memorizes, RL Generalizes
Given our observation that scaling up SFT degrades visual recognition capabilities (Figure 8), we hypothesize that SFT locally overfits to reasoning tokens while neglecting recognition tokens, possibly due to the higher frequency of reasoning tokens (see Figure 11 as example). We leave fur- ther investigation to future work.
inforcement learning: Theory and algorithms. CS Dept. UW Seattle, Seattle, WA, USA, Tech. Rep, 32, 2019. 3
Allen-Zhu, Z. ICML 2024 Tutorial:
Physics of Lan- guage Models, July 2024. Project page: https:// physics. a allen- whw..o//.........
Allen-Zhu, Z. and Li, Y. Physics language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316 2023a. 1,2
Limits of RL in corner cases. As discussed in Sec- tion 5.4, SFT necessary for effective RI. training on Llama-3.2. We investigate applying RL an overly-tuned SFT checkpoint. As demonstrated in Figure 19, RL is un- able to recover out-of-distribution performance when start- ing from such checkpoint. Example failure cases are il- lustrated in Figure 21, where the model collapses to the training rule. These results, together with findings in Sec- tion 5.4, indicate that RL has limited effectiveness when applied to extremely underfit or overfit initial checkpoints. Further research is needed to delineate the conditions under which SFT facilitates effective RL.
Allen-Zhu, Z. and Li, Y. Physics of language mod- cls: Part 3.2, knowledge manipulation. arXiv preprint arXiv:2309. 14402, 2023b. 2
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.05405, 2024.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, Bosse- lut, A., Brunskill, E., al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. 2021.
Impact Statement
This paper presents work aimed at advancing the field of Machine Learning. While the study includes tasks such as GeneralPoints s, which is synthetic environment, and V-IRL, real-world map simulator, our work is confined to controlled research settings. The V-IRL environment is designed as a simulated proxy for real-world tasks, but no deployment or interaction with actual real-world systems or data was involved. The methods, environments, and tasks investigated in this study were constructed to advance our understanding of model generalization without introducing any foreseeable societal ethical implications.
Bousquet, O. and Elisseeff, A. Algorithmic stability and generalization performance. volume 13, 2000.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sas- try, G., Askell, A., et al. Language models few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020. 1,2
Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F,, and Zhang, C. Quantifying memorization across neu- ral language models. arXiv preprint arXiv:2202.07646, 2022. 2
Acknowledgements
YZ would like to thank Xiaoxuan Feng for beautifying Fig- ure We would like to thank Jincheng Mei and Doina Pre- cup for feedbacks on earlier manuscripts. Yi Ma would like acknowledge support from the joint Simons Foundation- NSF DMS grant #2031899 the ONR grant N00014-22-L 2102, the NSF grant #2402951, and also support from and the HKU startup, the Hong Kong Center for Construction Robotics Limited (HKCRC) Award 052245, and JC Club of Hong Kong.
Chen, G., Liao, M., Li, C., and Fan, K. AlphaMath al- most zero: Process supervision without process. arXiv preprint arXiv:
2024a. 3
Chen, J., Han, X., Ma, Y., Zhou, X., and Xiang, L. Un- lock the correlation between supervised fine-tuning and reinforcement learning in training code large language models. arXiv preprint arXiv:2406. 10305, 2024b.
Chen, L., Li, J., Dong, X., Zhang, P., He, C.. Wang. J., Zhao, F,, and Lin, D. ShareGPT4V: Improving large multi-moda. models with better captions. arXiv preprint arXiv:231 / 12793, 2023. 3
References
Abdulhai, M., White, I., Snell, C., Sun, C., Hong, J., Zhai, Y., Xu, K., and Levine, S. LMRL Gym: Benchmarks for multi-tumn reinforcement learning with language models. arXiv preprint a
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling cnttuuttionfinetunan language models. arXiv preprint arXiv:22 10. /416, 2022.
2023. 2
Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. Re-
9

---

SFT Memorizes, RL Generalizes
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110.14161 2021. 2,3
Conference on Computer Vision, pp. 113-132. Springer, 2025. 3
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved base- lines with visual instruction tuning. arXiv preprint arXiv22310.05744, 2023.
DeepSeekAI et al. DeepSeek-RI: Incentivizing rea- soning capability in LLMs via reinforcement learning, 2025. URL https: /ttss///axiii.e org/ abs/ 12948. 3,7
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. LLaVA-NeXT: Improved rea- soning, ocr, and world knowledge, 2024. URL https://llava-vl..
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 Herd of models. arXiv preprint arXiv:2407.22 1783, 2024. 2,5, 6
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng. H., Chang, K.-W., Galley, M., and Gao, J. Math- Vista: Evaluating mathematical reasoning of foundation models visual contexts. ICLR, 2023,
Feng, X., Wan, Z., Wen, M., McAleer, S. M., Wen, Y., Zhang, W., and Wang, J. AlphaZero-like tree-search can guide large language model decoding and training. arXiv preprint ar arXiv:2309.1711
2023. 3
OpenAL. GPT-4, 2023a. URL https:tpp// http://pppaaa...emmeee com/ research/gpt
Google, D. Introducing Gemini: Our largest and most capable AI model, 2023. URL https://blog http://booooccooleeeeeeeoooooo// wtwee//eeehhooooll./... ai..
OpenAl. GPT-4 technical report. arXiv, pp. 2303-08774, 2023b. 1,2
Ouyang, I.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. 2,5
Han, J., Zhan, H., Hong. J., Fang, P. Li, H., Petersson, L., and Reid, What images are more memorable to machines? arXiv preprint arXiv:2211.07625 2022. 2
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E.,
Qi, Z., Luo, H., Huang, X., Zhao, Z., Jiang, Y., Fan, X., Lakkaraju, H., and Glass, J. Quantifying generalization complexity for large language models. arXiv preprint #rXiv:2410001669, 2024.
Cai, T., Rutherford, E., Casas, D. d. L.,, Hendricks, L. A., Training compute-optimal
Welbl, J., Clark, A., large language models. NeurlPS, 2023. 2, 18
Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-STar: Training verifiers for self- taught reasoners. First Conference on Language Mod- eling, 2024. URL https: /oopnrevieeeeeeeeeeeeee net/ forum? dwm.m:.ddssmmmsss2
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et Learning transferable visual models from natu- ral language supervision. In International conference on machine learning, pp.
Jaech, A., Kalai, A., Lerer, A., Richardson, A., El-Kishky, A., Low, A., Helyar, A., Madry, A., Beutel, A., Car- ney, A., et al. OpenAl ol system card. arXiv preprint 1xXi:22412116720 2024. 2,
PMLR, 2021. 2
Rahmanzadehgervi, P., Bolton, I.., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vi- sion, pp. 18 18-34, 2024.
Jiang. M., Liu, K. Z., Zhong, M., Schaeffer, R., Ouyang, S., Han, J., and Koyejo, Investigating data contami- nation for pre- pre-train language models. arXiv preprint arXiv:2401. 240.00999.. 2024. 2
Ramamurthy, R., Ammanabrolu, P., Brantley, K., Hessel, Sifa, R., Bauckhage. C., Hajishirzi, H., and Choi, Y. Is reinforcement learning (not) for natural language pro- cessing: Benchmarks, baselines, and building blocks for natural language policy optimization. In The Eleventh International Conference on Learning Representations, 2023. URL https: : openreview net forum? i id=BaHzds2ubyB. 2
Kang. K., Setlur, A., Ghosh, D.,. Steinhardt, J., Tomlin, C., Levine, S., and Kumar, What learning dynamics reveal about generalization LLM reasoning? arXiv preprint 1tXXi::2411.07666 2024. 1.2
Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Za- mir, A., and Tombari, F. Brave: Broadening the vi- sual encoding of vision-language models. In European
10

---

SFT Memorizes, RL Generalizes
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 17XXv:1777.06447 2017. 3, 18
Wang. X., Antoniades, A., Elazar, Y., Amayuelas, A., Albalak, A., Zhang, K., and Wang, W. Y. Gener- alization vs memorization: Tracing language models' capabilities back to pretraining data. arXiv preprint arXiv:2407. 14985, 2024.
Setlur, A., Nagpal, C., Fisch, A., Geng, X., Eisenstein, J., Agarwal, R., Agarwal, A., Berant, J., and Kumar, A. Re- warding progress: Scaling automated process verifiers for LLM reasoning. arXiv preprint wrXiv:2410.08/44 2024. 3
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester. B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. International Conference on Learning Representa- tions, 2022a. URL https:/ openzevieeeeeeeeeeeeeeeeeeeeeeee net/ forum? ferummid==oe -d-.EgdGGoEggg....... 2
Snell, C., Lee, J., Xu, K., and Kumar, A. Scaling LLM test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.03314,
2024. 2, 3, 18
Wei. J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837, 2022b. 2
Sun, 7., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L., Wang. Y.-.., Yang, Y., Keutzer, K., and Darrell, T. Aligning large multimodal models with fac- tually augmented RLHF. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13088- 13110, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/112222. findings-acl. 775. URL https: haclanaaaonnooy..././ org/ 2024. mandings-aal.... 775.
Yang. J., Ding, R., Brown, E., Qi, X., and Xie, S. V-IRL: Grounding virtual intelligence in real life. In European conference on computer vision, 2024a. 1, 2, 3, 4, 6, 7, 13, 14, 15
Yang. J.. Yang, S., Gupta, A. W., Han, R., Fei-Fei, L., and Xie, S. Thinking in space: How multimodal large lan- guage models see, remember, and recall spaces. arXiv preprint arXiv:2412.1417/1 2024b. 3
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. MIT press, 2018. 3
Tian, Y., Peng, B., Song, L., Jin, L., Yu, D., Mi, H., and Yu, D. Toward self self-immprvement of LLMs via imagination, searching, and criticizing, arXiv preprint 1rXiv:2404112253. 2024. 3
Yang, Z., Lukasik, M., Nagarajan, V., Li, Z., Rawat, A. S., Zaheer, M., Menon, A. K., and Kumar, S. ResMem: Learn what you can and memorize the rest. In Thirty- seventh Conference
Neural Information Processing Systems, 2023. URL https: http://weeeeeeeeeeeeeeeeeeeeeeee forum? (aeiiid=FFFFAAuA.. OFAyNuqq. 2
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., et al. 1:mbrian-1: A fully open, vision-centric exploration of multimodal LLMs. In NeurIPS, 2024a. 3
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. 1,
Tong. S., Fan, D., Zhu, J., Xiong. Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruc- tion tuning. arXiv preprint arXiv: 14164, 2024b.
2
Ye, T. Xu, Z., Li, Y., and Allen-Zhu, Z. Physics language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint arXiv:2407.2 203111 2024. 1,2
Tong. S., Jones, E., and Steinhardt, J. Mass Mlass-prouucin fail- ures of multimodal systems with language models. In NeurIPS, 2024c. 3
Yue, X., Ni, Y., Zhang. K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Y., et al. MMMU: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In CVPR, 2024a. 3
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? Exploring the visual shortcomings of multimodal LLMs. In CVPR, 2024d. 3
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix. T., Rozière, B., Goyal, N., Hambro, E., Azhar, F, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arX1v:2302.13997. 2023. 2
Yue, X., Zheng, T., Ni, Y., Wang, Y., Zhang, K., Tong, S., Sun, Y., Yin, M., Yu, B., Zhang, G., et al. MMMU- Pro: A more robust multi- colti-discipline multimodal under- standing benchmark. arXiv preprint a4X1v22409.02813 2024b. 3

---

SFT Memorizes, RL Generalizes
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476- 15488, 2022. 3
Zhai, Y., Bai, H., Lin, Z., Pan, J., Tong, S., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., and Levine, S. Fine- tuning large vision-language models decision-making agents via reinforcement learning. In The ehirty-eigtth Annual Conference on Neural Information Process- ing Systems, 2024a. URL https: /owpnrevee...eeeeeeeee net/ forum? id-nB 2IU. 1, 2, 5,
Zhai, Y., Tong, S., Li, X., Cai, M., Qu, Q.. Lee, J., and Ma, Y. Investigating the catastrophic forgetting in mul- timodal large language model fine-tuning. Confer- ence on Parsimony and Learning, pp. 202-227. PMLR, 2024b.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning (still) requires rethink- ing generalization. Communications of the ACM, 64(3): 107-115, 2021.
Zhang, C., Ippolito, D., Lee, K., Jagiclski, M., Tramèr, F., and Carlini, N. Counterfactual memorization neural language models. Advances in Neural Information Pro- cessing Systems, (6:39321-39367) 2023.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M.. Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language mod- els. arXiv preprint arXiv:2205. 01068, 2022.
Zhong, M., Zhang, A., Wang, X., Hou, R., Xiong, W., Zhu, C., Chen, Z., Tan, L.,, Bi, C., Lewis, M., et al. Law of the weakest link: Cross capabilities of large language 2024.7
models. arXiv preprint arXiv:2409.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. LIMA: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024a. 2
Zhou, Y., Zanette, A., Pan, J., Levine, S., and Kumar, A. ArCHer: Training language model agents via hierarchi- cal multi-turn RI. arXiv preprint arXiv:2402. 19446, 2024b.
7hu, 7., Xue, Y., Chen, X., Zhou, D., Tang, J., Schuurmans, D., and Dai, H. Large language models can learn rules. arXiv preprint arXiv:2310.07064 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Rad- ford, A., Amodei, D., Christiano, P., and Irving, G. Fine- tuning language models from human preferences. arXiv preprint Xtx::1999.08599 2019.
12

---

SFT Memorizes, RL Generalizes
A. Details on the General Points Environment
shift experiments (Section 5.2), train the model black suits + and evaluate out-of-domain performance red suits
In this section, we demonstrate the design details for GeneralPoints mentioned in Section 4.1. We first present the data used for this environment (Appendix A. Then, we show examples of the environment's transition dynamics (Appendix A.2), followed by description of key arguments and reward design specification (Appendix A.3),
Reward design. An episode terminates when either correct equation is generated or the maximum verification step of 5 is reached. The reward function is as follows:
A.1. Data
r For generating legal equation that equals the target point
GeneralPoint card quadruples sampled from deck of 52 standard poker cards. Each sampled quadruple guaranteed to have at least one solution equals the target point, Lee 24. We ensure this by using an expert solver during the sampling process.
= 1: For legal equations using each card once but not equaling the target point
r = 1: For exceeding maximum verification step
A.2. Detailed Examples on the Transition Dynamics
2: For legal equations containing numbers among the given choices
As shown in Figure and Figure 12, we treat the system prompt as vi and then subsequently appending future outputs vie and verifier info wi into the prompt for get- ting the t + 1 output. Figure 11 provides example with the visual inputs, while Figure 12 shows the language only case.
-== For all other illegal equations
In the vision-language variant (Genera1Points-VV) additional penalty of -1.5 is applied when the agent fails to correctly recognize the given cards.
A.3. Additional Ectails on the Environmental Design
Arguments. The GeneralPoints environment sup- ports the following configurable arguments:
B. Details on the V-IRL Environment
Similar to Appendix we present the design details for V-IRL discussed in Section 4.2. First, we introduce the database used for this environment (Appendix B.1) and demonstrate transition examples (Appendix B.2). We then describe the environment by explaining its funda- mental component-route. Finally, we outline our mod- ifications and reward design choices made to adapt the original V-IRL for reinforcement learning training (Ap- pendix B.3).
Target point: Any positive integer
Face cards rule: Two options
- 'J. 'Q', and 'K' all count as '10'
'J. 'Q' and 'K' count as '11,, '12', and '13' respec- tively
Card sampling: Two options
Sample cards without replacement from deck of poker cards
B.1. Data
Sample at least one card from 'J, 'Q', and 'K'
Leveraging the data collection pipeline of Yang al. (2024a), construct training database with 1000 unique routes from New York City. We evaluate all rule-variant experiments and visual in-distribution experiments using randomly sampled routes from this database. For visual out-of-distribution experiments, we directly adopt the VL.N mini benchmark from Yang et al. (2024a). This benchmark consists of 18 distinct routes across nine cities: Milan, New Delhi. Buenos Aires, London, Hong Kong, New York," Melboume, Lagos, and San Francisco, with routes per city.
Card color: Three options
Black suits only:
Red suits only:
All suits:
For all experiments, fix the target point at 24. In Fig- ure 5, training and in-domain evaluation use the rule where face cards count as '10'. For out-of-domain evaluation, we use the alternative face cards rule and require at least one face card, forcing calculations with numbers above 10 that are not encountered during training. For visual distribution
"These NYC routes in the VLN mini benchmark do not lap with our training data.
13

---

SFT Memorizes, RL Generalizes
System Prompt (v%")
[Tass Description] You are an expert 24 points card game player. You are observing these four cards in the image. Note that ', 'Q', and 'K' count as '10', and each card must be used once. Your goal is to output formula that evaluates to 24 using numbers from the cards and operators such as '4', V. ",, Y. ', Y, and '='
[Output]
Your response should valid json file the following format:
"cards": [x. w]. where '. 'Q', and 'K' count as '10:. "number": [a. b, dJ. where b. c. and d the numbers the cards, "formula": "am equation that equals 24",
Appending model and verifier outputs to obtain v.""
o²"" [00" 1o 0i 1.. ..... 01-1: 22-1)
vi concat (vo [e wh Tk=o)
Model output (vt
"cards": ['A', '3, 'K,, '6).
"number": [1 3, 13, 6).
"formula": 1(1+2)*3+1==24",
Verifier Output (x)
You failed this trial because your formula incorrect.
b o = concat (ut", ot tt
An example of prompt update for constructing v using wit vt and vt".. This example provides an optional vision input for VLMMs, adding visual recognition challenge. The brown parts marks task and related information, and the purple parts denote the state (st) specific info. The blue and red deserihe the output from the model and verifier, respectively.
Figure 11:
B.2. Detailed Examples on the Transition Dynamies
Oracle information: Expert observation data for each movable point
We provide detailed transition examples of the V- F-IRL en- vironment in Figure 13 (vision and language) and Figure 14 (pure language).
Expert trajectory
Instruction
B.3. Additional Details on the Environmental Design
Although the instructions in Figures 4, 13 and 14 are pre- sented different formats, they convey equivalent infor- mation, with Figure 4 using natural language.
Concept of route. The route serves the fundamental navigation object in the I-LRL environment. As illus- trated in Figure 4, each route corresponds to a real-world path with associated language instructions and visual sig- nals. Using Figure 4 as an example, a route comprises:
Simplification and arguments. We simplify the original V-IRL design from Yang et al. (2024a) to better accom- modate RL training. The modifications include eliminating the 2-stage navigation pipeline that required separate vi- sual detector for street view processing, and removing line queries to reduce training time and cost. Our IRLL environment contains 2 additional configuration arguments compared with the original design:
Destination: Shuka
Starting point: Start
Turning points:
The Dutch, Lola Taverna
Roads connecting turning points, start- ing point, and destination
Straight road:
Action space: two options
Street views: 360-degree panoramic views at each movable point
Absolute direction:
"turn direction(x)" where XE ('north'' 'northeast',
14

---

SFT Memorizes, RL Generalizes
System Prompt (v)))
[Task Description] You are an expert 24 points card game player. You are observing these four cards in the image. Note that ', 'Q' and 'K' count '11,, 12', and '13' respectively, and each card must be used once. Your goal is output formula that evaluates 24 using numbers from the cards and operators such 't' ", 14 Y, ', Y, and '=!.
[[nput]
Cards: I'A', 3', 'K', '61
[Output]
Your response should be valid json file in the following format:
"cards": [x, y; Z, w). where 'J, 'Q', and 'K' count '10,, "number": [a, b, c, d]. where a, b, c, and are numbers on the cards, "formula": "an equation that equals 24",
Appending model and verifier outputs to obtain v!"
o,"" (0g vo vi v 2t- 1t
et" concat (vi), [e 22 Ik=o)
Model output (u(
"cards": [AA], 33, 'K, '6].
"number": [1 3, 13, 6].
"formula": "(1+6)"3+1==24"
Verifier Output (n²**)
You failed this trial because your formula incorrect.
of concat ot o
An example of our prompt update for constructing o using vi ot and et This example provides an optional vision input for VLMs, adding visual recognition challenge. The brown parts marks the task and related information, and the purple parts denote the state (st) specific info. The blue and red deseribe the output from the model and verifier, respectively.
Figure 12:
'east', 'southeast', 'south', 'southwest', 'west', 'northwest'), "forwardO", "stop("
Reward design. An episode terminates when either the navigation agent stops at the destination or the maximum verification step of reached. The reward function is as follows:
Relative direction:
"turn direction(x)" where xE''left', 'right, "slightly left', 'slightly right'), "forward()", "stop("
1: For generating correct action at the current coordinate
Maximum straight road length: any positive integer
r -1: For generating wrong action at the current coordinate
The action space argument accommodates the rule variants described in Section 4. For experiments shown in Fig- ure 5, we use absolute direction action space during train- ing and in-domain evaluation, while using alternative rule for out-of-domain evaluation. We implement a maxi- mum straight road length to limit the number of movable coordinates between turning points, preventing sequences of repetitive "forward() actions. We conduct visual dis- tribution shift experiments (Section 5.2) via training the model on New York City regions and evaluating the out-of- domain performance on the worldwide navigation routes from the benchmark released by Yang al. (2024a).
r -1: For exceeding maximum verification step
-1.:: For failed detection of landmarks
15

---

SFT Memorizes, RL Generalizes
System Prompt (v%))
[Task Description]
You are an expert in navigation. You will receive a sequence instructions to follow while observing your surrounding street views. You are also provided with your observation and action history in text. your goal is to take the action based on the current observation and instruction.
[Instruction]
First, turn left to face east.
2. Move forward until you reach the next intersection where Hotel 320ne is on your right behind.
3. Turn right face north.
Move forward until you reach the next intersection where Dragon Gate Chinatown SF is on your right front.
5. Turn left to face cast.
6. Move forward until the destination Café de la Presse is on your right.
[Current observation]
You observe 2x2 grid of street view images with following headings: [front, right
back, left]
You need to identify if any of the landmarks in the instruction visible in the street view grid.
[Action space]
"forward()": indicates moving forward step:
"turn directionnx)".: indicates turn direction to the target heading, where xE nel'no,th' 'northeast', 'east', 'southeast',
'south', "southwest", 'west", 'northwest']:
"stopO": indicates the navigation is finished;
[Observations and actions sequenee]
O_1: landmarks nearby;
A_1: turn directionncast)
0_2: No landmarks nearby;
A forward()
0_3: No landmarks nearby;
A_3: forward()
4: You observe an image of 4 views; You observe an intersection
A_4:
[Output]
Your response should be valid json file in the following format:
"current observation": latest observation from the street view grid,
"current instruction": analyze the full instruction and identify the sentence to be executed, "action": the action to be taken chosen from the action space,
Appending model and verifier outputs to obtain v.." 0
[00". v0 vi vi
27-11 v
e concat (ui), [e ((()))
Model output (u
"current observation": "Hotel 32One is on my right behind; I observe an intersection", "current instruction": "Turn right to face north.". "action": "turn_direction" (oortcctiiooooott)"""""
Verifier Output (n)"))
Correct solution.
Move to the next scene upon correct action
Figure 13: An example of our prompt update for constructing e/t, using vi v. and o, This example provides optional vision input for VLMs, adding visual recognition challenge. The brown parts marks the task and related information, and the purple parts denote the state (st) specific info, The blue describe the output from the model and verifier, respectively.
16

---

SFT Memorizes, RL Generalizes
System Prompt (ut")
I'Task Description]
You are an expert in navgation. You will receive a sequence of instructions to follow. You are also provided with your observation and action histroy in Your goal first analyze the instruction and identify the next Ience executed. Then, you need to provide the action to taken based on the current observation instruc- tion.
[Intrrcction]
First, turn left to face cast.
2. Move forward until you reach the next intersection where Hotel 320ne is your right behind.
Turn right to face north.
4. Move forward until you reach the next intersection where Dragon Gate Chinatown SF is on your right front.
5. Turn left to face cast.
6. Move forward until the destination Café de la Presse is your right.
[Action space]
"forward()":: indicates moving forward step;
"turn direction(x)": indicates turn direction to the target heading, where xC(north', 'northeast', 'east', 'southeast',
'south', 'southwest', 'west,, 'northwest]::
"stopO": indicates the navigation is finished;
[Observations and actions sequence]
I: No landmarks nearby:
I: turn direction(east)
O 2: No landmarks nearby;
A_2: forward()
0_3: No landmarks nearby;
A_3: forward()
0_4: Hotel 320ne is on your right behind; You observe an intersection
A_4:
[Output]
Your response should be valid json file in the following format:
"current observation": latest observation from the street view grid,
"current instruction": analyze the full instruction and identify the sentence to be executed, "action": the action to be taken chosen from the action space,
Appending model and verifier outputs to obtain v."
[0 vo vi ..... 21 v]=1]
e concat (ui), [e (())))
Model output (u!"))
"current observation": "Hotel 32One is on my right behind; I observe an intersection", "current instruction": "Turn right to face north.". "action": "turn_dircae dineciiicction"oott)."
Verifier Output (n)"r)
Correct solution.
Move to the next scene upon correct action
Figure 14:
An example of our prompt update for constructing v using vi ot and ot The brown parts marks the task and related information, and the purple parts denote the state (s.) specific info. The brown parts marks the task and related information, and the purple parts denote state (s.) specific info. blue red describe the output from model and verifier, respectively.
17

---

SFT Memorizes, RL Generalizes
C. Experimental Setup
ize the model with SFT, then separately scale up the com- pute for SFT and RL (Schulman et al., 2017), starting from this initialized model. For all experiments of SFT and RL in the main body, we tune all components using shared learning rate per experiment. All training experiments are conducted on an H800 machine (80GB).
This section details the experimental setup used in Sec- tion 5. We first describe our data collection setup for su- pervised fine-tuning (Appendix C.I). Then, we present the training pipeline (Appendix C.2). Finally, describe our evaluation metrics and the statistical tools used for gener- ating plots (Appendix C.3).
C.3. Evaluation Metric
C.1. Data
Per-step accuracy. We report the per-step accuracy for V-IRL-VL task in Figures and An individual step is considered correct when the model's chosen action matches the expert trajectory at that position. Note that intermediate verification steps are counted independent samples here.
SFT data collection. As illustrated in Figures 11 to 14, GeneralPoint. and V-IRL environments naturally align with prompt-response dialogue structures. We cre- ate training samples by pairing each system prompt with its corresponding expert response. All SFT experiments in the main body use optimal single-turn prompt-response pairs,
Success rate. We report the success rate (%) of GP-L. GP-VI, V-IRL- and V-IRL-VI in Figures 5 and 6. In the Genera Points task, success is defined as succeed- ing at least once during the inference time verification. In the V-IRL task, a sample is recorded as success when the model takes correct action at each movable point on the route.
without any verification revision steps.
SFT on sub-optimal trajectories To examine how more diverse SFT data affects the out-of-distribution perfor- mance of SFT, we conduct an ablation study on GP-L us- ing sub-optimal trajectories as training data. Unlike ex- pert prompt-response pairs, these sub-optimal trajectories include errors and verification messages in their prompts. This format aligns with evaluation scenarios where multi- ple verification iterations are allowed, similar to the data being used for the downstream RL training. In Figure 15, we observe that SFT still merely memorizes the training data with degraded out-of-distribution performance. This evidence suggests that memorization occurs due to the fun- damental nature of SFT training rather than the SFT data.
Computation estimation. We estimate the FLOPS for training X following the similar manner of (Snell et al., 2024; Hoffmann et al., 2023), where Xtrain = GNDannnn and Xinference = 2NN Dinference Here, N represents the model parameters and Derain represents the number of to- kens during training. Suppose our SFT and RL experients starts from checkpoint trained Dimit tokens, we estimate the training computation of SFT and RL via the following equations:
100
XSFT (DN(Dini snit + DSFT)
Success Rate (%)
80
X 6Dit + DrL.) 2N Dbuffer
80
Oul-of-dittrBulian
Note that the used on-policy RL algorithm PPO (Schulman et al., 2017) contains iterative stages of replay buffer collec- tion and optimization, hence requiring additional inference computation. For simplicity, we approximate the term via:
In-distributin
Ed,do
Douffer fer 73
D
DRL
Computation (GFLOPs)
1e9
=DRL
Figure 15: SFT experiments on GP-L with suboptimal trajectories. Similar to results in Figure 5, SFT overfits the training data even we increase the trajectory diversity.
where E E N denotes the number of auto-regressive gen- cration processes, d,, do denote average input tokens and output tokens. We estimate the for Genera i nts and IRL as 6 and 5.1 respectively after calculation.
C.2. Training Pipeline
Line smoothing and error bar. All line plots in our pa- per adopt Savitzky- Golay filter with polynomial order 3 as smoothing function. We assume each evaluated data point
As illustrated in Section 5, we follow the training pipeline by RLAVLM (Zhai al., 2024a), where we first initial-
18

---

SFT Memorizes, RL Generalizes
All Components Tunable
Freeze Vision Encoder
Freeze Vision Encoder and Adapter
30
1e-7
1e-7
1e-7
25-
Se-7
1e-6
25
Se-7
Succass Rale (%)
1e-6
1e- 8
20
1e-5
20-
20
10-4
15-
15-
6-
5-
0-
1e10
1e10
1e10
Training Computation (GFLOPs)
Ablation studies on GeneralPoints-vz SFT. We ablate the learning rate and report the in-distribution episode success rate (%) of all experiments. None of the experiments shows an increasing trend beyond 30% success rate.
Figure 16:
follows a binomial distribution and approximate the stan- dard error using V P(1-P) where P is the demical success rate and N is the number of samples.
1e-6
50
Success Rate
40
D. A Additional Experimental Results
35-
30
In this section, we provide additional experimental results that are not covered in the main body.
20
2.8
3.0
34
Training Computation (GFLOPs)
1e8
D.I. Ablation Studies on GP-VL
As mentioned in Section 6, we observe an abnormal phenomenon that SFT fails to achieve comparable in- distribution performance with RL (see Figure subplot row column 3). To further explore this, we conduct ablation studies over different hyperparameter choices.
Figure 17: Ablation studies on GeneralPoints-Vi
RL. Echoing Figure 16, we ablate the learning rate and me- port the in-distribution episode success rate (%) of the two experiments. All components are tunable here.
success rate is significantly more demanding metric since it aggregates per-step errors. For example, random policy achieving 10% per-step accuracy would achieve achieve only approximately 10 10-8% success rate enough routes averaging 10 steps in length.
SFT. We ablate the hyperparameter choices under the same task setting of GP-VL in Section 5.1. For experi- ments fine-tuning all parameters, we search learning rates from {1x 10-4, x 10-4, x 10-5 1,1,,xx x 10-6, 5x 5, 10-7). Freezing the vision encoder, we search learning rates {1 x 10-6, x 10-7}. Freezing vision encoder and adapter, we search learning rates (1 x 10-6, 5 10-7, 1 10-7]. We provide the in-distribution success rate curve in Figure 16.
7, 1
D.3. Failure Cases
In this section, we present 2 failure cases in our experi- ments as mentioned in Sections 5.4 and 6.
Without SFT, RI fails. In Figure 9, we present the train- ing dynamics of failed RL experiments without SFT initial- ization. We additionally provide output examples of these experiments in Figure 20, where the model tends to gener- ate unstructured response and fail.
RL. Finding suitable hyperparameters for RL experi- ments requires minimal effort. We conduct search over learning rates 2 x 10-6, 1 x 10-6, with the in-distribution success rate curves shown in Figure 17, All parameters are tunable in our RL experiments.
RL cannot save overfitted checkpoints. As shown in Figure 19, RL cannot recover the nut-of-distribution performance when initialized from extremely overfitted checkpoint that has an initial per-step accuracy of less than 1%. We additionally provide an output example in Fig- ure 19, where the model fails to adjust to the new rule.
D.2. More results on V-IRL-VL
Echoing per-step accuracy results in Figure 5, we report the overall success rate of -II-RLL in Figure 18. Due to the task's ' complexity, both training methods achieve over- all success rates no higher than 1%. For IRL, the overall
19

---

SFT Memorizes, RL Generalizes
V-IRL-VL
V-IRL-VL OOD
1.5
1.50
16-7
1.25
Per Per-step Per-steppAaaae Aczuracy (%)
Success Rake (%)
1.00
1.0
40
0.75
30
0.50
20
0.5-
0.25
0.00
2
3
2
3
GFLOPs
1e10
GFLOPs
1e10
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Intialization
SFT
RL
Training Computation (GFLOPs)
169
Figure 18: Overall success rate (%) GFLOPs for V-IRL-VI under rule variants. Due the nature of the task requiring aggregating a trajectory of correct ac- tions, neither training method achieves reasonable out- of-distribution performance.
Out-of-distribution per-step accuracy (%) GFLOPs for V-IRL-VL under rule variants with overfitted initial checkpoint, Evaluation metric details can be found in Appendix C.3.
Figure 19:
System Prompt (v)")
I'Task Description] You are an expert 24 points card game player. You are observing these four cards in the image. Note that 'J, 'Q' and 'K' count '11', '22,, and '13' respectively, and each card must be used once. Your goal is output formula evaluates 24 using numbers from cards and operators such as 't', 1. * 'C '=!.
[Input]
Cards: [2' T, 33', K']
[Output]
Your response should be valid json file in the following format:
{
count as '10', "number": [a. b, c, d], where a, b, c, and d are the numbers on the cards, "formula": "an equation that equals 24".
"cards": [x, y, z, w], where T, 'Q',
Appending model and verifier outputs to obtain r
ttth [00 00 vi ver out
13
ot = concat (vb [o oo (())))
Failed model output (t
To solve this problem, we
brute force approach by generating all possible combinations of the cards and checking any of them can used form an equation that equals 24.
Here is a Python solution for this problem: "'python
def calculate_ calculate_formulaccace formula(cardss):
# Define the mapping of cards their corresponding numbers
Verifier Output (e)
You failed this trial because your formula incorrect.
b o = concat out
Figure 20: Example model outputs without SFT initialization. We record model responses using prompts similar the one shown in Figure 11. The results demonstrate that Llama-3.2- Vision- IB fails to follow instructions properly. We omit the long response which tries to solve the puzzle via code but fails to finish within finite context length.
20

---

SFT Memorizes, RL Generalizes
System Prompt (vo")
[Task Description]
You expert in navigation. You will receive sequence of instructions fol- low while observing your surrounding street views. You are also provided your observation and action history text. Your goal first analyze the instruc- tion and identify the next sentence to be executed. Then, you need to provide the action be taken based on the current observation and instruction. [Instruction]
First, turn right to face northwest.
2. Move forward until you reach next intersection where Korean War Memorial is on your left.
3. Turn left face southwest.
4. Move forward until you reach next intersection where Korean War Memorial on your left behind.
Turn right to face north.
6. Move forward until you reach next intersection.
7. Turn left to face east.
8. Move forward until you reach next intersection.
Turn left to face north.
10. Move forward until you reach next intersection.
11. Turn right to face east.
12. Move forward until you reach next intersection.
13. Turn left face northeast.
14. Move forward until you reach next intersection where 9/11 Memorial & Museum on your left.
15. Turn right to face northwest.
16. Move forward until you reach destination where The destination 9/11 Memorial & Museum is your right front.
[Action space]
"forward()": indicates moving forward one step "turn_ "irr__ddrcctoo()).. indicates adjust the ego agent direction towards direction. x could be any following ['left', 'right', 'slightly left', 'slightly right']
"stop(": indicates the navigation is finished.
[Observations actions sequence]
I: You observe an image of views:
A 1:
[Output]
"current observation": latest observation from the streetview grid, "current instruction": analyze the full instruction and identify the sentence to be executed, "action": the action to be taken chosen from the action space,
Appending model and verifier outputs to obtain e." thth
[00 out
vo
out ver ...., our v
vt = concat (vō [b]... bh (k=)))
Failed model output (u!th)
"current ohservation": "No landmarks nearby", "current instruction": "Tum right to face northwest.",
"action": "turn_ cian_ddicction(nottwwss))
Verifier Output (o)
Incorrect action.
b Fail sample.
Figure 21: Failed example of V-IRL transition due to overfitting. This phenomenon happens more frequently during scaling up supervised fine-tuning.
21


Reviews:
[{"review_id": "hZtXHgqrss", "reviewer": "Reviewer_rx4i", "summary": "This study compares the effects of supervised fine-tuning (SFT) and reinforcement learning (RL) on the post-training of foundation models, particularly in terms of generalization and memorization. It introduces two tasks -- GeneralPoints and V-IRL -- to evaluate how these techniques influence model performance in rule-based reasoning and visual domains. The results indicate that RL enhances generalization across both textual and visual tasks, while SFT tends to lead to memorization of the training data and struggles with out-of-distribution generalization. Further analysis reveals that RL improves the model’s visual recognition capabilities. Despite RL’s good generalization, SFT is found to be necessary for stabilizing the model’s output format. Additionally, the study shows that increasing the number of verification iterations during inference time improves RL’s generalization capability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Response\nDear reviewer rx4i,\nThank you for your appreciation of our work. We are glad that you find our work comprehensive, well-designed, and recognize our SOTA results on V-IRL.\n\nRegarding your concerns, we provide the following feedback.\n> Q1. Suggestion on more diverse experiments for larger impact\n\nWe agree with you that more diverse experiments will increase the impact of our work. For this purpose, we provide additional experiments in three folds:\n- Diversity on initial checkpoints: we provide additional experiments on [Qwen-2.5-VL-3B](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), where **“SFT memorizes, RL generalizes” still holds for Qwen-2.5VL-3B.** Specifically, RL achieves an increase of +3.45% on OOD while SFT causes a drop of 8.48%. Experimental settings are the same as our original paper except we train fewer steps due to time constraint. See more detailed curves in Figure 22 in [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ).\n- Diversity on training data: we conduct extra SFT experiments on diversified trajectories generated by both RLed checkpoints (Figure 24 in [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ)) and synthetic approaches (Figure 15 in [original paper](https://openreview.net/pdf?id=dYur3yabMj)). In all three settings, we observe decreasing trends in OOD performance. Meanwhile, distillation on RLed checkpoints demonstrates faster in-distribution performance increase with less out-of-distribution degradation compared to the original SFT experiments.\n\n> Q2 Regarding readability and theory\n\nThanks for your suggestions! We will make sure to update the manuscript, to provide a better introductory paragraph at the beginning of the experiment section for better readability. Regarding theory, we are actually actively investigating the theoretical explanations for such differences between RL and SFT, thanks for pointing out this insightful direction!\n\nThank you again for your careful review and appreciation of our work. If you find our feedback alleviates your concerns, feel free to improve your rating accordingly, and once again we thank you for your appreciation and insightful feedback!", "claims_and_evidence": "The claims in the submission are supported by experimental evidence. For instance, in the GeneralPoints task, RL achieved a +3.5% improvement in OOD performance, while SFT showed an -8.1% degradation. In the V-IRL task, RL improved OOD performance by +11.0%, whereas SFT experienced a -79.5% drop. These results suggest that RL enhances generalization across rule-based reasoning and visual tasks, while SFT may lead to memorization and struggle with out-of-distribution generalization. Additionally, the study found that RL improved visual recognition accuracy in VLMs, contributing to better overall performance. The authors also highlighted the necessity of SFT for stabilizing model outputs, which is crucial for effective RL training.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate for comparing SFT and RL in terms of generalization and memorization. The tasks and environments cover various reasoning and visual capabilities. The metrics used, such as success rate, per-step accuracy, and computational resources, offer a comprehensive assessment of model performance.", "theoretical_claims": "The submission does not include formal proofs or theoretical claims that require verification of mathematical correctness. The focus of the study is on empirical evaluation through experimental methods and analysis.", "experimental_designs_or_analyses": "The experimental designs and analyses are sound and valid. The tasks and environments are suited for evaluating generalization and memorization, and the metrics used provide a comprehensive assessment of model performance.", "relation_to_broader_scientific_literature": "The contributions are related to prior research on post-training techniques, generalization, and visual capabilities. It investigates how SFT and RL affect model performance in rule-based and visual tasks, emphasizing the necessity of SFT for effective RL training. The study also explores the impact of scaling up inference-time computing on RL generalization, and shows that RL can enhance visual recognition in VLMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros：\n1. The study provides a comprehensive comparison of SFT and RL in terms of generalization and memorization, offering valuable insights into their respective strengths and limitations.\n\n2. The introduction of GeneralPoints and V-IRL tasks is well-suited for evaluating rule-based reasoning and visual generalization, effectively testing models' ability to generalize beyond training data.\n\n3. The study demonstrates state-of-the-art performance on the V-IRL mini benchmark, showcasing the effectiveness of the proposed RL approach.\n\nCons:\n1. The study's focus on specific tasks (GeneralPoints and V-IRL) may limit the broader applicability of the findings, suggesting a need for more diverse experiments.\n\n2. The paper includes many technical details, which might be challenging for readers unfamiliar with the field. Simplifying some sections and providing more intuitive explanations could enhance readability.\n\n3. This is a purely experimental paper. Although it provides some experimental analysis, it lacks theoretical analysis, which is my main concern. I hope the authors can provide some theoretical analysis, e.g., why RL is superior for generalization than SFT ?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See Strengths And Weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fHNAUFjypj", "reviewer": "Reviewer_Dv4H", "summary": "This paper compares supervised fine-tuning (SFT) and reinforcement learning (RL) on both textual and visual reasoning tasks. The authors introduce GeneralPoints, an arithmetic reasoning card game, and V-IRL, a real-world navigation environment, to evaluate model generalization to unseen variants involving novel textual rules and visual domains.\n\nThe main results are:\n- RL with outcome-based rewards, generalizes well to OOD scenarios in both two tasks, while SFT tends to memorize training data and struggles with OOD generalization.\n- SFT serves as a useful initialization step for RL by stabilizing output formats and providing a solid foundation for effective RL training.\n- RL training enhances the model’s underlying visual recognition abilities, contributing to better generalization in visual tasks.\n\nOverall, the paper presents comprehensive experiments demonstrating that SFT is fundamental of stable RL training, and RL is effective at improving model's generalization in complex, multimodal reasoning environments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Response\nDear reviewer Dv4H,\n\nThank you for your appreciation of our work. We are delighted to hear that you find our research original, significant, and clear. We provide the following feedback and additional experiments for your concerns:\n> Q1. Regarding Verifier design and PPO configuration\n\nWe appreciate your carefulness on this point and we are with you that reward shaping and RL hyperparameters matter a lot in training. \n\nWe provide the detailed reward & verifier design in Appendix A.3 and B.3 for GeneralPoints and V-IRL respectively. Specifically, we adopt rule-based rewards for these two tasks, where the model receives positive rewards if and only if it correctly solves the problem. For different failure cases, we intuitively set up different negative rewards as punishment. We implement a very simple verifier for all our settings, where the verifier function takes in string output responses and outputs different verification information based on rewards. You may kindly refer to Figure 2 for examples.\nWe use a shared PPO configuration for all the experiments:\n\n| Parameter | Value |\n|------------|-------|\n| clip_param | 0.1 |\n| ppo_epoch | 4 |\n| value_loss_coef | 0.5 |\n| entropy_coef | 0.01 |\n| max_grad_norm | 0.01 |\n| gamma | 0.9 |\n| gae_lambda | 0.95 |\n\nWe did not put much effort on engineering reward shapes and PPO configuration, as we directly adopted the PPO configuration from RL4VLM [1].\n\n> Q3. Are there experiments on other models?\n\nYes, we conduct additional experiments on [Qwen-2.5-VL-3B](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), an up-to-date SOTA MLLM from a different family. We adopt the same training setting as our original paper and plot the performance dynamic on Figure 22, [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ). We observe that **“SFT memorizes, RL generalizes” still holds for this model**. Specifically, RL achieves an increase of +3.45% on OOD while SFT causes a drop of 8.48%. Due to resource constraints, we are unable to conduct experiments on larger models (>=32B) during the rebuttal period.\n> Q4. Deeper analysis?\n\n> Q4.1. on visual capabilities\n\nInsightful suggestion! We also find that there’s a lack of analysis on the visual encoder after being binded with the LLM, or furtherly trained by RL. We will explore this direction in our future work.\n\n> Q4.2. on alleviate the problem of overfitted checkpoints via reshape the rewards\n\nInteresting point! The answer may not be answered by simple yes or no but we have some thoughts on it. \n\nRemind that we design the reward purely based on outcomes. Considering an overfitted checkpoint with 100% ID acc and 0% OOD acc, the checkpoint will neither receive diverse reward signals nor be effectively updated during RL training as its response leads to uniform positive rewards. We think reshaping the values of reward functions does not help much in this case. On more general tasks or non-extreme cases, we believe that a careful design of reward functions will encourage exploration and benefit the generalization.\n\n> Q5. Regarding captions in figure 13 and 14 and literature to be cited\n\nThank you so much for pointing this out! We will reorganize the figures and captions in our next revision. We read the paper by Wang et al., 2024 and also found it related to our research. We will cite it in our future revision as well.\n\nWe would like to thank the reviewer again for insightful and careful suggestions, if you feel like our response and additional results further improve the quality of our work, please feel free to improve your rating, thank you very much in advance! \n\n> References\n\n[1] Zhai et al. \"Fine-tuning large vision-language models as decision-making agents via reinforcement learning.\" Advances in Neural Information Processing Systems 37 (2024): 110935-110971.", "claims_and_evidence": "The key claims are convincingly supported by clear experimental results and analyses. The authors are transparent about limitations, and most claims are well validated.  \n\nOne potential caveat is that the claim “RL generalizes” holds under certain boundary conditions. The authors acknowledge that RL struggles to recover from overly-tuned SFT checkpoints (Section 6), indicating that RL’s generalization ability can be limited when applied to extremely underfit or overfit initial checkpoints.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-designed and appropriate for the problem at hand. The paper introduces two custom-designed tasks aimed at testing generalization:\n\n- GeneralPoints, an arithmetic reasoning card game, includes both rule variations (to assess rule-based generalization) and visual variations (to assess visual generalization).\n\n- V-IRL, a real-world navigation environment, involves complex spatial reasoning and visual recognition, with both action-space rule variations and visual distribution shifts.\n\nThe evaluation metrics—success rates for GeneralPoints and per-step accuracy for V-IRL—are well-defined and directly measure out-of-distribution generalization. In addition, the authors conduct scaling analyses (varying compute budgets and verifier iterations), making the evaluation more comprehensive and robust.", "theoretical_claims": "There are no theoretical claims or proofs to check in this submission. The paper’s contributions are empirical and experimental rather than theoretical.", "experimental_designs_or_analyses": "While the experimental design is overall solid and well-motivated, I have several suggestions for improvement:\n- The paper does not provide sensitivity analyses with respect to RL reward shaping, or PPO configurations.\n- The paper heavily relies on a verifier-based reward signal but provides limited details on the verifier’s architecture. It is unclear whether the verifier is rule-based, learned, or manually engineered.", "relation_to_broader_scientific_literature": "The paper contributes to the broader scientific literature in several ways:\n\n- Post-training techniques and generalization:\n\nPrior works on SFT emphasize its role in improving model abilities.Similarly, reinforcement learning (RL) has been used for model alignment and human preference optimization. This paper extends the literature by systematically comparing SFT and RL in the context of generalization vs. memorization, filling a gap where most previous works only focus on one method or one modality.\n\n- Scaling inference-time compute and verification:\n\nThis paper demonstrates that scaling verification iterations in RL training improves OOD generalization, providing further confirmation of inference-time compute scaling laws.\n\n- Contribution to model interpretability:\n\nThe paper contributes to the broader discussion on model interpretability and reliability. Understanding the distinct roles of post-training techniques helps to know how generalization comes.", "essential_references_not_discussed": "The paper discusses the difference between memorization and generalization in large language models (LLMs), but does not sufficiently reference Wang et al., 2024 (“Generalization vs memorization: Tracing language models’ capabilities back to pretraining data”). This work systematically analyzes the relationship between model capabilities and pretraining data, providing direct theoretical support for the paper’s key conclusion that SFT tends to encourage memorization, while RL promotes generalization.", "other_strengths_and_weaknesses": "Strengths\n\n- Originality:\nThe paper presents a systematic comparative study between SFT and RL on both text-based and visual reasoning tasks. \n\n- Significance:\nThe work addresses an important  question in foundation model research — how SFT and RL respectively contribute to generalization. The results offer valuable insights for designing post-training strategies in large-scale multimodal models. \n\n- Clarity:\nThe paper is well-structured, with clear explanations, and informative figures and tables. The design decisions, and experimental setups are well-documented.\n\nWeaknesses\n\n- Limited scope of applicability:\nWhile the authors demonstrate that RL benefits from SFT initialization, this finding is tested only on a single backbone (Llama3.2-Vision-11B). It remains unclear whether the same observations hold for other architectures and domains.\n\n- Lack of deeper interpretability analysis:\nAlthough the paper shows that RL improves visual recognition capability, it remains unclear whether this stems from changes in the visual encoder representations or purely from downstream policy optimization. More analysis at the representation level could strengthen this claim.", "other_comments_or_suggestions": "I have no additional comments or suggestions.", "questions_for_authors": "- Your experiments are conducted on Llama3.2-Vision-11B. Have you observed similar SFT vs. RL generalization patterns on other model families or scales? \n- You mention that RL cannot recover OOD performance when starting from an overfit SFT checkpoint. Could alternative RL reward shaping or curriculum design alleviate this problem?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gjVmsC79gL", "reviewer": "Reviewer_NgMz", "summary": "This paper studies the generalization of RL and SFT. \nIt uses two visual-language reasoning tasks, and shows that RL has better generalization and SFT mainly memorizes the training samples and struggles with the OOD samples. \nFurther analysis shows that RL can also improve the model's underlying visual recognition capabilities.\nDespite RL's superior generalization, SFT is still helpful in stabilizing the output format.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Response\nDear reviewer NgMz,\n\nThank you for your appreciation of our work, especially the importance of our studied problem. We also acknowledge your constructive feedback on improving the simplicity of our works. Here is our feedback:\n\n> Q1. Do RL and SFT start from different checkpoints?\n\nA quick clarification is that – while our RL starts from a SFTed LLama checkpoint (say `ckpt A`) for warmup, we still continue the future SFT from `ckpt A`. So we believe our experiments are still fair, as we scale up RL and SFT training flops from a SFTed LLama checkpoint.\n\nFor more details, say in the top left picture in Figure 5, we started both RL & continue SFTed on the second leftmost point, which have been SFTed for 1.6e9 GFLOPs, and the dotted curves are the SFTed warmup flops. This consistent initialization ensures a fair comparison between the two approaches. The performance of these base models (`ckpt A`) is recorded as \"init\" in Figure 6, providing a clear baseline for measuring the relative improvements from each method.\n\n> Q2. Concerns about experimental complexity and task selection\n\nWe appreciate your suggestion for a simpler setup.\n\nThe sequential revision framework is chosen to align with multi-turn reinforcement learning framework. Our experiments also involve the case without sequential-revision demonstrated in Section 5.5, Figure 10, noted as VIter 1. Generalization is also observed here.\n\nThe rule-based decision-making tasks allow us to conduct controlled experiments via switching different rules. Math problems, though well-studied, remain difficult to design out-of-distribution scenarios. We thank you for this insightful point and will continue to figure out rigorous ID/OOD tests for these tasks in future works.\n\n> Q3. Effect of model sizes & different training steps\n\nThanks for the suggestion and we do more experiments:\n\n- **RL generalizes holds across different model sizes**: We conduct experiments on [Qwen-2.5-VL-3B](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), where we find that **“SFT memorizes, RL generalizes” still holds for this model**. See results in Figure 22 in [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ).\n- **RL generalizes holds across checkpoints of different initializations**: We provide results of two starting checkpoints, initialized by different amounts of SFT. We observe that **RL consistently increases OOD performance**. See results in Figure 23 in [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ).\n\n> Q3.1. Is the evidence of SFT related to overfitting?\n\nYes, memorization exactly equals to overfitting in our work. The early improvements on ID tests directly support the evidence, where the model rapidly memorizes and overfits the training data while forgetting out-of-domain knowledge. \n\n> Q4. Concerns about OOD improvements\n\nWe appreciate your concern on the absolute OOD performance. The purpose of our study is to compare the behavior of SFT and RL instead of pursuing high performance increases. We believe “RL generalizes” holds as we observe decent percentages of ID increases transfer to OOD increases for RL with details provided in the table:\n| Task | GP-L | V-IRL-L | GP-VL | V-IRL-VL |\n|------|------|---------|-------|----------|\n| ID increase |+15.3%|+15.0%|+27.4|+3.29%|\n| OOD increase |+3.5%|+11.0%|+3.0%|+9.3%|\n| OOD improve / ID increase|22.9%|73.3%|10.9%|282.7%|\n\n**As opposed to the performance improvement in scaling up SFT, which results in performance decrease in OOD tasks.**\n\nMore evident OOD performance growth happens when we scale up the revision iterations. In Figure 10, we observe that 55% of ID increases transfer to OOD when scaling up the number of iteration to 10 on GP-L task. \n\n> Q5. Regarding evidences in LIMO\n\nWe thank the reviewer for mentioning a great concurrent work. In our humble opinion, we believe our results do not contradict with the LIMO, due to the different focus of our studies. Our focus is on “comparing generalization between SFT on ground truth data versus running vanilla end-to-end RL”, whereas LIMO [1] demonstrates that SFT on “well-curated data can also achieve remarkable performance on mathematical reasoning tasks”, similar to the recently released s1 paper [2]. Slightly extended to your insightful question, we believe that “generalization by comparing long CoT + RL (deepseek / o1) vs well curated data for SFT” could be another important question to study in the future. \n\nOnce again we would like to thank the reviewer again for insightful and careful suggestions, if you feel like our response and additional results further improve the quality of our work, please feel free to improve your rating, thank you very much in advance! \n\n> References\n\n[1] Ye et al., 2025. LIMO: Less is More for Reasoning. arXiv preprint arXiv:2502.03387.\n\n[2] Muennighoff et al., 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.", "claims_and_evidence": "The paper studies an important problem of what are the different roles of RL and SFT in generalization of LLM post-training. \nThe claim is that RL generalizes and SFT memorizes and the provided evidences are experiments on GeneralPoint Game and V-IRL task. \nTheir results show that the learning of RL can be better generalized to their chosen OOD tests. \n\nI understand that the authors want to have a controlled experiment, but the experimental setting weakens the reliability of your conclusion. \n- As you mentioned, your RL is based on the SFT trained model. Does this mean that your base model for RL and SFT are different? (RL starts from SFT-trained LLaMA and SFT starts from LLaMa). If it is so, I think your experimental results cannot support a comparison between RL and SFT. \n- Your setting couples too many factors, making the experimental results hard to understand. For example, why do you consider a sequential revision instead of directly outputting the answer in the GeneralPoint Game. This introduces an additional factor of whether the model can correct its own results. A cleaner and simpler setting would help much. Why don't you experiment on more well-studied scenarios like math or other reasoning tasks. \n- Have you tested other sizes of models or other training steps of the training periods? It seems the SFT training gains the improvement on in-domain tests in the early stages of SFT training, is your observation related to overfitting of SFT training?", "methods_and_evaluation_criteria": "To some extent.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "In most of the experiments, I would say the performance of RL on OOD test sets does not drastically drop rather than \"generalization\". \nFor example, in GP, the OOD performance of RL is merely 12 to 17, and the in-domain acc is 50+. The improvements are marginal.", "relation_to_broader_scientific_literature": "N", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "- Previous research has shown that smaller model sizes tend to memorize, while larger models are capable of generalization [1]. Does the discovery in this paper apply specifically to a limited model size? For instance, would even smaller models, such as 0.5B or 2B parameters, exhibit memorization in both the SFT and RL periods? Additionally, do larger models, such as 72B or beyond, demonstrate generalization?\n\n- Recent studies have found that when the diversity and quality of SFT data is sufficient, models can also learn generalized knowledge [2]. Does this paper potentially overclaim its findings, given that the cost of exploration during the RL phase is significantly higher than in the SFT phase?\n\n[1] Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data\n\n[2] LIMO: Less is More for Reasoning", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEsn7M2YXK", "reviewer": "Reviewer_KPrc", "summary": "This paper conducts a comparative study of SFT and RL for post-training on GeneralPoints, an arithmetic reasoning card game, and also considers V-IRL, a real-world navigation environment. Experimental results show RL leads to model generalizing better in OOD cases while models trained with SFT hardly generalize.  The authors also discuss the role of SFT by concluding that SFT is still necessary to\nstabilize the model’s output format, enabling RL to achieve its performance gains.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## General Response\nDear reviewer KPrc,\n\nWe sincerely thank you for your valuable feedback. We especially appreciate your advice on making the claim more rigorous. To best of our effort in the rebuttal period, we conduct the following experiments to strengthen our evidence:\n- Experiments on Qwen-2.5-VL-3B. See results in Figure 22 of [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ).\n- Experiments on Llama-3.2 when fine-tuned by distillation from the RL models. See results in Figure 24 of [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ). \n\nWe provide the following feedback for your specific concerns.\n\n> Q1. Concerns on experiments with different model sizes.\n\nThank you for your suggestion. We conduct experiments on [Qwen-2.5-VL-3B](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), an up-to-date MLLM with smaller model size. We observe that **SFT memorizes, RL generalizes still holds in this case.** Specifically, RL achieves an increase of +3.45% on OOD while SFT causes a drop of 8.48%. Detailed performance curve can be found in Figure 22 of [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ). Experimental settings are the same as our original paper except we train fewer steps due to time constraint. \nWe are also interested in scaling up our method to larger models. But due to resource constraints, we are unable to conduct experiments on larger models (>=32B) during the rebuttal period.\n\n> Q2. Concerns on diverse trajectories\n\nGood point and we add a group of experiments on distillation of the RL model. As is illustrated in Figure 24 of [rebuttal material](https://drive.google.com/file/d/1WheCe-fkbX7jLKn2hsO7E701nGEPkuhJ), **we observe faster in-distribution performance increase with less out-of-distribution degradation compared to the original SFT experiments**. This evidence still aligns with our original finding but demonstrates the positive effect of diverse SFT data. We appreciate you for pointing this out and we believe that “generalization by comparing long CoT + RL (deepseek / o1) vs well curated data for SFT” could be another important question to study in the future. \n\nOnce again, we would like to thank the reviewer again for insightful and careful suggestions, if you feel like our response and additional results further improve the quality of our work, please feel free to improve your rating, thank you very much in advance!.", "claims_and_evidence": "1. SFT memorizes, RL generalizes. This is the main claim of this paper and is a very strong claim. I think it deserves more rigorous evidence under more controllable setups. The author concludes this by experimenting with GeneralPoints and V-IRL. They train models to learn the tasks with SFT and RL respectively and evaluate the performance in synthetic OOD settings. The synthetic nature ensures the validity of OOD scenarios. Nonetheless, the learning setups are questionable as the quality of SFT data is well controlled. It is unclear whether the gaps arise from RL learning more generalizable CoTs/trajectories. \n\n2. Scaling RL improves visual recognition accuracy in VLM training. The authors extend the OOD generalization benefits in visual tasks to study the underlying reasons and find that performance improvement correlates with visual recognition accuracy.\n\n2. SFT is still necessary to stabilize the model’s output format. The author supports this by showing RL without SFT initialization fails.", "methods_and_evaluation_criteria": "See discussions about the claims.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See discussions about the claims.", "relation_to_broader_scientific_literature": "This paper correlates to the study about the generalizability and training efficiency of data from different sources. For instance, (1) fine-tuning with on-policy synthetic data (RL) enhances learning efficiencies in math reasoning [1].\n\n[1] RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold", "essential_references_not_discussed": "Not found.", "other_strengths_and_weaknesses": "All experiments in this paper are conducted with fixed and ~7B-scaled models. It is unclear how larger pretrained models affect the generalization of SFT and RL.", "other_comments_or_suggestions": "It would be better to have better control of the SFT data and conduct a more thorough comparison of different SFT trajectories. \n\nFor example, the authors can SFT by distillation from the RL models to ensure the distribution of CoTs is consistent.", "questions_for_authors": "Have you experimented with larger models and are there any generalization benefits brought by better pretraining?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This study examines supervised fine-tuning (SFT) and reinforcement learning (RL), two prominent post-training techniques for foundation models. While reviewers raised valid concerns regarding the model training methodology, theoretical contributions, and novelty of the approach, they unanimously agreed on the importance of investigating how different training strategies affect foundation model performance. After careful consideration, I find that the study's strengths - particularly its timely research question and comprehensive evaluation - ultimately outweigh its limitations. I therefore recommend acceptance.
Decision: Accept (poster)