The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs
Authors: Jierun Chen
Tiezheng YU
Haoli Bai
Lewei Yao
Jiannan Wu
Kaican Li
Fei Mi
Chaofan Tao
Lei Zhu
Manyi Zhang
Xiao-Hui Li
Lu Hou
Lifeng Shang
Qun Liu
Venue: Accepted by TMLR ()

Abstract: Large vision-language models (VLMs) increasingly adopt post-training techniques such as long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL) to elicit sophisticated reasoning. While these methods exhibit synergy in language-only models, their joint effectiveness in VLMs remains uncertain. We present a systematic investigation into the distinct roles and interplay of long-CoT SFT and RL across multiple multimodal reasoning benchmarks. We find that SFT improves performance on difficult questions by in-depth, structured reasoning, but introduces verbosity and degrades performance on simpler ones. In contrast, RL promotes generalization and brevity, yielding consistent improvements across all difficulty levels, though the improvements on the hardest questions are less prominent compared to SFT. Surprisingly, combining them through two-staged, interleaved, or progressive training strategies, as well as data mixing and model merging, all fails to produce additive benefits, instead leading to trade-offs in accuracy, reasoning style, and response length. This "synergy dilemma" highlights the need for more seamless and adaptive approaches to unlock the full potential of combined post-training techniques for reasoning VLMs. Code, dataset, and fine-tuned models are available at https://github.com/JierunChen/SFT-RL-SynergyDilemma.

Published in Transactions on Machine Learning Research (01/2026)
The Synergy Dilemma of Long-CoT SFT and RL: Investi- gating Post-Training Techniques for Reasoning VLMs
Jierun Chen"1, Tiezheng Yu 1, Haoli Bai 1 Lewei Yao', Jiannan Wu', Kaican Li², Fei Mi', Chaofan Tao,, Lei Zhu', Manyi Zhang', Xiaohui Li', Lu Hou', Lifeng Shang', Qun Liu Huawei Technologies, HHKUST
Reviewed OpenReview: http:///ooonerreeeee////ooooootd
Abstract
Large vision-language models (VLMs) increasingly adopt post-training techniques such as long chain-of of- thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL) to elicit sophisticated reasoning. While these methods exhibit synergy in language-only models, their joint effectiveness in VLMs remains uncertain. We present systematic in- vestigation into the distinct roles and interplay of long-CoT SFT and RL across multiple multimodal reasoning benchmarks. We find that SFT improves performance on difficult questions by in-depth, structured reasoning, but introduces verbosity and dogrades per- formance on simpler ones. contrast, RL promotes generalization and brevity, yielding consistent improvements across all difficulty levels, though the improvements on the hard- est questions are less prominent compared to SFT. Surprisingly, combining them through two-staged, interleaved, progressive training strategies, well as data mixing and model merging, all fails to produce additive benefits, instead leading to trade-offs in accuracy, rea- soning style, and response length. This "synergy dilemma" highlights the need for more seamless and adaptive approaches to unlock the full potential of combined post-training techniques for reasoning VLMs. Code, dataset, and fine-tuned models are available al httpss//github..o
1 Introduction
Large language models (LLMs) like OpenATs o13 (laech all 2023 and DeepSeek-R1 (Guo ef all 2025)
have demonstrated remarkable reasoning abilities by thinking before answering. These models go beyond mere pattern matching, exhibiting sophisticated cognitive bchaviors like multi-step planning, reflection, error correction, well as summarization (GandhieL_a) all, 2025; Wang el al,, 20250). This reasoning capability is primarily enabled by two core post-training techniques: Supervised Fine- Sine-Tuning (SFT) on long chain-of- thought (CoT) data (Muennighoti et all, 21P4; Moshkov et all, 2124; Sun et all, 2122 and Reinforcement Learning (RL) with verifiable feedback (Lyv et all, 2025: Luo ef all 2025). In language-only domains, these
methods often show synergistic effects, yielding substantial improvements on complex reasoning benchmarks when applied sequentially iteratively (Li al.
20A Team et
Yeo et all 4125)
This paradigm has naturally motivated researchers apply similar paradigm to large vision-languee language models (VLMs) in pursuit of comparable gains in multimodal reasoning (Huang et al, 2025: Zhang et al], 2025: Wang all, 2025a). However, the results have been inconsistent and controversial. On one hand, some findings suggest that even small-scale SFT on long-CoT traces can elicit step-by-step multimodal reasoning and improve accuracy on multimodal math benchmarks eLl 2125). On the other hand, some studies report that SFT, even when followed by RI, can dograde performance (Chen of all 202ba). These discrepancies point to a complex and still poorly understood interplay between training strategies in the multimodal domain.
"Equa contribution; # Corresponding author: baihaoli www.oolihhuve...om

---

Published in Transactions on Machine Learning Research (01/2026)
SFT RL
Two-stage
Interleaved
Progressive
Data Mixing
Model Merging MMStar
MathVision
MathVerse
MathVista
MMMU
25
20
15
Accuracy Gain (%)
10
5
0
-5
-10
-15
2 4 2
2
5
3
1
Difficulty Level
Figure 1: Accuracy gains from various post-training techniques across five difficulty levels (L1, easy to L5, hard) on five multimodal reasoning benchmarks. Long-Cor SFT boosts Qwen2.5-VL-7B on harder questions but hurts casier oncs, while RL yields steady gains across the board. Hybrid strategics consistently trade off strengths rather than achieving true synergy.
In this work, we conduct a systematic study, covering diverse post-training paradigms, comprehensive eval- nations, and varied models, to answer two key questions:
1) What unique roles do long-Cot CoT SFT and RL play in shaping the reasoning abilities of VLMs?
2) Can we effectively combine them to realize the best of both worldsstructured reasoning and robust perfor- mance?
To answer the first question, we zoom in current multimodal reasoning benchmarks through the lens of question difficulty, a factor that has often been overlooked. Unlike textual reasoning benchmarks such as AIME25, MATH500 (Hendrycks et all, 2021),, and GPQA (Rein all, 2024),, which emphasize logically demanding and difficult tasks, current multimodal reasoning benchmarks like MathVista (Laree aall 2123)) MathVerse (Zhang all, 2I2a5), and MMMU (Yueettall, 20221) contain a large proportion of simple questions focused perception and fine-grained visual understanding rather than complex cognitive reasoning. After categorizing benchmark questions by difficulty, we find that long- CoT SFT improves performance primarily on hard questions but degrades it on easier ones by introducing unnecessary verbosity and overthinking. In contrast, RL offers steady gains across questions through concise responses and better generalization. That said, gains on the hardest questions are less significant than those achieved by SFT.
Building on these insights, we then explore several strategies for combining SFT and RL, including two- stage, interleaved, and progressive training, as well as data mixing and model merging. These strategies differ in timing, adaptability, and method of combination. Two-stage and interleaved training focus on when each method is used: the former separates SFT and RL into distinct phases (first SFT, then RL), while the latter interleaves them step by step. Progressive training adds adaptability, fading hints in SFT over time to smoothly transition toward pure RL. Meanwhile, data mixing blends data distilled from SET and RL. for new round of fine-tuning, while model merging directly combines finc-tuned models parameters via interpolation. Despite their promise, they all hit a synergy dilemma: efforts fuse long-CoT SFT and RL often produce trade-offs rather than true complementarity, shown in Fig.
2

---

Published in Transactions on Machine Learning Research (01/2026)
For instance, interleaved training balances performance but cannot surpass standalone RL, and data mixing preserves neither SFTs strength on the MathVision benchmark nor RLs broad gains across other benchmarks.
By surfacing these insights, we provide new clarity on various post-training techniques for reasoning VLMs. We demonstrate that synergy between SFT and RL is fragile. Achieving it requires not just method stacking, but nuanced control over adaptivity, compatibility, and diffculty-swareness..
2 Distinct Effects of Long-CoT SFT and RL
In this section, we investigate the distinct effects of long-CoT SFT and RL in cnhancing multimodal reason- ing for VLMs. We begin by analyzing how long-CoT SFT influences performance depending on the modality, scale, and reasoning quality of the training data, as discussed in Sec. 211 We then examine RL training dy- namics, highlighting the importance of KL regularization and the necessity of incorporating simple questions 2 systematical comparison between SFT and RL, revealing that SFT tends to benefit harder questions through verbose, structured reasoning, while RL yields steadier improvements with concise responses. Together, these findings offer a nuanced view of how each method contributes to reasoning capabilities in multimodal settings.
(see Sec. ) Finally, present in Sec.
2.1 Long-CoT CoT SFT
Supervised fine-tuning (SFT) with long chain-of-thought (CoT) data has proven effective for language mod- els (Gno al, 2026), particularly in the mathematical field (Muenmigoof et all 2025). However, its efficacy for multimodal reasoning remains debated. For example, fine-tuning Qwen2.5-ViL VL-7B with an R1-Onevision reasoning dataset yields marginal improvements on MathVision (Wang el. all kIPa)) and even declines on MathVerse (Zhang et all 2Izaa), regardless of increased model size or data scale (Chen et all, 20251). This raises the question of whether long-CoT SFT offers any tangible benefits for multimodal reasoning. To answer this, we access the effectiveness by varing the data sources and modalities.
Data. VLM reasoning is typically regarded as textual reasoning conditioned on visual inputs. Therefore, we hypothesize that SFT with purely textual long-CoT data can enable multimodal reasoning. We use the sl dataset with 1k diverse and challenging questions (Muennighot et all 2025). It has two versions, sl and s1.1-R1, distilled from Gemini2.0-flash (Google, 20221) and DeepSeek-R1 (Gino et 2025), respectively. To study the effect of data modality, construct multimodal reasoning dataset by distilling from s1.1-R1 SFT-ed model on MM-Eureka queries (Meng et all, 2025). We generate eight responses per query, retain the shortest correct one, and obtain our Mureka-Distill with 34k samples. For each query, we append a reasoning instruction: Please reason step by step within <think </think> tags, and put your final answer within \boxed(}.
Training settings. We employ LLaMa-Factory (Zheng et all, 2022!) as the training framework, and Qwen2.5-VLL-nsteee
(Bai of all, (12R) the main bascline model for its strong performance and broad adoption in recent work (Wang et all, 202aat Yang et all 2025at Huang et all, 21255.. The VIT visual en- coder and MLP connector remain frozen during training, as this way compares favorably to unfreezing them. We use a learning rate of 1x10-5 and a batch size of 32. We train on sl and s1.1 for 15 epochs and on Eureka-Distill for 5 epochs. Checkpoints are saved after each epoch.
Evaluation settings. We evaluate models using VLMEvalKit (Duan ef all, 2023) on both mathematical and multi-disciplinary reasoning benchmarks, including MathVision test (Wang et all, 21za1), MathVerse test mini (Zhang et all (024a), MathVista test mini (Lu all, 4i23), MMMU val (Xue et all, 20241,, , and MMStar val (Chen et all, 2024)). We append the aforementioned reasoning instruction to the benchmark questions when evaluating finc-tuned models. For answer extraction and judgement, we use rule matching and Qwen2..5VVL-322 as the judge. For inference, we enable sampling using a temperature of 0.6, a lop-p of 0.95, a top-k of 20, and a maximum generation length of 24k. Results are averaged over 4 runs to mitigate statistical variance. To ensure fair comparison under one framework, we reproduce Qwen2.5-VI. VL-7B results. For fine-tuned models, we report their best checkpoint results. We maintain consistent evaluation settings throughout the paper.
3

---

Published in Transactions on Machine Learning Research (01/2026)
Results and analysis are summarized follows:
Using just 1k textual data, s1.1-R1 improves multimodal reasoning across four benchmarks: MathVision, MathVerse, MathVista, and MMMU val, with only a slight trade-off on MMStar compared the baseline. This is particularly notable given that s1.1-R1 consists purely of textual long-CoT traces, yet outperforms Eureka-Distill, a much larger multimodal dataset with 34k samples (see Tab. M). These results suggest that high-quality reasoning traces, even without visual input, can effectively transfer to multimodal tasks due to the underlying alignment between language and vision in VLMs.
Long-CoT SFT improves more prominently on difficult benchmarks that inherently require deeper reasoning, such MathVision and MathVerse. As illustrated in Fig. performance gains correlate positively with response length, indicating that longer reasoning chains more beneficial for complex queries. This pattern highlights the utility of long-CoT supervision in teaching models to break down and solve harder problems step-by-step.
Mong-CoT does not guarantee performance gains. The elfectiveness of SFT depends heavily on the quality of the reasoning traces. For example, when fine-tuning with s1- L-Gemini2, despite using the same query set as s1.1-R1 and generating responses that are 815 times longer than the baseline (see Fig, Z), model performance deteriorates across nearly all benchmarks (see Tab. M.. These findings underscore that verbosity alone insufficient; the quality of the reasoning steps crucial for realizing effective test-time scaling.
Table 1: Comparison of SFT with different textual and visual long CoT data. M. is short for Math.
Model
Modality Data
Vision
M. Verse
Visita
MMMUval
MMStar
Avg.
Qwen2.5-Vi 5- VL-7B
26.1
42.3
66.4
53.5
63.2
50.3
SFT w/ sl-Gemini2
Text
1k
25.2
41.8
66.8
50.0
59.5
48.7
SFT w/ sl.1-R1
Text
1k
30.6
48.1
67.6
54.4
61.8
52.5
SFT w/ Mureka-Distti
Visual
34k
29.7
47.2
65.6
53.6
60.8
51.4
Baseline
SFT w/ 2l-Gemini2
s1.1-R1
Baseline
w/ shifted
13.kk
Avg Length (Tokens)
104
8.0k
7.7k
4.7k
2.5k
2.7%
2.7k
Accuracy (%)
60-
1.54
10*
Math
0.4k
0.4k
MathVission
107
0.1k
Math Vision
MathVerse
MathVista
MMMU val
MMStar
Avg Length (Tokens)
Figure 2: Response length comparison.
Figure 3: Acc. by response length.
2.2 RL
Unlike the above SFT relying on external reasoning traces, reinforcement learning (RL) focuses enabling models to self-explore and self-improve by interacting with the environment and receiving feedback signals. Notably, the Group Relative Policy Optimization (GRPO) algorithm (Shao ct all, 2024) simplifics the train- ing pipeline by eliminating the need for a value model, making the process more efficient and scalable. Mathematically, let Q be the query set and {or, 02. ..o) be the sampled outputs from the old policy model Told- GRPO optimizes the following objective:
logl min (,, ())p() -.11-,,++4 (al--,,++,AA., -DDL (Te Tret)
1 G
(G) Ey (v, aa
Here, A, Tu me-maa (a is the estimated advantage using a group of rewards (ri, r,, "h, ra), is the KullbackLeibler (KL) coefficient contorling the deviation of current model ny from the reference model Teff r0) is for importance sampling, and is the clipping hyper-paranen parameer.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Building this algorithm, some studies et.ll, 2025 Chen all, 2022d) have proposed removing the KL term for unconstrained exploration. Additionally, it has become common practice to pre-filter dynamically filter easy samples where the model consistently predicts all answers correctly (Peng et all 21255 Yt st all, 2025 Wang ot all, 20255). We examine these two variations as they can play a crucial role in model training and performance.
Training Settings. Using the Verl framework (Sheng et al, MIZa!), we train models on multimodal queries from Eureka-Distill Distill described in Sec. 2TI, with the following settings: a learning rate of 10-6, batch size of 128, mini-bat batch size of 64, temperature of 1, rollout number of 8, maximum completion length of 4k tokens, and training epochs. For reward functions, we combine two types of rewards: accuracy and format. The accuracy reward is set to 0.9 for correct answers and 0 otherwise, while the format reward is set to 0.1 for correctly formatted responses and 0 otherwise. We use the same reasoning instruction as previonsly used for SFT fine-tuning: Please reason step by step within <think> </think> tags, and put your final answer within \boxed(}.
Results and analysis are summarized follows:
The KL term stabilizes the training process. Without it, the training reward collapses after step 300, and the model exhibits lower entropy and dramatic fluctuations in response length (see Fig. EI). Tab. @ also shows that KL regularization improves accuracy by a clear margin across all benchmarks.
Simple questions matter for maintaining baseline performance. Without them, the model's ability to handle simple questions could deteriorate after RL fine-tuning. To validate this, remove the casiest questions on which the bascline model predicts correctly across all 8 rollouts. Results in Tab. B show that RL with these easiest questions leads to higher accuracies. This is because, although the advantages for these easiest questions become zero after GRPO normalization, they still influence the training process via the KL loss, ensuring consistently high accuracy on these questions (see Fig. El..
KL-0
KL-0.005
KL-0 KL-0.005
KL-0
KL-0.005
0.4
Entropy Loss
Reward
0.8
0.3
750
0.2
0.6
0.1
250
0.0
200
200
Training Step
Training Step
Training Step
(a) Training Reward
(b) Training Entropy
(c) Response Length
Figure Training dynamics comparisons. Without KL regularization, RL training suffers from reward collapse, lower entropy, and more dramatic fluctuations in response length.
Table 2: Abalation of the KL term for RL training. M. short for Math.
w/o
90
Model
V.sisoon M.Verse
M.Vista MMMUval MMStar
Avg.
70
Qwen2.55VVL-77 RL, KL=0
26.1
42.3
66.4
53.5
63.2
50.3
Accuracy
50
27.7
47.0
71.5
54.8
63.5
52.9
RL, KL=0.005
29.0
52.1
72.6
55.1
66.5
55.1
1
3 Diffiulty Level
2
Table 3: Ablation of retaining the easiest questions during RL.
Model
Vision
M. Verse
M. Vista MMMUval MMStar
Avg.
Figure 5: Accuracy gap differs across questions of varying difficulty levels on MathVision.
Qwen2.5 VL-7B
26.1
42.3
66.4
53.5
63.2
50.3
RL w/o casiest
26.6
47.1
71.4
54.0
64.8
52.8
RL
29.0
52.1
72.6
55.1
66.5
55.1
5

---

Published in Transactions on Machine Learning Research (01/2026)
2.3 Comparisons of Long- CoT SFT and RL
With optimized data sources and training configurations, both long-CoT SET and RL yield partial or full improvements in model accuracy. However, the extent of these improvements varies across different benchmarks, as evidenced in Tab. gI This disparity motivates a granular comparison between SFT and RL fine-tuned models. For fair comparisons, we use the same Eureka-Distilles dataset.
SFT excels at the most difficult questions, while RL provides steadier, broader improvement. To analyze performance by difficulty, we categorize all benchmark questions into five levels (easy to hard), based on the baseline models pass rate P across 16 independant runs: level (P 2 12), level 2 (16 P << 1 level 3 16 I P < 16), level 4 ( 1 < P < 16), and level 5 (P < 16). We also provide a more gramular breakdown in Appendix Sec A. using 17 pass rate bins (from 0 to T6 From Fig. D. we see that the accuracy gain from SFT starts negative at level 1, climbs to positive gains at higher levels, and ultimately surpasses RL at the most difficult levels, 4 and/or 5. In contrast, RL provides more consistent improvements across all difficulty levels and benchmarks.
2
T6
SFT injects rich yet verbose reasoning tokens, while RL preserves concise responses but under- utilizes structured resoning. To better understand how these fine-tuning methods shape model responses, we measure token-level KL divergence before and after fine-tuning. As illustrated in Fig. E. the SFT-ed model consistenlly exhibits higher KL divergence al sentence beginnings. Those darker lokens highlight what we call reasoning pivotal tokens. These include structural words like first, second, and next; logical connec- tors such as then, alternative, and because; action words like let, consider, and verify; and meta-thinking phrases like wait, hmm, and maybe. After completing the thinking process, the SFT-ed model first provides summary before giving the final answer. However, the average response length is more than 10x longer than that of the baseline and RL-tuned models (see Fig. B), incurring the overthinking risk. In contrast, RL-tuned models remain close to the baseline, with little change in loken distribution (see Fig. E) and a continued preference for brief, to-the-point answers (see Fig. H) with only occasional use of reasoning words (sce Tab. 6)
Table 4: Comparison of SFT and RL with Eureka-Distill as the training set.
Model
MathVision
MathVerse
MathVista
MMMUval
MMStar
Avg.
Qwen2.5-VL-7B
26.1
42.3
66.4
53.5
63.2
50.3
+ SFT
29.7
47.2
65.6
53.6
60.8
51.4
+ RL
29.0
52.1
72.6
55.1
66.5
55.1
Question
Response by model after long-CoT SFT
17
20
16 What is the final result? Please reason step by step within <think> </think> tags, and put your final answer within Wboxed ():
④
<th inke Okay let's see. The image shows two separate operations here, each with two numbers and an operator, Then their results are added together, Let me break it down stop by stop. stop.First. on the left side; 17 plus 3. That's that's a
simple addition. 17 + 3 is 20, So the result here is Then, on the right side: minus 16. Another simple subtraction.
equals So the result here is 4. Now, these two results are then added together. The question mark is asking for the final sum So, take the lirst result, 20, and add the second result, 4, to it. 2.0
Respouse by model after RL
Wait, lot me verify that again. The two operations gave 2.0 and 4, Adding them gives 24, Yes, 20 24 But let me check If labeled the operations correctly. The left one Is addition because the
< <hinnk>To To solve the problem, we need to follow
the operations step by step 11 ** First Operation:
is indeed
=e - We start with the numbers 17 and 3.
circle has plus sign inside, right? 2 And the right one is subtraction because the little circle has a minus sign Inside. So, 17
The operation is addition (+). Calculation: I( 1
7 7 20 1).2. Second Operation: ** We
20, and 20 16 4. Then 20 + 24. Hmm, seems correct.
start with the numbers 20 and 16. The
Altemattvely. maybe should have drawn the operations more literally. Like, the first operation takes 17 and 3, combines them with addition to make 20.
operation is subtrection (-). Calculation: If 20 .16 16 = 4 1).3. Third Operation: 44 We take
the result from the first operation (20) and the
16, combines them with subtraction make 4. Then those two results (20 and 4) are added to give the final answer of Yes. that makes sense. So the final answer should be 24, Final
second operation takes
result from the second operation (4). The
operation addition (+). Calculation: 20-
Answer **) mooxed (24) ) First, solve the addition
the left: 1(17
4 = 24 1). Therofore, the final result is 24. </ think > >|boxed (24) 2244)..mmm... <jjm_end]- end]>
201). Next, solve the subtraction on the right: ((20 - 16 = 41). Finally. add the results the operations: 1(20 241).1( boxxed (24]1] <jim endi>
Figure 6: Illustration of token-level KL divergence, where darker tokens indicate larger divergence.
6

---

Published in Transactions on Machine Learning Research (01/2026)
SET Gain
Gain
MathVision
MathVerse
MathVista
MMMU
MMStar
Accuracy Gain (%)
-10
-20
2
3
5
1
2
3
4
5
1
2
3
4
5
Difficulty Level
Figure 7: Accuracy gain SFT and RT. across 5 difficulty levels on 5 benchmarks.
Baseline
SFT
RL
Table 5: Frequency of reasoning words evaluated on MathVision.
Arg Longth (Tokons)
10a
6:7%
3.4%
2.7k
Word
Baseline
SFT
R.L
0.8k
0.5k
0.4k
0.33
0.4k
0.4k
"wait"
0
249091
D.2K
0.2k
"check"
790
18501
671
104
0.1k
"mistake"
151
4345
"alternative"
96815
Mithoision
MathVerse
MathVista
MMMU val
MMStar
"however"
917
36625
764
Figure 8: Response length comparison.
3 The Synergy Dilemma of Long-Cot CoT SFT and RL
Long-Coll CoT SFT and RL each bring unique strengths and weaknesses in response styles, efficacy, and efficiency in solving problems of varying difficulty. The key question is: how can we integrate their strengths to create syneryy? As shown in Fig. B, we approach this by exploring several dimensions, including training alternation, data mixing, and model merging. For training alternation, we examine strategics such as the popular two-stage SFT and RL, interleaved SFT and RL, and progressive SFT and RL. Through systematic exploration, uncover fundamental synergy dilemma for reasoning VLMsLong- CoT SFT and RL often behave more like trade-off than perfect complement. In the following sections, we dive deeper into each method and evaluate their ineffectiveness in bridging this gap.
3.1 Training Alternation
Two-stage SFT & R.L. Starting from the best checkpoint obtained after SFT on the Sureka-Distill dataset for 4 epochs, we apply RL fine-tuning using the GRPO algorithm. While the experimental setup matched the optimal configuration in Sec. 20, extend the maximum generation length N from 4k to 16k to accommodate the models longer outputs. This adjustment is necessary because, with N=4k, the clip ratio of model outputs surges to 20% early in training, causing instability and crashes as visualized in Fig. II Results in Tab. B show that the two-stage SFT and RL approach fails to improve performance over SFT alone, with the average accuracy across five benchmarks stagnating 51.4%. The model appears to have overfit to the paradigm established during SFT fino-tuning. We have also attempted reducing the number of preliminary SFT epochs to 1, but this does not alleviate the issue, highlighting the difficulty of overcoming SFT-induced overfitting or catastrophic forgetting thourgh subsequent RL fine-tuning.
Interleaved SFT & RL. To mitigate the risk of overfitting and uneven task performance after SFT, we employ an interleaved SET and RL strategy designed to balance initation and exploration. Importantly, we not apply interleaved training to all samples. Applying SFT loss indiscriminately would cause it dominate the optimization process, thereby slowing even reducing training rewards and validation accuracy, and resulting in substantial increase response lengths (see Fig. 12). Instead, we apply the SFT loss exclusively to questions with zero pass rates, while the RT. loss is applied to the rest.. This approach aligns

---

Published in Transactions on Machine Learning Research (01/2026)
(a) Training Alternation Two-stage Training
Interleaved Training
Progressive Training
SFT the off- policy response
Response
on the on- policy rollouts
step
n-1
2
1
(b) Data Mixing
(c) Model Merging
SFT Model
Distiled data
SFT Model
Distilled Model
Model Merging
Merged Model
Distiled Data
RL Model
RL Model
Figure 9: Attempts to integrate SFT and RL, including training alternation, data mixing, and model merging.
Table 6: Acuracy comparison of various attempts for SFT-RL synergy. Values are reported as mean accuracy + 95% confidence intervals.
Model
MathVision
MathVerse
MathVista
MMMUval
MMStar
Avg.
Owen22.5VLL-7
26.1 + 0.40
42.3 + 0.18
66.4 0.91
53.5 + 0.51
63.2 + 0.40
50.3 + 0.28
+ SFT
29.7
0.72
47.2 1 0.19
65.6 1.42
53.6 1 0.76
60.8 1. 0.94
51.4 0.46
+ RL
29.0 i: 0.55
52.1 i: 0.31
72.6 0.44
55.1 + 0.58
66.5 1 0.72
55.1 1: 0.19
+ Two-stage SFT & RL
29.3
47.1
66.6
53.0 +: 0.61
60.9
51.4
+ Interleaved SFT RL
29.2 + 0.47
48.7 +: 0.54
71.8 0.36
54.1 + 0.72
64.3 + 0.48
53.6 +: 0.10
+ Progressive SFT & RL
29.8
51.0
72.4
55.5 + 0.82
65.9
54.9
+ Data Mixing
29.2 + 0.64
51.2 + 0.26
72.0 0.54
55.1 + 0.97
62.7 + 0.35
54.0 + 0.40
+ Model Merging
29.6
50.4
71.8
53.7
+ 0.72
66.2
54.3
Baseline
SFT
RL
Two-stage
Interleaved
Progressive
Data Mixing
Model Merging
249k
266k
104
19k
22k
5k 4k
21k
37k
31k
1k
6k
9k
Count:
790
876
864
917
764
219
102
5 5
10
100
"wait"
"check"
"mistake"
"alternative"
"however"
Figure 10: Frequency of reasoning words across fine fin-tuning methods, evaluated on the MathVision.
with our objoctive of leveraging the complementary strengths of SFT and RL. As shown in Tab. E. the models accuracy consistently falls between that of SFT and RL across all benchmarks. Furthermore, Fig. [لا reveals similar trend in the frequency of reasoning-related words. These results indicate that interleaved training achieves a balance between SFT and RL, rather than a complete synergy of their respective ad advantages.
Progressive SFT & RL. In previous methods involving SFT, the reasoning traces during training are entirely sourced from external models. However, during inference, the model generates each token based on self-generated prefix tokens, resulting a mismatch between training and inference. This inconsistency can be more pronounced in reasoning models that produce long responses. To alleviate this, we explore an alternative approach called progressive SFT and RI. Early in training, for difficult questions with a pass
8

---

Published in Transactions on Machine Learning Research (01/2026)
rate of 0, the model relies on full external traces for SFT. As training goes on, the use of external traces is gradually reduced, starting with only prefix and eventually removing them entirely. For loss computation, prefix tokens are assigned an SFT loss with a weight (empirically set to 0.2), while self-generated tokens are assigned an RL loss. As illustrated in Fig. DR, the training reward and validation accuracy curves of progressive training closely align with those of pure RL, while producing longer responses and exhibiting greater step-to-step fluctuations. As expected in Fig. M. progressive training generates a higher frequency of reasoning-relater words compared to pure RL. Benchmark results in Tab. indicate that progressive training outperforms two-stage and interleaved training, with the average accuracy improving from 51.4 and 53.6 to 54.9, approaching RLs 55.1. Notably, progressive training achieves an accuracy of 29.8 on MathVision surpassing RLs 29.0 and almost the same as SFTs 29.7. However, this improvement comes at a cost: progressive training sacrifices performance on MathVerse and MMStar, ultimately falling short of achieving the desired synergy.
N-4k
N-8k
N-16k
N-4k
N-8k
N 16k
Nw4k
N=8k
Nw16k
Response Lende
5000
0.8
Reward
4000
0.3
0.4
3000
0.2
2000
0.1
0.0
1000
0.0
0
50
100
150
200
0
50
100
150
200
50
100
150
200
Training Step
Training Step
Training Step
(a) Training Reward
(b) Response Length
(c) Clip Ratio
Figure 11: Training dynamics of RL following SFT. The model demonstrates a higher clip ratio when the maximum new token length is set to N=4k and 8k compared to N=16k, and collapses after approximately 100 steps.
8L
ISR w/ SET for all
1SB wt SET for hard
1SR w/ SPT all
ISR hard
for for
2000
0.9
Von
0.9
Resporse Len
1500
0.8
1000
Val
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
Training Step
Training Step
Training Step
(a) Training Reward
(b) Response Length
(c) Validation Accuracy
Figure 12: Training dynamics of Interleaved SFT and RI. (ISR) applied to all samples or with SFT loss only for hard questions. Pure RL is compared as the baseline.
R1 Progressive SPT & RL
RL
Progressive SFT & RL
RL
Progressive
Response Length
1500
0.9
Reward
1000
0.7
0.7
0.6
100
200
300
400
500
100
200
300
400
500
100
300
500
Training Step
Training Step
Training Step
(a) Training Reward
(b) Response Length
(c) Validation Accuracy
Figure 13: Training dynamics of Progressive SFT and RL, compared to pure RL training.
3.2 Data Mixing
Data mixing is a simpler alternative to the above training alternation. It blends data distilled from SFT and RL models, followed by an additional round of SFT. In this method, we generate 8 responses per question using the RL model on the 34k Eurcka- Distill dataset. We collect only the correct ones, getting 230k samples. For questions with pass rate of 0, we add long-CoT responses, creating final dataset of 243k samples. The

---

Published in Transactions on Machine Learning Research (01/2026)
model is then trained this dataset for epochs, with all other settings consistent with those described in Sec. 2.1 As shown in Fig. M. data mixing encourages more frequent use of reasoning tokens. However, it results in a 3.8% accuracy drop on MMStar and unxxpectdlly produces responses that are 10 times longer, as shown in Tab. @ While data mixing demonstrates potential fostering adaptive reasoning, its overall accuracy remains lower than that of pure RL.
Table 7: Comparison of model response length and accuracy between RL and data mixing approach.
Model
MathVision
MathVerse
MathVista
MMMUval
MMStar
Avg.
RL
514
338
228
416
337
Length
Data Mixing
637
381
324
424
1999
753
RL
29.0
52.1
72.6
55.1
66.5
55.1
Accuracy
Data Mixing
29.2
51.2
72.0
55.1
62.7
54.0
3.3
Model Merging
Model merging is a training-free approach that combines the weights of two or more models to harness their individual strengths. This technique typically involves interpolating the parameters of pro-trained models. We conduct experiments using MergeKit (Goddard et all, 2024), a versatile open-source toolkit for model merging. We adopt three model merging methods:
Linear Merging was introduced in the "model soups" approach by Wortsman el all 21122 which calculates a straightforward weighted average of the models' parameters.
TIES Merging sparsifies the parameter changes (task vectors) and resolves sign disagroements before averaging (Yaday al 2023).
SLERP Merging performs Spherical Linear Interpolation the weight space between two models, enabling smooth transitions and preserving geometric consistency the parameter space (Shoemakd, 1985
For SFT and RL models, we adjust the merging ratio of the SFT model, testing values of 0, 0.25, 0.5, 0.75, and ratio of corresponds to the pure RT. model, while ratio represents the pure SFT model. Results are visualized in Fig. Iel Using the TIES method, SFT and RL models exhibit compatibility issues. In contrast, the Linear and SLERP methods show performance interpolation between SFT and RI, with roughly monotonic increase MathVision and decrease the other four benchmarks. The checkpoint with the highost average accuracy is achieved using the Lincar method with an SFT ratio of 0.25. As shown in Tab. E, this configuration preserves the high accuracy of the SFT model on MathVision and the high accuracy of the RL model on MMStar, while performing less competitively on the remaining datasets. These findings underscore the persistent challenges in achieving effective synergy between SFT and RL models.
MathVision
Varhe
MathVista
MMMU
MMStar
Linear
TIES
SLERP
10.0
20
10.0
Accuracy Gain(%)
7.5
7.5
5.0
5.0
-20
2.5
2.5
-40
0.0
0.0
-2.5
-2.5
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Merging Ratio SFT model
Figure 14: Accuracy gains of model merging methods (Linear, TIES, SLERP) by merging ratio.
10

---

Published in Transactions on Machine Learning Research (01/2026)
3.4 Generalizing to Models of Other Sizes and Families
To validate the generalizability of our prior conclusions to other model sizes and families, employed Qwen2.5-V V--3B and Gemma3-4B, trained via diverse post-training methods, and evaluated them on Math- Vision and MMStar (Tabs. B, B). Consistent with carlier findings, RL achieved the best overall performance, while SFT yielded limited, unstable gains. Among SFT-RL fusion strategies, Progressive training and model merging performed nearly as well as pure RT. Notably, interleaved training was unsatisfactory, revealing conflicts between fast and slow thinking training and associated stability risks. Further analysis of problem difficulty Fig. I+ shows the two models align with Qwen2.5- to hard questions, outperforming RL on the hardest tasks but degrading by around 30% on simple ones, whereas RL delivers more balanced, stable improvements across all difficulty levels.
SFTs accuracy gains rise from easy
Table 8: Acuracy comparisons with Qwen2.5VI-3B.
Table 9: Acuracy comparisons with Gemma3-4B.
Model
MathVision
MMStar
Avg.
Model
MathVision
MMStar
Avg.
Qwen2.5-VI-3B
21.3
0.68
53.4
0.70
37.4
0.64
Gemma3-1B SFT
22.9 0.40
17.6 1 0.27
35.3 1: 0.33
+ SFT
21.8 0.68
55.3
0.98
38.5 i 0.71
20.7 1 0.45
47.9 1 1.81
34.3 1 0.79
+ RL
26.5 0.67
59.4
0.54
42.9
0.56
RL
27.4 0.55
50.8 + 0.90
39.1 + 0.61
+ Two-stage
21.6
55.1
38.3
Two-stage
22.3
51.0
0.33
36.6 :: 0.31
+ Interleaved
19.4 0.59
49.7
1.49
34.5 = 0.96
Interleaved
21.2 1 0.27
45.8 + 1.47
33.5 + 0.67
+ Progressive
24.0 = 0.35
58.5
41.2
Progressive
26.4 0.61
51.7 + 0.45
39.0 + 0.23
+ Data Mixing
25.2 0.40
57.9
0.20
41.6 0.21
Data Mixing
22.2 + 0.47
49.8 + 0.66
36.0 + 0.46
+ Model Merging
24.3 = 0.33
58.1 1.25
41.2 0.74
Model Merging
25.6 0.52
+ 1.32
38.9 + 0.71
SFF
RL MathVision (Qwen2.5-VL-33))
Two-stage
Interleaved MMStar (Qwen2.5.VL-3BB
X Progressive
Data Mixing MathVision
Model Merging MMStar
(Gemma3-4B)
(Gemma3-4BB)
30
20
Accuracy Gain (%)
10
-10
-20
-30
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3 4
5
Difficulty Level
Figure 15: Accuracy gains for Qwen2.5-Vi VL-3b and Gemma3-4b across five difficulty levels.
4
Related Work
The remarkable success of language reasoning models, such as OpenAI (Jacch all, 2023)) and DecpSook- RI (Guo el. all 2025), has sparked significant interest in improving the reasoning capabilities of vision- language models (VLMs). These advancements predominantly driven by novel post-training techniques, with supervised fine-tuning (SFT) and reinforcement learning (RL) as the two common and key approaches.
Early efforts focused on leveraging SET to euhance long-form chain-of-thought (CoT) reasoning in VLMs. For example, LLaVA-CoT (Xn all anze)) introduced a structured reasoning dataset that guides models through scquential stages of summarization, visual interpretation, logical reasoning, and conclusion gen- eration. Mulberry (Yao et all, 2022)) utilized the collective knowledge multiple models to collabora- tively hypothesize, search, and identify effective reasoning trajectories leading to correct answers. Addi-
11

---

Published in Transactions on Machine Learning Research (01/2026)
tionally, large-scale rewriting of original responses into CoT-style rationales using off-the-shelf models has been shown to significantly improve VLM reasoning performance (Gnio et all 2022).. On the other hand, RL-based approaches have also demonstrated remarkable success in enhancing reasoning capabilities. MM- EUREKA (Meng et all, 2026) introduced the MMK12 dataset and a two-stage training strategy to stabilize (Zhou el all, 2023) replicated R1-Zeros "Aha Moment" R for multimodal
RL training. VisuaaThinker-R1-Zea reasoning on non-SFT VLMs. VIAAAThhnking (Chen ef all 202bal challenged the effectiveness of SFT for cross-modal reasoning transfer and showed that training GRPO with mixed-reward objective yields superior performance. Vl-rethinker (Wang et all, 20255l introduced Selective Sample Replay (SSR) and Forced Rethinking techniques to address the vanishing advantages problem and encourage slow-thinking during reasoning.
Many works have explored combining SFT and RL, particularly within a two-stage training paradigm. These approaches differ primarily in their fine-tuning dataset design and RI. strategics. --Ressseee (Zhang
all 20246) distilled rationales from GPT-4o and applicd Direct Preference Optimization (Ratailov cf all,
002% to improve CoT reasoning and generalization. Vision-R1 (Huang el. all
2026 performed SFT using a
synthetic multimodal CoT dataset generated via modality bridging and trained GRPO with their Progressive Thinking Suppression Training (PTST) strategy. R1-VL (Zhang et all, 2025) proposed StepGRPO, an RL framework that rewards step-wise wise accuracy and logical validity rather than merely imitating correct reasoning paths. Reason-rft ef.al, 2 incorporated three distinct types of accuracy rewards during RL. training. Additionally, several studies have emphasized the importance of SFT data construction, systematically evaluating dataset generation methods to support effective RL from a cold start (Yang et all, 202M1 Wei
aLl 2026: elall 2025:: Wen 2022; Shen el all, 2025). In this work, we present systematic analysis of the performance and behavior of VLMs trained exclusively with cither SFT or RL. Furthermore, we explore various approaches to combine the strengt hi of SFT and RL from multiple perspectives, including training strategies, data mixing, and model merging,
5 Summary and Implications for Future Study
This study provides a systematic investigation into the roles of long-CoT SFT and RL in enhancing the reasoning capabilities of VLMs. Long-CoT SFT demonstrates strong performance on complex problems by introducing structured, step-by-step reasoning but suffers from verbosity and reduced accuracy on simpler tasks. RL on the other hand, promotes concise responses and robust generalization, delivering consistent gains across varying difficulty levels, though its gains on the hardest questions are lower than those from SFT. Attempls to combine SFT and RI, through two-stage, interleaved, and progressive training, as well as data mixing and model merging, all reveal a persistent "synergy dilemmn"" where trade-offs in accuracy, reasoning style, and response length dominate, preventing truc complementarity.
We summarize below the key factors explaining why SFT and RL failed to synergize in our altempts, supported by experimental evidence. First, unlike sufficiently large models (e.g., >a000--arraeteer models such as DeepSeek-R1 and Kimi-k1.5) that use self-distilled long-CoT data for high compatibility and low overfitting, the relatively smaller models employed in this paper rely on distillation from external models. This reduces data-model compatibility, causes SFT-induced overfitting that RL cannot mitigate, and results in no improvement when combining SFT and RL compared to standalone SFT. Second, integrating SFTs structured "slow thinking" and RL's concise "fast thinking" requires adaptive switching between distinct reasoning styles, which is more complex than focusing on one mode. This results in trade-offs: SFT improves performance on hard questions but degrades on casy ones, RL performs stcadily across all difficulty levels but lags behind on the hardest questions, and brid strategies only balance these strengths without surpassing RL.
To address these challenges, future rescarch should prioritize: 1) constructing model-compatible or self- distilled long-CoT datasets to mitigate data-model incompatibility or catastrophic forgetting after fine- tuning, using techniques such as prompt engineering and in-context learning; and 2) developing adaptive frameworks capable of accurately identifying problem difficulty, selecting optimal rcasoning modes, and avoiding interference between different reasoning patterns. By overcoming these obstacles, VLMs can evolve into truly versatile models capable of reasoning efficiently and effectively across diverse multimodal tasks.
12

---

Published in Transactions on Machine Learning Research (01/2026)
References
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkum Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xic. Sft or rl? an early investigation ion into training rl-like reasoning large vision-language models. arXiv preprinl 0rXiv:2504.11400 2025a.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xic. Sft or rI? an early investigation into training rl-like reasoning large vision-language models, 2025b. URL
https://arxiv, www.oorgg/////eooooooooooooeee/o 2504. 11468.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprinl ar Xiv:2403.20330, 2024.
Mingrui Chen, Haogeng Liu, Hao Liang, Huaibo Huang. Wentao Zhang, and Ran He. Unlocking the potential of difficulty prior in rl-based multimodal reasoning, arXiv preprint arXiv:2505.1321 13261, 2025c.
Shuang Chen, Yue Guo, Zhaochen Su, Yafu Li, Yulun Wu, Jiacheng Chen, Jiayu Chen, Weijic Wang, Xiaoye Qu, and Yu Cheng. Advancing multimodal reasoning: From optimized cold start to staged reinforcement learning. arXiv preprint arXiv:2506.00 (al:2566.02007, 2025d.
Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji- RoRg Wen. Virgo: A preliminary exploration on reproducing ol-like mllm. arXiv preprint ar Xrv:22000000 01904, 2025.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. VInevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Mullimedia, pp. 11201, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 2025.
Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcces mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 477-485, 2024.
Google. Gemini 2.0 flash thinking mode (gemmn-20-faaahhhhhkknnggg-ggppp December 2024. URL https: c/cloud. attpe//coneeeeeeteeeee/eeeetttte
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXio preprint arXiw:2501.1004 2025.
Jarvis Guo, Tuncy Zheng, Yuclin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Noubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint Xrxim:24 12. 22.02227. 05237, 2024.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arX2::2103.03874, 2021.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. or Xio preprinl 0rXiv:2503.067/3 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Ei-Kishky, Aiden Low, Alec Helyar, Alek- sander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 0rXiv:2412.16799, 2024.
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron 1.1: Advancing math and code reasoning through sft and rl synergy. arXiv preprint arXiv:2506. 13284, 2025.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-Av, bard, and other large multimodal models. CoRR, 2023.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongla, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepscaler:
Surpass-
ing ol-preview with 1.5b model by scaling rl. http::/ppeetyyyaaloobo.ooo..... DeepScaLeR Prrpessnng-------oe-------- -o RL- wtg.aaa............ooooooooooooa 2025. Notion Blog.
Chengqi Lyu, Songyang Gao, Yuzhe Gu, Wenwei Zhang, Jianfei Gao, Kuikun Liu, Ziyi Wang, Shuaibin Li, Qian Zhao, Haian Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502. 000.n86002,7,,, 2025.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-cureka: Exploring the frontiers multimodal reasoning with rule-based reinforcement learning. arXiv preprint urXiv:2503.0733 2025.
Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Heukel, Benedikt Schifferer, Wei Du, and Igor Gitman. Aimo-2 winning solution: Building state-of-the-art mathematical reasoning models with openmathreasoning datasct, 2025. URL https://arxiv: wwtp.///.rraab///55/// 16811966...1661.111
Niklas Mucnnighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimolo. sl: Simple test-time scaling, 2025.
URL https: //appii//axxv...oo org/ abs/ 19398.
Yi Peng, Xiaokun Wang, Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyi- dan Xie, Li Ge, et al. Skywork rlv: Pioncering multimodal reasoning with thoough. arXiv preprint arXiv:2504.0553 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728 53741, 2023.
David Rein, Betty Li Hou, Asa. Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&aa benchmark. In First Conference on Language Modeling, 2024.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint 2rXiv:2402, 0000:02400 2024.
Wei Shen, Jiangbo Pei, Yi Peng, Xuchen Song, Yang Liu, Jian Peng, Haofeng Sun, Yunzhuo Hao, Poiyu Wang, Jianhao Zhang, and Yahui Zhou. Skywork-rlv8 technical report, 2025. URL https://arxiv..om abs/2507.066 06167.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang. Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. or Xiv preprint arXiv: 2409. 19256, 2024.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245-254, 1985.
Yiyou Sun, Georgia Zhou, Hao Wang, Dacheng Li, Nouha Dziri, and Dawn Song. Climbing the ladder of reasoning: What Ilms can-and and still can't-soly after sft? orXiv preprinl arxi::2004. 11 2024.
Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXie preprint orXiv:2503.20752.
2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with 1lms. arXiv preprint Xr:::2250 12599, 2025.
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentiviz ing self-reflcction of vision- laaionga models with reinforcement learning. arXiv preprint arXiv:2504.08897, 2025a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095 95169, 2024.
Peiyu Wang, Yichen Wei, Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jian- hao Zhang, Yunzhuo Hao, et al. Skywork rlv2: Multimodal hybrid reinforcement learning for reasoning. arXiv preprint ar Xiv:25004 16656, 2025b.
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang, Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025c. URL
http:///axxvv.rr/abbbbo 21080/
Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang. Advancing multimodal reasoning via reinforccment learning with cold start. arXiv preprint (al::25552233.. 2025.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint ar 1014v22503.10460, 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes, Ari S Mor- cos, Hongseok Namkooug, Ali Farhadi, Yair Carmon, Simon Kornblith, el al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Interna- tional conference on machine learning, pp.
PMLR, 2022.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. aii preprint orXiv:2, 11. 10440 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 2023.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, Bo Zhang, and Wei Chen. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint Xix::2503 10615, 2025a.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXim:2503. 10615, 2025b.
Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering milm with ol-like reasoning and reflection via collective monte carlo tree search. arxiv preprinl or arXie:2412, 18319, 2024.
Edward Yco, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yuc. Demystifying long chain-of-thought reasoning in llms. Xie preprint or arXiv:2502.03: 03373, 2025.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Tiu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint Xix::22003 14476, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi- discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao, R1-v1: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint Xiv:2500 1 12937, 2025.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kei-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024a.
Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning, arXio preprint 2rX 16198, 2024b.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL http://arxiv. http:://axxvvoorgaaate btwggabb//4400 13372.
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1-zero's' aha moment' in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.00 05112,, 2025.
A Appendix
A.1 Analysis with More Fine-Grained Separation of Difficulty Levels
To increse transparency and gramularity of the 5-level difficulty categorization and validate that core trends are not artifacts of discrete binning, we supplement the main analysis with 17-point quasi-contimuous results, where one data point stands for cach possible pass rate of the bascline model (Qwen2.5-VIL-7) across 16 independent runs, ranging from 0/16 (extremely hard, baseline never solves) to 16/16 (Irivially easy, baseline always solves). These results are visualized in Fig. IIG M. where each point represents the average accuracy gain of post-training method (long-Cof SFT, RL, two-stage training, etc.) relative to the baseline, calculated for all benchmark questions falling into that specific pass rate bin.
Consistent with the 5-level analysis in Fig M. the 17-point curves confirm three key trends: (1) For long-Con CoT SFT, accuracy gains transition from negative (for level to level 7, corresponding L1 casy questions) positive as the difficulty level increases, with the largest gains concentrated at level 15 to level 17 (aligning with L4L.5 retdmexy hard questions); (2) RL maintains more steady positive gains across level 3 to
16

---

Published in Transactions on Machine Learning Research (01/2026)
level 17, confirming its strength in generalizing to both easy and hard tasks; (3) Hybrid strategies (e.g.. interleaved training, model merging) continue to exhibit trade-offs rather than synergy: their curves hover between the SFT and RL curves for most pass rates, never combining SFTs high gains at low pass rates with RLs consistent gains at high pass rates.
Notably, the 17-point results exhibit slightly more fluctuations than the 5-level analysis, which expected given the smaller sample size per individual pass rate bin (e.g., fewer questions have an exact pass rate of 8/16 14/16, compared to the aggregated L3 or L1 bins). Despite this noise, the consistency in directional trends across both discrete and continuous representations confirms that the papers findings and conclu- sions, including SFTs difficulty-deeendent trade-off, RLs broad generalization, and the SFT-RL synergy dilemmaare robust to the choice of difficulty granularity.
A.2 Remarks on the Differences to Leaderboard Results
The performance of Qwen2.5-VL as reported in our paper, is slightly lower than the performance on the Open VLM Leaderboard across the MMStar, MathVista, and MMMU val benchmarks. We clarify that this slight discrepancy stems from minor variations in evaluation settings, which do not alter the core findings and conclusions of this paper. We have taken steps to align our experimental setup with the leaderboard much possible: we adopted GPT-4o-mini
the judge for MathVista and MMMU val evaluations, and GPT-4-0125 for MMStar, consistent with the leaderboards configuration. Additionally, we observed that the leaderboard results are derived from a single run (likely the optimal result selected from multiple attempts). Likewise, we reproduced the baseline performance of Qwen2.5- VL- by taking the maximum accuracy across independent runs. This maximum value is highly consistent with the leaderboard results (see Tab. m. For robustness against random errors, the main results reported in the paper are the average accuracy across runs, which is slightly lower than the maximum accuracy expected.
To reduce the cost of GPT API calls, used the locally deployed Owen2.55VL-322 as an alternative judge. Although the absolute accuracy values obtained with the Qwen judge are slightly lower than those with GPT judges, the trends in performance gains from SFT and RL remain consistent across both judge settings. Specifically, when using GPT as the judge, SFT leads to an average accuracy drop 1.2% and RL leads to an average gain of 3.0%; when using Qwen2.5-VT-322. as the judge, SFT results a 1.0% average drop and RL in a 3.7% average gain (see Tab. LI). This consistency confirms that the impact of SFT and RL on model performance is not dependent on the choice of judge, further validating our conclusions.
Table 10: Comparison of Qwen2.5 --- Performance Across Different Evaluation Settings.
Qwen2.5-VL-7B
Judge
MMStar
MathVista
MMMU val Avg.
Reported on Open VLM Leaderboard
GPT
64.1
68.1
58.0
63.4
Reproduced (Max. of 4 runs)
GPT
64.1
68.3
58.6
63.7
Reproduced (Avg. of 4 runs)
GPT
63.5
67.5
57.8
62.9
Table 11: Performance Gains of SFT and RL Under Different Judges.
Judge
Model
MMStar
MathVista
MMMU val
Avg.
Qwen2.5-VL VL-BB
63.5
67.5
57.8
62.9
GPT
+ SET
60.7
66.2
58.1
61.7 (-1.2%)
+ RT
66.5
72.6
58.7
65.9 (+3.0%)
63.2
66.4
53.5
61.0
Qwen2.5-VL-32B
+ SFT
60.8
65.6
53.6
60.0 (-1.0%)
+ RL
66.5
72.6
55.1
64.7 (+3.7%)
1 Open VL. Leaderboard.
http:///wwuuuptttecc/cccc/cse//s vla leaderboard
17

---

Published in Transactions on Machine Learning Research (01/2026)
A.3 Attempts to Elicit Long-CoT CoT with Prompt Engineering
We explored prompt engineering techniques to elicit long, structured Long-CoT reasoning traces from our target models. Specifically, we tested range of strategies: zero-shot prompts (e.g., 'Think through this step by step in dctail"), few-shot examples with verbose reasoning demonstrations, and itcrative prompting to encourage extended elaboration. However, these efforts yielded limited success: the models consistently produced concise, task-oriented responses rather than the detailed, multi-step reasoning we aimed for.
We attribute this to two constraints: 1) The tested models (S 7B parameters) lack the scale (2 100B) needed to generate complex, high-fidelits Long-Cor CoT traces. 2) Being instruction-tumedl they are optimized for direct, coherent responses aligned with instruction-follownng goals, which conflicts with the unconstrained verbosity required for Long-CoTI. This contrasts with base models like DeepSeek-R1, which retain flexibility to produce the 'aha moments' and extended reasoning seen in self-distileed Long-CoTl data.
MathVision
20
SFT + RL
Two-stage
Interleaved
X Progressive
Data Mixing
Model Merging
10
Accuracy Gain (%)
X
-10
-20
1
3
7
9
11
13
15
17
Difficulty Level (1=Easiest, 17=Hardest)
MathVerse
30
SFT RL
Two-stage
Interleaved
x Progressive
Data Mixing
Model Merging
20
Accuracy Gain (%)
10
x
-10
-20
1
9
11
13
15
17
Difficulty Level (1=Easiest, 17=Hardest)
Figure 16: Accuracy gains from various post-training techniques across 17 difficulty levels.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Math
SFT RL
Two-stage
Interleaved
x Progressive
Data Mixing
Model Merging
30
20
Accuracy Gain (%)
10
-10
20
-30
Difficulty Level (I=Easiest, 17=Hardest)
MMMU
30
SFSFT RL
Two-stage
Interleaved
x Progressive
Data Mixing
Model Merging
20
Accuracy Gain (%)
10
X
-10
-20
-30
13
Difficulty Level (1=Easiest, 17=Hardest)
MMStar
30
SFT RL
Two-stage Interleaved
Progressive
Data Mixing
Model Morging
20
Accuracy Gain (%)
10
X
-10
-20
1
7
9
11
13
17
Difficulty Level (1=Easiest, HarHard)st)
Figure 17: Accuracy gains from various post-training techniques across 17 difficulty levels (continued).
19


Reviews:
[{"review_id": "jdgjF097Tc", "reviewer": "Reviewer_Jkqh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper empirically compares long chain-of-thought (CoT) supervised fine-tuning (SFT) and reinforcement learning (RL) for reasoning in large vision-language models. SFT improves performance on hard problems via structured, step-by-step reasoning but tends to induce verbosity that can hurt easier tasks. RL yields more concise, broadly improved answers but offers smaller gains on the hardest questions. Combining the two—via two-stage, interleaved, and progressive training, data mixing, or model merging—reflects a “synergy dilemma”: rather than additive benefits, the combination leads to a trade-off between the two approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by experiments comparing long-CoT SFT and RL on five VLM benchmarks (MathVision, MathVerse, MathVista, MMMU-val, and MMStar-val) using the Qwen2.5-VL-7B model. The combination of the two is examined via training alternation (two-stage, interleaved, progressive), data mixing, and model merging. The evidence is consistent with the stated “synergy dilemma”: attempts to combine SFT and RL do not yield additive gains. The analyses (e.g., accuracy gains versus average length trade-offs) are presented clearly.\nHowever, the explanation for why VLMs exhibit a “synergy dilemma” when combining long-CoT SFT and RL, in contrast to the improvements sometimes observed in LLMs, remains insufficiently clear and not fully convincing.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a topical question—how long-CoT SFT compares with RL for reasoning in vision-language models—and reports negative/neutral results on their combination (the “synergy dilemma”). Given the current interest in multimodal research, this work may be of interest to parts of the community, even as a negative result.", "requested_changes": "- Include at least one additional base model to test whether the conclusion still holds beyond Qwen2.5-VL-7B.\n- Report results across multiple random seeds and assess statistical significance (e.g., by including variance or error bars).\n- Add a brief paragraph explaining why VLMs exhibit a “synergy dilemma” when combining long-CoT SFT and RL, whereas some LLM papers report improvements; summarize plausible factors and indicate which are supported by your experiments.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RDv3ZjF8sx", "reviewer": "Reviewer_XFAT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper examines the roles of two key steps (long-cot SFT and RL) in the VLM post-training stage. Authors explored the distinct results and the combined results from three different perspectives and found interesting conclusions - the synergy dilemma of long-cot SFT and RL, which shows combining two methods does not lead to performance gains. Additionally, authors also investigate through detailed analysis, such as token KL divergence.\n\nStrengths:\n\n1. The paper structure and writing are clear and easy to understand. The figures are clear and sufficient.\n2. The result comparison is extensive and fair.\n\nWeaknesses:\n\n1. The datasets and models selected are insufficient to demonstrate the conclusion. More datasets and models at different scales are needed.\n2. Paper lacks more analysis, such as the key reason why VLMs behave differently from LLMs.", "claims_and_evidence_yn": "No", "claims_explanation": "As mentioned, to demonstrate the existence of the synergy dilemma in VLMs (a relatively serious claim), more datasets and models at different scales are needed.", "audience_yn": "Yes", "audience_explanation": "The research question of the paper is still interesting and provides a data point that I believe can foster future works, despite its weaknesses.", "requested_changes": "If compute resources allow, please address the aforementioned weaknesses.\n\nA few more questions that I am interested in and might be added to the manuscript:\n\n1. The core difference that makes VLMs behave differently from LLMs.\n2. If using rejection sampling instead of the Eureka-Distill dataset, will the model still exhibit the verbose reasoning process? Typically, RL would elicit a surge in response length and 'aha' words while the opposite is observed in this paper.\n3. According to the results, for the L1 questions, almost all methods lead to a performance drop. What is the reason behind?\n4. For the model collapse during RL training, what could be the reasons behind?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IScyvsuWNX", "reviewer": "Reviewer_Teh3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper compares finetuning of multimodal reasoning models through Reinforcement Learning (RL) and Supervised Finetuning (SFT). By fine-tuning Qwen2.5-VL-7B, the paper finds that SFT only achieves performance gains on the hardest problems, while RL achieves performance gains on all difficulty levels, but less pronounced ones than SFT on hard problems. Experiments on how to combine both training strategies yield negative results.", "claims_and_evidence_yn": "No", "claims_explanation": "The paper promises a systematic investigation into the impact of SFT and RL on long-form reasoning in MLLMs. However, the experimental setup of the paper is limited in the following aspects:\n\n**(L1)** All experiments involve only one model, `Qwen2.5-VL-7B`. However, to show that results generalize, it is necessary to test multiple models. For example, experiments could be conducted with 2 models of different sizes (e.g. 2B vs. 7B) using models from 2 different model families (e.g. Llava vs. Qwen-VL) .\n\n**(L2)** The \"systematicity\" of the study is not properly defined. For this, I would expect a clear definition of which axes of variation are evaluated, and why they are the most relevant. It is unclear if the experiments on combining SFT and RL are a collection of potential solutions or follow any structure, e.g. one derived from a literature survey or accepted taxonomy of the field.\n\n**(L3)** The performance gains achieved through finetuning (both SFT and RL) are consistently below the online leaderboard performance of Qwen2.5-VL-7B [a] (concerning MMStar, MathVista, and MMMU val). Since no confidence intervals or similar measures are reported, it is unclear how significant the reported results are.\n\n**(L4)** The categorization of problems into difficulty levels is not well motivated, and the choice of passing rate intervals in Sec. 2.3 is not explained. For example, instead of using discrete difficulty levels, these results could be represented as a curve showing the accuracy on the problems that pass at least $k$ times, showing 16 points in the proposed setup. This would allow a more transparent assessment.\n\n[a] https://huggingface.co/spaces/opencompass/open_vlm_leaderboard", "audience_yn": "Yes", "audience_explanation": "The research questions asked in this paper are very important, and providing answers could be very impactful. Finding ways to effectively combine RL and SFT, as well as understanding their respective strengths and weaknesses, has important implications for training stronger and more generalizable models.", "requested_changes": "To improve the validity and show the reported insights generalize, the paper should\n * Show that findings hold when varying model properties along meaningful axes of variations, such as model size and model family (as a proxy for training data and objective)\n * Clearly define which aspects of the study are \"systematic\" and what is the underlying system from which experimental variations are derived\n * Strengthen the validity by reporting confidence intervals, error barss, or p-values; or alternatively provide justification why the reported performance differences are statistically significant.\n * Explain the differences to leaderboard performance\n * Better explain the categorization of problems into difficulty levels, or present results in a way that is independent of such classification\n\nI believe that these changes are necessary to prove the validity of the presented conclusions, and including these recommendations will significantly strengthen the paper.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)