SPECS: Decoupling Multimodal Learning via  Self-distilled Preference-based Cold Start
Authors: Kun Chen
Peng Shi
Haibo Qiu
Zhixiong Zeng
Siqi Yang
Wenji Mao
Lin Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Vision Language Models
Reinforcement Learning
Reasoning
Cold-Start
Preference Optimization
Direct Preference Optimization (DPO)
Self-Distillation
Primary area: foundation or frontier models, including LLMs
TL;DR: Our Self-distilled Preference-based Cold Start (SPECS) method better prepares Vision Language Models for reinforcement learning, significantly boosting their reasoning performance.

Abstract: Reinforcement learning with verifiable rewards (RLVR) has recently catalyzed a wave of "MLLM-r1" approaches that bring RL to vision language models. Most representative paradigms begin with a cold start, typically employing supervised fine-tuning (SFT), to initialize the policy before RL. However, SFT-based cold start adopts the reasoning paradigm intertwined with task solution and output format, which may induce instruction-style overfitting, weakens out-of-distribution generalization, and ultimately affects downstream RL. We revisit the cold start along two views, its training method and data construction, and introduce the Generalization Factor (GF) coefficient to quantify the generalization capability under different methods. Our empirical study finds that preference–based training methods (e.g. DPO) generalizes better than SFT-based methods in cold start. Motivated by this, we propose $\textbf{SPECS}$—a $\textbf{S}$elf-distilled, $\textbf{P}$r$\textbf{e}$ference-based $\textbf{C}$old $\textbf{S}$tart framework that decouples multimodal learning: (1) generates introspective preference data pairs via self-distillation, avoiding reliance on larger teachers or manual annotation; (2) performs preference–based training to learn, focusing on shallow, transferable surface-form criteria (format, structure, style) rather than memorizing content; and (3) hands off to RLVR for deep reasoning results. Experimental results across multiple multimodal benchmarks show that our decoupling learning framework yields consistent performance gains over strong baselines, improving MEGA-Bench by 4.1\% and MathVista by 12.2\%. Additional experiments indicate that SPECS contributes to reducing in-distribution "stuckness," improving exploration, stabilizing training, and raising the performance ceiling. 

Project Page: https://kwen-chen.github.io/SPECS-VL/

Published as a conference paper at ICLR 2026
SPECS: DECOUPLING MULTIMODAL LEARNING VIA SELF-DISTILLED PREFERENCE-BASER COLD START
Kun Chen 1,,,, Lanll,,,Pee,ee Peng Shi Haibo Qiu?, Zhixiong Zeng?.2, Siqi Yang", Wenji Mao
Lin M
School of Artificial Intelligence, University of Chinese Academy of Sciences MAIS,, Institute of Automation, Chinese Academy of Sciences
3 Meituan
{chenkun2024, wenji mao)e a a.aa..c.... cn. shipengl 00me tuan. com, forest 1inma@ gmail com
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) has recently catalyzed a wave of "MLLM-r1" approaches that bring RL to vision language models. Most representative paradigms begin with a cold start, typically employing supervised fine-tuning (SFT), to initialize the policy before RL. However, SFT-based cold start adopts the reasoning paradigm intertwined with task solution and output for- mat, which may induce instruction-style overfitting, weakens generalization, and ultimately affects downstream RL. We revisit the cold start along two views, its training method and data construction, and introduce the Generalization Factor (GF) coefficient to quantify the generalization capability under different methods. Our empirical study finds that preference-baset training methods (e.g. DPO) generalizes better than SFT-based methods in cold start. Mo- tivated by this, we propose SPECS Self-distilled, Preference-ben based Cold Start framework that decouples multimodal learning: (1) generates introspective prefer- ence data pairs via self-distillation, avoiding reliance on larger teachers or manual annotation; (2) performs preference-basee training to learn, focusing shallow, transferable surface-form criteria (format, structure, style) rather than memoriz- ing content; and (3) hands off to RLVR for deep reasoning results. Experimental results across multiple multimodal benchmarks show that our decoupling learning framework yields consistent performance gains strong basclines, improving MEGA-BEN by 4. % and MATHVISTA by 12.2%. Additional experiments in- dicate that SPECS contributes reducing in-distribution "stuckness," improving exploration, stabilizing training, and raising the performance ceiling. Project Page:
github. io. Lo/SPECS-VL/
INTRODUCTION
2025), effectively enhancing the reasoning capabilities of large models through reinforcement learning (RL) with verifiable reward (Lambert et al., 2024; Guo et al. 2025). a growing body of work has begun to apply RL directly to This has led
Recently, inspired by the success of DeepSeek-RI (Guo et al.
vision language models (VLMs).
of exciting "MLLM-rI" research (Meng
et al. 2025|0 Shen et al.
2025
Peng et al. 2025
Zhou et al.
2025
Zhang et al. 2025b Wang et al.,
2025b Zheng al."
2025
Ma et al. 2025
et al. 2025 Quu et al. 2025). which leverage
similar principles to ad advance multimodal reasoning.
Previous research has indicated that prior RL, employing a pre-training or warm-up phase (which is termed "cold start"), can significantly improve the readability, stability, and even the final perfor-
mance of RL training (Guo et al. 2025) Currently, the most commonly used cold start strategy is
supervised fine-tuning (SFT). where the model is first fine-tuned on a set of high-quality reasoning
data to provide better initial policy for the subsequent RL phase (Wei al.) 2025
Yang et al.
"qqual contribution. CCoreesponding authors.

---

Published as a conference paper at ICLR 2026
20256 Huang et al. 2025 Deng et al., 2025b). This strategy enables the model to be trained on complex reasoning data during the cold start phase, thereby acquiring reasoning ability.
The common understanding behind SFT-based cold start is that reasoning abilities, reasoning for- mat and other learning objectives can be jointly Iearned during the cold start phase. However, such an SFT-based joint learning paradigm may largely affect the model's generalization capability (Wul et al. 2025 Chu et al., 2025), and consequently degrade subsequent RL (Chen et al.] (2025a). This raises an important research issue of quantifying and improving the model's generalization capabil- ity during cold start and working in concert with subsequent RL.
To address the above limitations, consider alternative learning paradigm, which separates the learning process into hierarchical stages based on the idea that cold start phase focused more on shallow learning to avoid prematurely getting stuck in in-distribution problem solving, while subse- quent RL focuses on the deep-level learning of solution to boost the overall performance (Bengio al. 2009). Thus, the intuition of our adopting decoupling learning for multimodal reasoning is that the selection of pre-training methods in cold start needs to better support the subsequent RL, both in terms of generalization and by having separate objectives to facilitate better final results.
Another important issue is the generation of cold start data. Previously, the prohibitive cost of human annotation has motivated a growing body research to explore the use of synthetic data. This often involves using more capable large model as a "teacher" to distill data for smaller "student" model. (Zhang et al. 2025c: Yao et al. 2024; Xu et al., [2024] Huang et al. 2025). However, when the capability gap between the teacher model and the student model is too large,
lead to decline in model performance (Zhang et al. (023). Alternatively. the DeepSeek-RI- RI-Zero paradigm (Guo) et al. 2025) first directly applies RL to the base model for obtaining RI RI-Zero and then generates cold start data by zero model itself. This paradigm has achieved very remarkable performance; yet it still has the limitation of reliance on the SFT cold start and the constraints between SFT and subsequent RL, thereby leaving room for further improvement.
In this paper, to examine the suitable cold start training method, we propose the Gemeralization Fac- for (GF) coefficient in Section 2 to quantify the generalization capability of the model and conduct an empirical study evaluate different training methods. We identify that Direct Preference Opti- mization (DPO) (Rafailov et al. (023) based on preference data is cold start approach that enables the model to have better performance. On this basis, we present the Self-distiled Preference-bast Cold-Start (SPECS) framework in Section 3 By decoupling the learning objectives during DPO to focus on output format, we create a pre-aligned model that serves as a superior starting point for the final RL fine-tuning, Our experiments show that this method leads to more stable, efficient training, and a higher performance ceiling compared to the advanced and strong baseline.
The main contributions of this paper can be summarized as follows.
1. We present the SPECS framework, a three-stage cold start strategy. It generates preference data through self-distillation, uses DPO for cold start training, and separates training objectives so that the model first aligns with output formats, providing a stronger starting point for RL.
2. We propose Generalization Factor as a metric to evaluate a model's generalization capability under different cold start training methods by comparing its performance in-distribution and out-of-distribution tasks.
3. We reveal the importance of Decoupling Learning between the cold-start and RL phases. This separation improves exploration and reduces the risk of the model getting stuck on in- distribution solutions.
4. Our experiments prove that a DPO cold start gives the model stronger generalization ability. In terms of the model's 's final results, it achieves consistent performance gains across benchmarks, improving MEGA-Bench by 4.1% % and Math Vista by 12.2% over strong baselines.
EMPIRICAL INVESTIGATION
2.1 EVALUATING DEGREE OF GENERALIZATION
To evaluate the impact of preference-base versus supervised data on a model's generalization ca- pabilities under fixed sample size, we introduce the metric of Generalization Factor (GF).
2

---

Published as a conference paper at ICLR 2026
Setup. We define evaluation function (f P) R that measures the performance of model ff on a data distribution P and n refers to the size of the training data samples. A higher value of e indicates better performance.
Generalization Factor. To accurately evaluate the generalization ability of a model, we first need to test the model's performance on in-distribution (ID) and out-of-distribttion (OOD) tasks. Among them, ID tasks require the same as the task requirements during training, while OOD tasks require different from the task requirements during training.
ID Performance: (n),n), is evaluated on hold-out set from the same distribution Pirain-
(nn(n) ( Parain)
OOD Performance: WOOD(n), the weighted average performance across a set of m distinct OOD distributions, Q Qm. with weights defined by a distribution
J.
WOOD (n) = Eq-a (vnn Q))
We establish baseline model, fo. which serves as reference point. The performance gains over this baseline are calculated as:
GiD(n) = (ID(n) - VID
GOoD(n) N (n) OOD
We define GF, ((n) as the Fg-score of the model with respect to OOD performance gains and ID performance gains. The reason for adopting this metric is that the F3-score is particularly suitable for average ratios. Its most prominent feature is that the result tends to lean toward the smaller number. This perfectly aligns with our needs: as long as either the ID or OOD performance is very poor, the final score will be very low. We can also control the size of B to reflect the degree of importance we attach to OOD performance gains during the training process.
r(n) = (1 + 8²) Goo(n)oooo(z)
B²2 GiD(n) + GooD(n)
where the weighting coefficient 3 is generally set to to reflect the importance of the OOD perfor- mance gain in the generalization capabilities of the model. To ensure that the metric behaves well and is dimensionless, the evaluation function yb should be normalized to a consistent range.
2.2
EXPERIMENTAL FINDINGS
ID Performance
OOD Performance
Generalization Factor
100
100
100
Generalicaiion Facctor
000 Performance
Performance
60
60
40
40
thiise
DP0
line
DRD w. <B locc
BAg
oro
DTO
SFT
Training
Taining
Training Step
Figure 1: Performance Comparison: DPO vs. SFT on In-Distribution and
Task
To preliminarily examine how preference data and supervised data affect model generalization, we construct a preference dataset Preet { v." Mi J where y, is the chosen response and y/ is the rejected response, and supervised dataset DSFT y(x,,))) H with y: y," around reasoning tasks defined by a specific answer format. Under equal data budgets, we evaluate two settings: (i) an in-distribution setting in which the required reasoning format matches that used in training, and (ii) an
setting in which the required reasoning format differs We compare DPO training, SFT training, and DPO training augmented with SFT loss (see Section 3.3
The resulting V/D (n), (n), and the I'(n) are reported in Figure]
For detailed description, see Appendix B
3

---

Published as a conference paper at ICLR 2026
From the experimental results, it can be observed that SFT achieves the fastest convergence ID tasks. However, due to its reliance on a single cross-entropy loss that maximizes the log- likellhoooodd of the correct answer, it demonstrates poor OOD performance. By contrast, DPO converges more slowly at the beginning of ID tasks but yields better OOD performance. Remarkably, the model trained with combination of DPO with SFT loss achieves the strongest generalization capability overall. As the number of training steps increases, the GF gap between the SFT training method and the DPO training method also increases.
METHODOLOGY: THE SPECS FRAMEWORK
3.1 SELF-DISTILLED PREFERENCE COLD-START
A model with superior generalization capabilities provides a more effective starting point for RL. Inspired by the discussion in Section 2 we employ self-distillation construct preference data focusing format learning. This data is then used in place of standard SFT data to enhance the model's s generalization performance during the cold-start phase.
To implement this, we propose SPECS, illustrated in Figure 2 three-stage training optimization strategy consisting of 1) Self-Distllaation for Preference Data Generation, 2) DPO-based Pre- Alignment for Cold-Start, and 3) Final GRPO Fine-tuning.
Sell-Distillotion Preference- bosed Cold Start Stap1 Oura-zero RL Training and Preference Data Generstion
Scep2 DPO:
Phrroporne withen etnees otheko
O
Ansear filter
Format
Prafereace
foreot pollution
Base
Base Nodel
rejectede
rejected
GAPO
Step3 Final 68P6 Fine- Tuntin
Ansear filter
fornot Supplement
Ours-zero
choeeny
chosen
Curs-2ero
Rejected Resporse Pollution.
Chosen Response Fileration.
Solution
Bemoa
Figure 2: Method Overview. We propose the SPECS cold-start strategy, a three-stage pipeline to enhance final RL fine-tuning. Firstly, where we generate preference dataset focused on teaching the correct output format by self distillation. Next, The base model is pre-aligned on this data using DPO to create a format-aware "Warmup Model". Finally, this pre-aligned model undergoes Final RL tuning with GRPO, allowing the optimization process to focus on enhancing reasoning.
3.2 SELF-DISTILLATIE FOR PREFERENCE DATA GENERATION
The foundational stage of our framework aims to achieve two interconnected goals: first, cultivate preliminary "seed model" with enhanced reasoning capabilities, and second, to leverage this model to autonomously generate a high-quality preference dataset through a process we term self-distillation.
Objective:
Methodology: A critical initial challenge is that standard base VLM often lacks the capability generate outputs of sufficient reasoning ability. To address this, we first conduct brief. initial phase of RI fine-tuning on the base model using GRPO. This step aims not at achieving the final performance, but at creating an initial policy, denoted RGRPO-2ero: which more adept at exploring the solution space.
With the exploratory TGRPO zero model, we proceed to generate the preference dataset. The data construction process involves four key steps:
Response Generation. We prompt two models, our exploratory AGRPO- and Thase with specific format instructions dataset, which is designed to contain pairs of responses that are both correct in their final answer, but differ in their reasoning paradigm and answer format.
bobinn>..</thinn-aaaate
/aw.s///n/www>>a to create
4

---

Published as a conference paper at ICLR 2026
As shown in the figure, it known that <AED DE 10, 5, the length of BC is
4A, we SADE 3AC3. So, S\free(/EH(BB Hfrac(DE) Given 10, AE 5, and DE can calculate $Mfroc(5) Mfrccf0H8CSS. Therefore, $BC Mfrac122H1} Thus, the answer is 12.
Since </ED
Model
<thinio Given criterion). Therefore, the rotios of curresponding sides are equal, which gives us:
4CAB, conclude
similarity
Af (BrCc(DCHBC) (Ar Minplies \froc(6MBC)) (frac(5)11)) Vimpliss (frac(6)(C)) Xfracf1H22 Vipliis BC = 12
Ours-zero
Thus, the length of BC 12. </thinko
<ondwer> The answer is $1boxedf12)$ c/onswer>
Figure 3: Example of self-distilled preference data pair.
use Gemini-2.5 flash (Co- manici et al. 2025) as an evaluator. Assesses whether the reasoning path in the TGRPO response aligns correctly with its final answer. Only responses in which the reasoning and the answer are consistent are retained, forming high-quality pool of candidates. For more analysis of this content, please refer Appendix
Chosen Response Filtration. For the chosen response (v,*). ).
Rejected Response Pollution. For the rejected response ( ). we select responses that also contain the correct answer, but deviate from the required format. Recognizing that some gen- erated responses might incidentally have the correct format, We randomly apply one of the following five types of format corruption to these responses to ensure a clear learning signal.
1. Remove all tags (<think>, </think>,
2. Remove the <answer> and </answer> tags.
3. Remove the <think> and </think> tags.
4. Remove the <answer> and </answer> tags and move the closing </think> tag to the end of the response.
5. Replace the <answer> tags with the string Answer: and remove </answer> tags.
Preference Pair Construction via Self-Distillatio.. We construct the chosen response and
the rejected response into pairs of self-distilled preference data (v²,," As shown in Figure
3 Both Chosen Responses (y,+) and Rejected Responses (v₂") are selected from the filtered
pool and contain the correct final answer. This data set is designed to facilitate decoupled learning, separating the learning of reasoning paradigms and answer formats from the core
logical reasoning ability. This approach serves as a more effective cold-start method for the final alignment stage.
3.3 DPO-BASED PRE-ALIGNMENT FOR COLD-START
Objective: The primary goal of this stage is leverage the self-distileed preference dataset gener- ated in the Stage (Section 3.2) to pre-align the base VLM. This process yields a "cold-start" model that serves as significantly improved starting point for the final RL fine-tuning. We conceptualize this phase "warrmupp"" which shifts the model's policy into more advantagcous region of the policy landscape before the intensive final training.
Methodology: To achieve this pre-alignment, we employ DPO (Rafailov et al. 2023 powerful technique that directly optimizes the language model on preference data without the need for an explicit reward model. The standard DPO loss function is defined as:
LDPO(TO; meet) -R (log (9log
(T (x) Teat (Vro
To Te (x)) Teet (31 (x))
3 log
where Ty is the policy being optimized, Fref is the reference policy (the initial base model), / is temperature parameter, and (x, Fues 31) represents a triplet of prompt, chosen response, and rejected response from our self-distilled dataset D.
5

---

Published as a conference paper at ICLR 2026
To augment this process, incorporate SFT loss computed the "chosen" samples, which serves as form of regularization. It ensures that while the model learns the directional preference signal from DPO, it does not drift far from the core distribution of high-quality text embodied by the chosen responses |Rao et al. 2025). The combined loss function thus:
Lhybrid EDPO + XCSFT
where LSFT is the conventional negative log-likelihood loss the chosen responses, and À is weighting coefficient to balance the two objectives. For discussion for X, see Appendix
3.4 FINAL GRPO FINE-TUNING
Objective: To achieve peak performance by fine-tuning the pre- cold-start model, focusing computational resources on enhancing complex reasoning capabilities.
This final stage leverages the cold-start model obtained from Stage as the initial- ization point for RL, rather than starting from the base model or a conventional SFT model. The pre-alignment from the DPO phase ensures that the model has already mastered the output format. Consequently, the model is not required to expend resources on learning basic structural compli- ance, Instead, credit assignment during RL training can more accurately attributed to the core challenge: improving the quality and precision of its reasoning process. This targeted optimization explains the observed stable convergence in our experiments and the model's ability to achieve a higher performance ceiling.
Methodology:
For the final stage of fine-tuning, we employ the GRPO algorithm (Shao et al. 2024). This process is guided by composite reward function that combines format and accuracy components to evaluate the model's output, o, for given question,
The total reward Rootel, is the of format reward Roormats and accuracy reward Racei
Reotal (o, q) Rfoommat (0) + (o)
The format reward Rtormat (o), assigns a fixed value of 0.5 for structurally correct outputs, rein- forcing the policy's formatting discipline.
The accuracy reward Race (o, q), provides a binary signal: 1.0 for a correct answer and 0 other- wise. We use hybrid mechanism determine correctness based on the question type, T(g):
Race(o,g) q) = Rrute(O, ④) if (4)) E (Multiple-Choice, Numerical)
(o,
if (q)) = Short-Answer
For objective types like multiple-choice and numerical questions, a rule-based function assesses correctness. For subjective short-answer questions, employ GPT-4o as external judge.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
The data utilized for training TGRPO-sen model in Stage I and for the final GRPO fine-tuning of the cold-started model in Stage is composed of the Orsta47K (Mal et al. 2025) and vir139K (Wang et al.) 2025a) datasets. In Stage 2 of cold start training, we used 9K self-distilled data. This composition designed enhance the model's general and mathematical reasoning capabilities. We conduct evaluations multiple benchmark datasets, including MEGA- Bench (Chen et al.) 2025b), MMMU (Yue et al.) 2024). Math Vista (Lu et al. 2024b). MATH-Vision Wang et al. 2024a). and Math Verse (Zhang al. (024)).
Dataset and Benchmark:
Baseline: Our comparative analysis is grounded on two primary categories of models. The first category comprises open-source general VLMs, including QwenVL-2-7B (Wang et al. b024b). Owe25L-2 (Bai et al. 2025). VL2-8B (Chen et al. 2024]. Intern VL2.588B (Chen et al.]
2024). Kimi-Vis VL-A3B (Team al. 2025). ) and DeepSeek- VL-7B (Lu al. 2024a). The second cat-
egory focuses on models specifically engincered for advanced reasoning tasks. This group includes
Kimi Limi-VL-A3B-Thinkeee (Team al. 2025), R1-Onevision (Yang al. 2025b VI.AA-Thinking

---

Published as conference paper at ICLR 2026
Table Model performance comparison on MEGA-Bench Core.
MEGA-Bench
MEGA-Bench
Knowledge
Mathematics
Perception
Coding
Info.
Plumning
Science
Metrics
Core
Open Source General Models
QwenVL-2--B
39.96
25.95
39.99
31.49
40.29
16.64
28.59
34,47
QwenVL-22.-7B
38.84
27.67
41.24
28.93
50.25
16.32
36.75
41.64
35.07
VLtom
33.94
22.08
32.15
24.7
29.13
12.17
24.61
39.96
25.96
V1.2-88
33.27
Intern VL3-8B
42.76
34,85
42.76
34.05
44.84
17.10
35.21
49.60
36,02
Llvva-V-7BB
31.37
22.11
27.64
13.9
17.07
9.16
24.38
37.31
21.36
Kimi-VI LLmi-VL-A3B
37.63
27.07
39.50
22.30
40.99
33.94
34,40
Reasoning Miillls
RI-Oneeission'
29.47
20.94
28.65
23.38
43.04
12.67
26.84
42.19
27.18
VLAA-Thinking?
Kimi VL-ABB Thinkiigg
33.45
17.76
28.11
14.69
41.14
12.64
28.60
43.97
27.08
MM-Burcka-7B
40.12
31.59
39.71
28.75
49.32
16.64
37.25
46.39
35,96
VL Rethinkerr-7
40.65
30.08
42.02
29.87
$2.03
17.83
36.82
46.90
37.25
Orsta-78
41.63
31.48
43.84
32.82
54.07
17.83
36,91
41.66
38.31
Ours-zero
42.44
29.87
43.77
49.59
17.76
37.39
47.32
37.96
Ours-BB
42.64
31.71
44.58
34.14
51.68
18.76
38.73
51,87
39.17
(Ours Backbone)
+3.8
+4.0
+3.3
+5.2
+144
+2.4
+2.0
+10.2
+4.1
evalisated VI.MEvalK remaining results from MEGAA-Benc Leadeshoard and Ma
indicutes
2025
Table 2: Model Performance Comparison On Other Benchmarks
MMMU val
Math Verse vision only
Overall
Model
Vision
Math Visita
Backbone
OwenVL.-2.5-77
54.24
25.40
63.70
38.20
45.38
QwenVZ-2.5-78 based Reasoning Models
E1-Onevision
49.67f
29.90
64.1
40.0
45.92
"LAA-Thinking
52.67+
26.40
68.00
48.20
48.82
MM-Bureka-7IB
55.55t
26.90
73.00
47.58t
50.76
BL-RRethinke-7B
56.7
29.70
73.60
48.98
52.25
Orsla-7Bt
54.33
25.76
70.20
32.10
45.60
Ours-zero
54.3
26.88
72.90
47.33
50.35
Ours-7B
56.78
29.50
75.90
48.73
52.73
A (Ours Backbone)
+2.5
+4.1
+12.2
+10.5
+7.3
The *symbol indicates that results were evaluated with VLMEvalKi
Chen et al. 2025a), MM-Bureka-7B (Meng et al. 2025
VL-Rethinker--7B
(Wang et al. 2025a
and Orsta- 7B (Ma et al. 2025)
Implementation Details: We utilize the open-source Multimodal Large Language Model, Qwen2.5-VL-7B (Bai al. 2025)_ as our base model. For the GRPO training in Stage and Stage 3, we employ the MM-EUREKA framework. The training batch sizes are both set to 128, with rollouts generated per sample. For the DPO training in Stage 2, as well for the comparative SFT experiments, we leverage the LlamaFactory | framework. In this configuration, the training batch size set to 64, and the hyperparameter for the hybrid loss function set to 1. The prompt used during training is shown in Appendix B Some more detailed settings can be found in Appendix F
4.2 MAIN RESULTS
presents the overall performance of our model on MEGA-Bench Core, in comparison with other baseline models. Table 2 reports the performance of various inference models built on the Qwen VL-2.572 backbone across additional benchmarks. Our model has improvements in general task benchmarks (MEGA-BENCH core, MMMU) and mathematical reasoning benchmarks (Math- Vision, MathVisita. Math Verse), and some benchmarks in leading position among models of the same size, demonstrating the effectiveness of our approach.
Table

---

Published as a conference paper at ICLR 2026
4.3 ABLATION ON GELF-DISTILLTTON AND DECOUPLED DATA STRATEGY
Self-distillation proves more effective than external teacher models. First, we evaluate the ef- fectiveness of the self-distillation mechanism by substituting it with preference data generated from powerful external teacher models, specifically QwenVL-22.5322B and QwwnVVL-2.5772 shown in Tabel Clearly indicate that our our approach outperforms both teacher-based alternatives. We also observe that performance degradation is more pronounced when using the QwenVL-225. 32B model, whose output distribution diverges more significantly from base model. This finding suggests that preference data closely aligned with the model's intrinsic capability distribution is more effective for alignment than guidance from a more capable but dissimilar external model.
The results
Distilling from GRPO-zero instead of the base model. We directly perform RL on the base model to obtain GRPO-zero, and then distill the chosen response through the GRPO-zero model. This choice of scheme is based considerations of data utilization and training data quality. As shown in Table 33 below, we have conducted statistics on some indicators of the responses of the original model and the GRPO-zero model to the training data questions. Obviously, the GRPO-zero model has higher data utilization in the collection of chosen responses due to its higher format accuracy and answer accuracy. At the same time, we counted the number of reasoning words (including transition words, causal words, sequential words, etc.) per 1000 characters for both models. We also used the same data collection method to collect responses with correct formatting and correct answers distilled from the base model as chosen responses. The experimental results shown in Table The
Table 3: Statistical indicators of response in training data for the base model and GRPO-Zero model
Model
Format Acc. (%)
Answer Acc. (%)
Reasoning Words 1k Chars
Qwen2.5-7B Frren..57BBInsstuee
41.62
30.42
4.26
Our--GRPO-zero
96.74
$2.82
4.99
decoupled data strategy outperforms the coupled approach for DPO cold-starting. Next, we investigate the impact of our decoupled data strategy for DPO cold-starting. We compare it against a "coupled"' DPO approach, where preference data is mixed, containing pairs that differ in both answer correctness and reasoning format. The experimental results shown in Table 4 demonstrate the clear superiority of the decoupled approach. We found that while coupled data helps initially, decoupled data provides better foundation for the main RL phase. We attribute this to decoupled data's sharp focus: it trains only the reasoning paradigms during the cold start, which ultimately leads to better performance after RL, even if the initial cold-start performance lower.
Table Ablation Results to show the impact of Self Distillation and Decoupled Data
Model
Megabench
MMMU
Vista
Math Vision
VathVerse
AVG
Qwen- VL-2,5-7B
35.07
54,2
63,70
25.40
38.20
43.31
Qwen32b Distillation
/7.04/29.89
51.44 56.67 /5
66.90/ / 71.50
25.53 / 28.03
43.53 / 46.07
42.89/
/46.43
Qwen72b Distillation
34.00 /7.30
53.89 / 58.56
67.50/ / 73.30
25.62 / 28.91
43.53 / 46.83
44,90 /88.98
Base model Distillation
35.37 37.992
53.11
56.11
67.90 74.40
25.55
28.68
43.40 / 46.82
45.07 48.79
Seef Distillation
37.52/39.11
54.89 56.78
72.00/ / 75.90
25.75 29.50
48.773
47.27
50.02
Coupled Data
37.02/38.76
55.44/55.44
71.10/ 73.10
27.37 28.65
47.46/47
47.67 48.68
Deccoupled Dota
39.52/39.17
54.89 / 56.78
72.00 75.90
25.75
29.50
46.19 448.73
47.27 50.02
Thee value the side the slash represents the score after cold start RI.
represents the score after cold start training, value the right side of the
4.4 ANALYSIS OF THE IMPACT OF DPO-BASED COLD START AND SFT-BASED COLD START
We examine the downstream effects of our DPO cold-start strategy, assessing its impact on the efficiency and stability of the final RL phase.
Performance and Training Efficiency. To evaluate performance and training efficiency, we tracked HEEGA-Bench scores throughout the GRPO training process. As illustrated in Figure 4 the DPO- based GRPO model begins with a substantially higher initial score, demonstrating the immediate benefit of preference-based pre-alignment. Furthermore, it maintains clear advantage throughout
8

---

Published as a conference paper at ICLR 2026
Checkpoint Comparison
Policy Loss Curves
Format Rewards
Wex 6m
1:13
893
20-
0.003
Format Rowardds
Pollcy Loos
0. 0.495
Perforance
88
0.002
0.480
0.001
37-
0.485
Dare
0.000
6/0am -ST Besk
270-0100
400 Training Stops
200
500
0.480
200
400
600
Training Step
Training Stop
Figure 4: Impact on RL Training Efficiency and stability.
training, converging more rapidly and ultimately achieving a higher performance ceiling than its SFT-based GRPO counterpart. The final performance comparison of other benchmarks is shown in Table 5 below. For more analysis this content, please refer to Appendix D
Table Performance Comparison of SFT-based, and DPO-based Models on Various Benchmarks
Model
Megabench
MMMU
Math
Math Vision
MathVerse
AVG
Qwen2.5-7B Intttntcctt
35.07
54.20
63.70
25.40
38.20
43.31
SFT-hased GRPO
37.52
54.44 56.78
74.10
28.61
43.60
47.65
DPO-based GRPO
39.17
75.90
29.50
48.73
50.02
Training Stability. Beyond performance metries, we analyzed training stability by comparing the policy loss curves, presented in Figure The curve for DPO-based GRPO is visibly smoother and more stable, indicating a more consistent and reliable optimization trajectory. In contrast, the SFT- based GRPO policy exhibits greater volatility, suggesting that the RL algorithm make more drastic and potentially erratic updates. In terms of format rewards, RI. based on SFT cold start is also weaker than RI. based on DPO cold start in terms of the stability of format rewards. Regarding the impact different cold-start training methods on the stability of model training. we believe this is related to the training objectives of the cold-start phase and the RL phase. The SFT training objective to maximize log likelihood, which a form of imitation learning, while the loss function of DPO can be directly optimizing an implicit reward model consistent with preference data, which is more aligned with the subsequent reward-driven GRPO optimization objective. Therefore, using DPO-based model starting point also brings more stable training for subsequent RL.
4.5 ANALYSIS OF THE RELATIONSHIP BETWEEN GF AND FINAL PERFORMANCE
We evaluate the correlation between the model's GF value during the cold-start phase and its final performance (represented here by the average score on MEGA-Bench, MMMU, By comparing three cold-start methods with different GF val-
SFT
DPO w. SFT
48.2
DPO
Final Performance (AVG]
SFT4ua
46.0
Trand (R) 0.864)
and Math Verse. Vision_Only
47.8
ues, presented in Figure 5 we can see that GF and the model's final performance are corre- lated to certain extent. This also confirms that the stronger the model's generalization ability in the cold-start phase. the more it will con- tribute to the model's improvement the RL phase.
DF0
47.6
47.4
47.2
SF7
47.0
70
75
80
85
90
95
Generalization Factor
In addition, we can also prove that the cold-start method based on preference data has higher generalization ability compared with the tradi- tional SFT cold-start paradigm, thus bringing greater potential for improvement to subsequent RI..
Figure 5: GF vs. Final Performance
9

---

Published as a conference paper at ICLR 2026
5 RELATED WORK
The application of RL has emerged as a highly effective method for enhancing the reasoning capabil- ities of large language models, with notable successes in the text-only domain such as DeepSeek-RI (Guo et al. 2025). which leverages RLVR (Lambert et al. [022 Guo et al. (025). Inspired by these advancements, a substantial and rapidly growing body of research has begun adapting RL techniques for VLMs. This has catalyzed wave of "MLLM-rl" studies, all aiming to harness similar princi- ples to unlock more advanced multimodal reasoning abilities. For instance, MM-Eureka (Meng et al. 2025) explores the enhancement of multimodal reasoning abilities through rule-based RI. by constructing high-quality multimodal reasoning datasets. VL--Rthinker (Wang et al.) (025a) stim- ulates the slow thinking and self- self-reflection abilities of VLMs through RL. Orsta (Ma et al. (025)
establishes a unified RL system that supports VLMs in jointly learning visual reasoning and per- ception tasks. VLM-R1 (Shen et al., 2025) extends RI RI-style RL to VLMs for visual understanding tasks to improve their visual reasoning abilities. LMM-R1 (Peng et al. 2025) enhances the model's basic reasoning ability and multimodal generalization ability through two-stage training strategy of basic reasoning enhancement and multimodal generalization training. RI-VL (Zhang et al. 2025b) realizes the self-improvement of MILMs' reasoning ability by solving the sparse reward problem through Step-wise GRPO. DeepEyes (Zheng et al. 2025) motivates the model's "Thinking with Images" ability through RL. PEhRLiZhang et al., 2025a) strengthens multimodal reasoning by explicitly anchoring it to verified visual evidence. BounalThiner-R1--11
(Zhou et al. 2025) per-
forms RL directly without any supervised fine-tuning of the model to reproduce the "aha moment".
A crucial precursor to effective RL is the "cold-start" phase, which initializes the model's policy before the RL stage begins. The conventional strategy for this phase is SFT, a foundational step adopted by many leading models to establish a strong baseline performance (Wei et al. 2025 Yang (ct al., 2025b; Huang et al.] 2025; Deng et al. 2025b). In parallel with refining cold-start methods, the prohibitive cost of human annotation has driven the field towards synthetic data generation. This approach often involves using powerful teacher models to distill vast amounts of data for training smaller student models (Zhang et al. [2025c] Xu et al. 2024 Huang al., 2025). Vision-R1 (Huang al. 2025) cold-starts the model before applying RL by synthesizing 100K high-quality long CoT instructions. VLaVA-Col (Xu al.) (024) integrates multiple mainstream visual question an- swering datasets and uses advanced large models to synthesize 99K valid maage-quettion-answer pairs. Yang et al. (2025a) explicitly decouple the two abilities of "abstract reasoning" and "strategy awareness" through two curriculum-style stages: SFT cold start and RL. R1-Onevision (Yang et al.) 2025b) adopts a two-stage training strategy of SFT RL. by synthesizing a 155K instruction set.
CONCLUSIONS
In this study, we introduced the Self- Belf-Distilled Preference- based Cold-Start framework, a novel three- stage methodology. By leveraging a self-distillation process to generate preference data, we decou- ple the learning of shallow objectives, such as output format, from the deep, logical reasoning skills targeted during the final RL phase. Our method utilizes DPO to pre-align the model, providing a superior initial policy for RL. The creative insight of decoupling learning objectives solves the practical problem of SFT-induced overfitting, which often constrains exploration and leads to sub- optimal performance. Our results demonstrate the practical value of this approach. The introduction of the Generalization Factor also provides valuable new metric for quantifying model generaliza- tion. This work shows considerable application prospects for developing more robust and capable multimodal reasoning systems.
Despite these promising results, this study has certain limitations that suggest avenues for future research. Our experiments were focused on the multimodal domain; further studies should be con- ducted to validate the efficacy of the SPECS framework in text-only reasoning tasks. The gener- alization of our findings could also be strengthened through more extensive testing across a more diverse set of out-of of-distribution benchmarks. Such investigations would continue to refine our un- derstanding of how to most effectively structure learning pipelines for complex Al systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is supported in part by the National Natural Science Foundation of China under Grant #72293575, and the Joint Research Project on the Integration of Culture, Science and Technology between Chinese Academy of Sciences and Hunan Province #2024JK4003.
ETHICS STATEMENT
All of the paper's authors have read and adhered to the ICLR Code of Ethics. This work focuses on advancing the reasoning capabilities of multimodal large language models through a novel training methodology. The core contributions are algorithmic, aimed at improving the efficiency, stability, and performance ceiling of reinforcement learning pipelines for such models.
The datasets used for training and evaluation, including Orsta47K, virI39K, MEGA--Bencl MMMU, Math MATH-Vision, and MathVerse, are publicly available datasets and benchmarks established within the academic community. Our use of these standard datasets intended to ensure transparency, facilitate reproducibility, and allow for direct comparison with prior work. We do not use any private or sensitive user data. The base model used, Qwen2.5-VL-7B. is an open-source model, promoting accessibility and further rescarch.
Data and Models:
External Evaluators: For evaluating subjective short-answer questions where automated rule-based metrics insufficient, we employed proprietary models (GPT-4o) external judges. We acknowl- edge that these models may have their own inherent biases. This approach was chosen to provide a consistent and scalable evaluation standard for complex, open-ended responses, common practice in current Al research. The specific prompts and evaluation criteria were designed to be as objective as possible mitigate these potential biases.
The goal of this research is to enhance the general reasoning abilities of AI systems. While this can lead to positive applications in fields like education, scientific research, and accessibility tools, we recognize that, like any powerful technology, it could potentially be misused. Our work does not introduce any new applications but rather improves the underlying training methodology. We encourage the responsible development and deployment of Al systems built upon these foundational research advancements.
Potential Societal Impact:
Our proposed framework, SPECS, is not designed for a specific downstream application and was evaluated on broad-domain academic benchmarks. We have not conducted an in-depth analysis of social or demographic biases, the datasets primarily consist of math, science, and general knowledge problems. We acknowledge that underlying base model and training data may contain biases, and future work should investigate how different training strategies impact the propagation or mitigation such biases.
Bias and Fairness:
REPRODUCIBILLIT STATRMENT
To ensure the reproducibility of our work, we provide a detailed account of our methodology and ex- perimental setup. The core SPECS framework, including three stages of self-distillation for data generation, DPO-based pre-alignment, and final GRPO fine-tuning, is described in Section 3 Our complete experimental settings, including the datasets, benchmarks, and baselines used for evalu- ation, detailed in Section 4.1 This section also specifies crucial implementation details, such as learning rates and batch sizes for all training stages We utilized publicly available frameworks, MM-EUREKA and LlamaFactory, for our implementation. Furthermore, the appendix offers ad- ditional resources to aid in reproduction, including the exact system prompts used for training and inference (Appendix B and a detailed analysis of the hybrid loss coefficient (Appendix C
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint a arXiv:2502. 13923, 2025.
11

---

Published as a conference paper at ICLR 2026
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference on machine learning. pp. 41 41-48, 2009.
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training rl-like reasoning large vision-language models. arXiv preprint arXiv:2504. 1468, 2025a.
Jiacheng Chen, Tianhao Liang, Sherman Siu, Zhengging Wang, Kai Wang, Yubo Wang, Yuansheng Ni, Wang Zhu, Ziyan Jiang, Bohan Lyu, Dongfu Jiang, Xuan He, Yuan Liu, Hexiang Hu, Xiang Yue, and Wenhu Chen. Mega-bench: Scaling multimodal evaluation to over 500 real-world tasks. In Interaational Conference on Learning Representations (ICLR), 2025b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14198- 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong. Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, r1 generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.1716.. 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprini arXiv:2507.0626/1 2025.
Jia Deng, Jie Chen, Zhipeng Chen, Daixuan Cheng. Fei Bai, Beichen Zhang. Yinqian Min, Yanzipeng Gao, Wayne Xin Zhao, and Ji- Ji-Rong Wen. From trial-and-eroor to improvement: A systematic analysis of Ilm exploration mechanisms in rlvr. arXiv preprint arXiv:2508.07534 2025a.
Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei wei Chang. Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement. arXiv preprint (SCI::25033173522 2025b.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.06744 2025.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training, arXiv preprint arXiv:2411.15122 2024.
Xiaohan Lan, Fanfan Liu, Haibo Qiu, Siqi Yang, Delian Ruan, Peng Shi, and Lin Ma. Metis-home: Hybrid optimized mixture-of-experts for multimodal reasoning. arXiv preprint arXiv:2510.20579 2025.
Haoyu Lu, Wen Liu. Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek- vl: Towards real-world vasion-language understanding, 2024a.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024b.
Yan Ma, Linge Du, Xuyang Shen, Shaoxiang Chen, Pengfei Li, Qibing Ren, Lizhuang Ma, Yuchao Dai, Pengfei Liu, and Junjie Yan. One rl to see them all: Visual triple unified reinforcement learning. arXiv preprint 18811 2025.
12

---

Published as a conference paper at ICLR 2026
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule- based reinforcement learning. arXiv preprint arXiv:2503.07366. 2025.
Yingzhe Peng. Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint MxXiv:2503305336 2025.
Haibo Qiu, Xiaohan Lan, Fanfan Liu, Xiaohu Sun, Delian Ruan, Peng Shi, and Lin Ma. Metis- rise: RI incentivizes and sft enhances multimodal reasoning model learning. arXiv preprint arXiv:2506. 13056, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53747 2023.
Jun Rao, Zepeng Lin, Xuebo Liu, Xiaopeng Ke, Lian Lian, Dong Jin, Shengjun Cheng, Jun Yu, and Min Zhang. Apt: Improving specialist I1m performance with weakness case acquisition and iterative preference training. arXiv preprint arXiv:2506.03444 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. VIm-rl: A stable and generalizable rl r1-style large vision-language model. arXiv preprint arXiv:2504.0760 2025.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, Congcong Wang, Dehao Zhang, Dikang Du Dongliang Wang. Enming Yuan, Enzhe Lu, Fang Li, Flood Sung, Guangda Wei, Guokun Lai, Han Zhu, Hao Ding, Hao Hu, Hao Yang. Hao Zhang. Haoning Wu, Haotian Yao, Haoyu Lu, Heng Wang, Hongcheng Gao, Huabin Zheng, Jiaming Li, Jianlin Su, Jianzhou Wang, Jiaqi Deng. Jiezhong Qiu, Jin Xie, Jinhong Wang, Jingyuan Liu, Junjie Yan, Kun Ouyang, Liang Chen, Lin Sui, Longhui Yu, Meng- fan Dong, Mengnan Dong, Nuo Xu, Pengyu Cheng, Qizheng Gu, Runjie Zhou, Shaowei Liu, Sihan Cao, Tao Yu, Tianhui Song, Tongtong Bai, Wei Song. Weiran He, Weixiao Huang, Weixin Xu, Xiaokun Yuan, Xingcheng Yao, Xingzhe Wu, Xinxing Zu, Xinyu Zhou, Xinyuan Wang, Y. Charles, Yan Zhong, Yang Li, Yangyang Hu, Yanru Chen, Yejie Wang, Yibo Liu, Yibo Miao, Yidao Qin, Yimin Chen, Yiping Bao, Yiqin Wang, Yongsheng Kang, Yuanxin Liu, Yulun Du, Yuxin WU, Yuzhi Wang, Yuzi Yan, Zaida Zhou, Zhaowei Li, Zhejun Jiang, Zheng Zhang, Zhilin Yang, Zhiqi Huang, Zihao Huang, Zijia Zhao and Ziwei Chen. Kimi-VL. technical report, 2025. URL httpss:/t/sxaiiii org/ abs 2as./250. 2500/55555000 07491
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. V1- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.0888. 08833777..08 2025a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirry-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL /tttps://openneee ew new wtt.te//fooromt//oooottttt
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-lann language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1219) 2024b.
Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung Ching Lin. Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning seff-improvement. arXiv preprint arXiv:2504.07934. 2025b.
13

---

Published as a conference paper at ICLR 2026
Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang. Advancing multimodal reasoning via reinforcement learning with cold start. arXiv preprint arXivv2505.22334 2025.
Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng. Xinyu Ye. Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508. 05080, 2025.
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. arXiv preprint 2xxx:2441110044 2024.
Siqi Yang, Zilve Gao, Haibo Qiu, Fanfan Liu, Peng Shi, Zhixiong Zeng, Qingmin Liao, and Lin Ma. Learning when to look: A disentangled curriculum for strategic perception in multimodal reasoning. arXiv preprint arXiv:2512.17227, 2025a.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin. Fengyun Rao, Minfeng Zhu, et al. RI-onevision: Advancing generalized multimodal rea- soning through cross-modal formalization. arXiv preprint vrXiv:2503.10611 2025b.
Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng. Li Shen, et al. Mulberry: Empowering milm with ol-like reasoning and reflection via collective monte carlo tree search. arXiv preprint urXiv:2412.18319, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint ar33v 14476, 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. Towards the law of capacity gap in distilling language models. arXiv preprint w 2023.
Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Zhixiong Zeng, Siqi Yang, Peng Shi, Lin Ma. and Jing Zhang. Perceptuul-evidence anchored reinforced learning for multimodal reasoning. arXiv preprint arXiv:2511.18437, 2025a.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503.11933: 2025b.
Letian Zhang, Quan Cui, Bingchen Zhao, and Cheng Yang. Oasis: One image all you need for multimodal instruction data synthesis. arXiv preprint arXiv:2503.0877 2025c.
Renrui Zhang. Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams visual math problems? arXiv preprint arXiv:2403.14622 2024.
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu. Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforeement learning. arXiv preprint arXiv:2505. /4362, 2025.
Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsieh. R1- zero's" aha moment" in visual reasoning on 2b non-sft model. arXiv preprint arXiv:2503.055. 05132,, 2025.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, we utilized the Large Language Model (LLLM Gemini-2.5-pro (Co- manici et al. 2025) to polishing the text. Its application was strietly limited to correcting spelling and grammatical errors. The authors manually reviewed and verified all Al-assisted modifications to ensure factual accuracy. The core ideas, methodologies, and figures presented are entirely the original work of the human authors.
B PROMPTS
The following are the System Prompts for the model during the cold start training phase and the RL training phase. In the instruction format generalization experiment in Section the Inference System Prompt for the ID Task is consistent with these.
System Prompt for Cold Start Training and ID Task Inference
Solve the question. The user asks question, and you solve it. first think about the reasoning process in the mind and then provide the user with the answer. The answer is in latex format and wrapped in $. The final answer must be wrapped using the \boxed[]
command. The reasoning process and </think> and <answer> </answer> tags, respectively, i.e., <think> since $1+1=2$, so the answer is $2$. </think> <answer> The answer is $ $\boxed(21$ </answer> which means assistant's output should start with <think> and end with </answer>.
enclosed within <think>
The following is the Inference System Prompt for the OOD Task in the instruction format general- ization experiment of Section 2
System Prompt for OOD Task Inference
Solve the question. The user asks question, and you solve it. You first think about the reasoning process in the mind and then provide the user with the $. The final answer must be wrapped using the \boxed[]
The answer is in Latex format and wrapped
command. The reasoning process and answer are enclosed within <cot> </cot> and <response> </response> tags, respectively, i.e., <cot> Since $1+1-2$, the answer is $2$. </cot> <response> The answer is (2boxed(21$ </response> which start with <cot> and end with </response>.
assistant's output should
The following Prompt is used for filtering chosen responses, see Section 3.2
System Prompt for Chosen Response Filtration
You are an export evaluator. Please analyze the following response and evaluate whether the final answer is consistent with the reasoning steps.
Question: (question)
Response to analyze: [chosen_answerl answer)
Please output only digit 1 if the answer is consistent with the reasoning, digit 0 it the reasoning has obviously theoretical errors OR the answer is inconsistent with the reasoning.
15

---

Published as a conference paper at ICLR 2026
C ANALYSIS OF LOss BALANCE COEFFICIENTS IN COLD-START TRAINING
C.I PRELIMINARIES
Supervised Fine-Tuning, SFT adapts pre-trained models by optimizing a cross-entropy loss func- tion to maximize the log-likelihood of a desired output for a given prompt æ. The loss is defined as:
(TF = (a..))-- |- log no (Uc[z))
By training exclusively on positive examples, SFT creates sharply peaked probability distribu- tion that closely mimics the training data. While this accelerates model convergence, it can limit generalization capabilities.
Direct Preference Optimization. DaOaaiii et al. 2023) refines models by learning directly from preference data, consisting of a prompt x, a chosen response Ye- and a rejected response yr- The DPO loss function is designed to increase the relative probability of the chosen response the rejected one:
(T(Uc)>)) Teer (yel
Ta ( (ar) Teat Urr (x:)
LDPO Feet) - -B log
(log
3 log
where wo is the policy being optimized, 're is reference policy, is the logistic function and temperature parameter. This approach maximizes the margin between chosen and rejected responses, cultivating smoother and more robust probability distribution that enhances the model's generalization performance.
C.2 ANALYSIS OF LOss BALANCE COEFFICIENTS
In Section 3.3
proposed that in our cold start, in addition the basic DPO loss, we also added the SFT loss to ensure that the model does not deviate too much from the chosen samples during the training process. The combined loss function thus:
Lhybrid LDPo + XLSFT
Among the Chybrid, the DPO loss function is designed to increase the relative probability of the chosen response over the rejected one:
Blog To <O(Me(z) (Te(())
3 log To To( [2) (Tr(()))))))))))
)] + (m. log mo (ye [x]]
Llybrid = - -B( log
We recorded the model's rewards for chosen, rewards for rejected, and margins during cold start training when À = 0, A = 0.5, and X = 1, as shown in Figure 6 below.
From the picture, chosen and the rejected, when = 0, this margin is the largest. However, this margin causes both the chosen and rejected rewards decrease (the rejected rewards decrease faster than the chosen rewards). When À = 0.5 and À = 1, it ensures that the chosen rewards increase during the DPO training process.
lot. Since the training loss of DPO can expand the margin between the
Chosen Rewards
Rejected Rewards
Reward Margins
1.5
Rejected Rewardds
Reward Margins
15
10.
-10
0.0
-15
-05
40
60
140
Training Step
Training Step
Training Step
Figure 6: Effect of À Chosen and Rejected Rewards and Reward Margins During Training
16

---

Published as a conference paper at ICLR 2026
Table 6: Average RBF by Sample Size
Sample Size
120
240
480
Model
base
1 1.7686
1.9166
1.9206
sft
1 .0066
1.8961
1.9336
ours
1.8216
1.9268
1.9596
D PERFORMANCE POTENTIAL OF DPO-BASED COLD-START VS. SFT
To evaluate the effectiveness of our proposed cold-start strategy, we first compare our DPO-based approach against conventional SFT baseline. For this comparison, the SFT model was fine-tuned exclusively on the "chosen" responses from our preference dataset, while our model was trained using the full preference pairs with the DPO algorithm.
Performance Potential via Pass @ K. We assessed the performance both the DPO and SFT cold-start models on the MMMU benchmark. As delineated in Figure 7 the DPO- based model demonstrates supe- rior performance across evaluated metrics, including @ass@8, and Pass @ 32. This indicates that the DPO cold-start method endows the model with greater ini- tial capabilities and higher potential for future align- ment tasks compared to the standard SFT approach.
90
MMMUge
86.88
87.33
MMMU@64
85
Score
80
76.88
77,55
74.39
75
70
66.89
Exploration Capability via Rollout Branching Factor (RBF). Beyond task performance, we inves- tigated the intrinsic exploratory capacity of models, critical factor for successful RL, We measure this using the RBF (Deng et al.) 2025a), which quantifies the diversity of a model's 's generation by counting the number of candidate tokens within the probability mass of top (0.95) during decoding. A higher RBF signifies greater generation diversity and, consequently, stronger capacity for exploration. As shown in Table [6] our DPO-base cold-start method yields a substantially higher RBF than the SFT baseline. This finding suggests that our approach cultivates broader exploration space, which highly beneficial for the subsequent RL phase, enabling the model to discover more diverse and potentially higher-quality solutions. This outcome highlights key advantage of our method in preparing models for alignment.
Model
Figure MMMU Pass @ Performance
E ANALYSIS OF CHOSEN RESPONSE FILTRATION
We believe it is crucial to ensure the quality and accuracy of chosen response in cold start parse. However, we found that there are a small number of cases (generated by GRPO-zero) in chosen response where the answers are not consistent with the reasoning, which could affect training, such as the following case:
Case Example
<think> To determine the total number dogs in the picture, we need to count each dog individually. From the image, we can see that there are dogs. </think>
<answer>Thh answer
3
</answer>
The reasons for this situation still need to be explored, and we think it may be related to reward hacking in GRPO training (because the answer to this training data was incorrectly labeled as 3, while the correct answer should be as in the reasoning process). To prevent these data from affecting the cold start training, we use another LLM to verify the data. This evaluation process is simple and objective, with significant emror, and will not introduce bias into data filtering. The
17

---

Published as a conference paper at ICLR 2026
table 7 below shows the differences in the final model performance between the data filtered by the advanced LLM and the randomly filtered data.
Table 7: Performance Comparison of Self-Distillation with and without Verification on Various Benchmarks
Model
Megabench
MMMU
Math Vista
Math Vision
Math Verse
AVG
Owen2.5-7B-Instee
35.07
54.20
63.70
25.40
38.20
43.31
w.o verification
38.72
55.11
73.50
26.90
46.44
48.13
verification
39.17
56.78
75.90
29.50
48.73
50.02
can be seen from the experimental results that the model after data filtering performs better than the final model obtained by randomly selecting data. On the other hand, regarding the selection of large models for screening, in addition to using some advanced large models, due to the simplicity of the task instructions, some open-source large models can also be used as alternatives.
F EXPERIMENTS IMPLEMENTATION DETAILS
All experiments were conducted on cluster of 32 NVIDIA H800 (80G) GPUs. We utilize the open-source Multimodal Large Language Model, Qwen2.55VL-77. (Bai et al. 2025), as our base model.
For the GRPO training in Stage 1 and Stage 3, we employ the MM-EUREKA rollout and training batch sizes are both set 128, with 8 rollouts generated per sample. The learning rate is configured to 1 x 10-0 Following the DAPO (Yu et al. 2025) methodology. we set the clipping thresholds to 0.2 (lower) and 0.28 (upper), and we do not apply a KL penalty. The maximum output length is restricted to 10, 240 tokens. This training phase spanned 400 steps and was completed approximately 12 hours.
framework. The
For the DPO training in Stage 2. well for the comparative SFT experiments, we leverage the LlamaFactory framework using a total dataset size of 9, 155 samples. In this configuration, the training batch size is set to 64, the learning rate is maintained at 1 10 and the hyperparameter for the hybrid loss function is set to 1. The maximum output length is extended to 16, 384 tokens. These models were trained for 140 steps, with the training process completing in under 30 minutes.
The prompt used during training is shown in Appendix B
We have counted the computing time for DPO and SFT training using 8 GPUs under the same training parameters (including steps, batch size, data length, max new tokens, etc.), as shown in the following table
Table 8: Comparison of training performance between SFT and DPO.
Training Method
Train Samples/s
Train Steps/s
Total FLOS
Run Time/s
SFT DPO
1.035
0.129
2.82 x 10
8848
0.979
0.123
6.22 10
9346
A
-0.056
-0.006
+3.4 x 10
+498
As can be seen from the table, in actual computation, the computational cost between SFT and DPO is not significantly different. For the computation of 9,000 pieces of data, the total computation time only increased by approximately 8.3 minutes.
18

---

Published as a conference paper at ICLR 2026
G QUALITATIVE EXAMPLES
QA Case Analysis: Case #001
Please answer the question requiring integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: Subtract all yellow matte blocks. Subtract all tiny brown cylinders. How many objects are left?
Question: Hint:
Ground Truth: 5
Qwen2.5-VL-7B response:
Ours-7B response
<think> To solve the problem, first identify all objects in the image:
Yellow matte cylinder
2. Green metallic cylinder (left)
Green metallic cylinder (front)
4. Silver metallic cylinder Tiny brown cylinder
6. Yellow matte block Red matte sphere Next, follow the instructions: Subtract all yellow matte blocks: Removing the yellow matte block leaves objects.
2. Subtract all tiny brown cylinders: Removing the tiny brown cylinder leaves objects.
Thus, the remaining objects are 5.
</think>
<answer> The answer is 5 </answer>
Qwen2.5-VL-7B response
To solve this, first identify all objects in the image: there 8 objects in total. The yellow matte block is one object, and the tiny brown cylinder is another. Subtracting these two objects leaves with 8 So, the number of objects left is 6.
6 objects.
19

---

Published as a conference paper at ICLR 2026
QA Case Analysis: Case #002
Web pages viatted
Person Number Web pages
Question: Hint:
Please answer the question requiring integer answer and provide the final value, e.g., 1, 2, 3, at the end. Question: Several
Bornno
Windton
people compared how many Web pages they had visited. What is the mean of the numbers?"
Parry
Devos
Ground Truth: 64
Qwen2.5-VL-7B response: 67.75
Canter
Ours- 7B response
<think> find the mean of the numbers, first sum all the numbers and then divide by the count of numbers. The numbers are: 84, 56, 70, 68, 60, 47, 75, 52. Sum of the numbers:
84 + 56 + 70 + 68 + 60 + 47 + 75 + 52
Calculating step by step:
84+ 56 140
140 70 210
210 + 68 278
278 + 60 338
338 47
385 75 460
460 52 512
The sum of the numbers is 512. Next, count the number of values: There are 8 values. The mean is the sum divided by the count:
Mean 512
64
So, the mean of the numbers is 64.
</think> <answer> The answer is 64 </answer>>
Qwen2.5-VL-7B response
To find the mean of the numbers, we need to follow these steps: Add up all the numbers in the list. 2. Count how many numbers there are. 3. Divide the sum by the count.
Let's start with step I: Add up all the numbers.
The numbers are: 84, 56. 70, 68, 60, 47, 75, 52. Sum 84 + 56 70 + 68 + 60 47 + 75 + 52 Sum 542
Mean 542/8
Mean = 67.75
So, the mean of the numbers is 67.75.
20

---

Published as a conference paper at ICLR 2026
QA Case Analysis: Case #003
Question: Hint: Please answer the question and provide the correct op- tion letter, e.g., A, B, C. D, at the end. Question: At a certain moment, there is a passenger ship at sea point P. and lighthouse A is measured the direction 30.0 north by cast of P. and is 50.0 nautical miles away. The passenger ship sails at the speed of 60.0 nautical mile/hour in the direction of 60.0 from north by west for hours to reach point B, then tan ABAP
Choices:
(A) 5 (B) 5 (C)
5 (D) 2v55
Ground Truth: (A)
Qwen2.5-VL--7B response: (B)
Ours-7B response
<think>>First need to determine the coordinates distances relevant to the prob- lem. The initial position of the passenger ship is at point P, and is 50 nautical miles away from lighthouse A in the direction 30.0 degrees north by east. The passenger ship then sails in the direction 60.0 degrees from north by west for hours at a speed of 60 nautical miles per hour, reaching point B. The distance PB can be calculated as follows:
PB = 60 x 3
40 nautical miles
The direction 60.0 degrees from north by west means that the angle LBPA 30.0 60.0 90.0%. Therefore, AABP P is right triangle with /BAP 30.0%. In a right triangle. the tangent of an angle is the ratio of the opposite side to the adjacent side. Here, tan LBAP BP Since AP = 50 nautical miles and PB = 40 nautical miles, we can use the Pythagorean theorem to verify:
AP
AB v AP BP V 502 402 2500 1600 44100 10v41 41
However, we only need the ratio
BP AP:
BP
40
4
tan LBAP
AP
50
5
Thus, the answer is
</tink>>
<answer>> answer is
</answer>
Qwen2.5-VL-7B response
To solve this problem, we need to determine the coordinates of points A and B rela- tive to point P. and then find the tangent of the angle /BAP. 1. Determine the coordinates of point A relative to point P:
2. Determine the coordinates of point B relative to point P:
3. Calculate the tangent of <BAP:
Simplifying the expression, we get 025 1250133-0--1 - 1500
-255v33 1000
1000031-1000-0 1100333-5500065344-00 625v33
2000
925./3 1000
-25 1000
1000
2000
2000
2000
5'
Therefore, the correct answer is
21


Reviews:
[{"review_id": "4vY4vWw7kd", "reviewer": "Reviewer_T4Pk", "summary": "This paper introduces SPECS, a Self-Distilled Preference-based Cold-Start framework that rethinks initialization for reinforcement-learning-based multimodal large language models (MLLMs). The authors demonstrate that standard supervised fine-tuning (SFT) couples reasoning content and output format, resulting in overfitting and poor out-of-distribution (OOD) generalization.\nSPECS decouples learning objectives through three stages:\n1. Self-Distillation – Generates preference pairs focused on output format without large teacher models.\n2. DPO-based Pre-Alignment – Applies Direct Preference Optimization with a hybrid (DPO + SFT) loss for stable format learning.\n3. GRPO Fine-Tuning – Performs final reinforcement learning with verifiable rewards for deep reasoning.\nA new Generalization Factor (GF) metric measures cold-start generalization and correlates strongly (R² ≈ 0.86) with final RL performance.\nAcross MEGA-Bench, MMMU, MathVista, MathVision, and MathVerse, SPECS achieves consistent gains over strong baselines (e.g., VL-Rethinker, MM-Eureka), improving +4.1 % on MEGA-Bench and +12.2 % on MathVista. Ablations confirm that self-distillation and decoupled data outperform teacher-based and coupled variants.", "questions": "1. How does SPECS empirically or conceptually differ from Vision-R1 and AdaViP, which also integrate RL and preference-based training?\n2. Can the proposed GF metric predict RL success across different architectures (InternVL, Kimi-VL)?\n3. What is the size and diversity of the self-distilled preference dataset, and how does scaling influence results?\n4. How were biases in GPT-4o/Gemini evaluations mitigated (e.g., standardized prompts or aggregation)?\n5. Could SPECS generalize to text-only or perception-heavy domains (e.g., R1-VL, Chain-of-Focus models)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles a pressing issue in multimodal RL pipelines—how to design an effective cold-start phase for MLLM-r1 models—complementing recent efforts such as Vision-R1 and Advancing Multimodal Reasoning via RL with Cold Start.\n- The self-distilled preference data strategy avoids dependence on large teacher models while maintaining strong alignment quality. The decoupled DPO training elegantly separates surface-form learning from reasoning acquisition.\n- The proposed Generalization Factor (GF) offers a measurable and interpretable indicator of generalization potential, empirically shown to correlate with final RL performance.\n-  Evaluations across multiple reasoning benchmarks, together with well-designed ablations (self-distillation vs. teacher-distillation; decoupled vs. coupled data), substantiate the paper’s claims.\n- The writing is structured and transparent, with detailed hyperparameters, dataset composition, and implementation frameworks (MM-EUREKA, LlamaFactory) that support reproducibility.", "weaknesses": "- The related-work section omits recent concurrent methods such as Vision-R1 (Huang et al., 2025), AdaViP (Lu et al., 2025), Chain-of-Focus (2025), and DeepEyes (2025), which would situate SPECS more clearly within the multimodal RL ecosystem.\n- Current evaluations emphasize mathematical reasoning; inclusion of more general-domain benchmarks (e.g., ScienceQA, MMBench) would better demonstrate scalability.\n- The GF analysis is confined to one model family (QwenVL). Extending the metric across other backbones (e.g., InternVL, Kimi-VL) would confirm its generality.\n-  Dependence on GPT-4o and Gemini-2.5 for preference filtration introduces potential bias; multi-judge or calibration procedures could strengthen reliability.\n- While the framework improves RL convergence, the additional computational cost of the DPO stage is not explicitly quantified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DPDswsWNMF", "reviewer": "Reviewer_bc3X", "summary": "This paper proposes SPECS, a cold start strategy for preference-based multimodal learning. They first introduce a generalization factor metric to evaluate models' generation capacity. They then design a 3-stage learning framework: 1) generate format-aware preference data via self-distillation; 2) DPO cold start for learning the format consistency; 3) GRPO to improve deep reasoning.", "questions": "- For the rejected samples, why are rollouts taken from the base model instead of the exploratory GRPO-zero model?\n- Could the authors clarify the mathematical definition and interpretation of GF?\n- Why is the zero-model necessary for self-distillation? Given the strong LLM-as-a-Judge, can we directly sample from base model rollout?\n- Since you use both SFT and DPO loss for cold start training, how is the format decoupled?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Experiments demonstrate the performance gain of the proposed method against selected base models and vanilla GRPO on various benchmarks.\n- The paper is overall clearly written and easy to follow.\n- Ablation study is conducted to validate the performance of decoupled (format) DPO data.", "weaknesses": "- The paper is not technically sound. It largely reaffirms the well-established observation that RL-based methods improve generalization compared to supervised finetuning,  but does not offer significant new technical insight. \n- The introduction of the zero model (trained with GRPO before self-distillation) appears unnecessary. It is unclear why the chosen and rejected samples cannot be directly generated from the base model, especially since both are later used for format-level preference training.\n- The definition of Generalization Factor and lack clarity theoretical justification. The formula in L131–133 appears confusing and unsimplified, possibly due to a typo.\n- The proposed method is costly. The cold start stage requires GRPO training on 86k data, as well as gemini-2.5-flash as strong LLM-as-a-judge.\n- The effectiveness of DPO cold start is not directly validated; it would be helpful to compare with a control baseline that performs SFT cold start + RL using the same chosen data to isolate the benefit of preference-based training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKFKW2ib5P", "reviewer": "Reviewer_hLrc", "summary": "This paper proposes SPECS, a framework for training multimodal models using DPO and RL. The framework involves first generating preference data from the model, training it on the preference data with DPO, and then performing GRPO training. The authors show that using DPO is preferable to SFT for training an initial RL checkpoint due to its superior generalization, and yields more stable RL training. Their final model outperforms solid baselines or training using only RL.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Model outperforms prior work and the additional DPO is clearly helpful.\n- Ablations show the distillation strategy is quite effective\n- Showing DPO generalizes better than SFT with the generalization gap metric is interesting.", "weaknesses": "- I think the work needs more careful ablations, particularly around the self-distillation.\n  - Gemini-flash is used as the LM judge when choosing responses. How does this compare to just distilling from Gemini? Is using it as a judge cheaper?\n  - The authors do some additional RL to create a model to distil from and claim “[it] is more adept at exploring the solution space”. What is the justification for this? Does distilling directly from the base model (with some careful prompting) perform worse? A stronger justification here would be nice.\n  - In table 3, you claim that decoupled outperforms coupled data for cold-starting, but it appears it underperforms the coupled data (slightly) pre-RL training, but does indeed outperform post-RL training. It would be useful to explain why this is! Similarly, considering that the average difference is only around 1.5 points, it would be useful to know what the average noise on the dataset is (i.e., some sort of statistical significance measure on the results - is 1.5 within noise or not?).\n- The generalization gap metric is interesting, but its connection with better cold start checkpoints seems tenuous. If we do RL training on only in-domain prompts, does it matter that one method has better OOD performance? A more concrete comparison where you train an SFT model with RL using the same setup as the DPO+GRPO model would be more convincing. The authors have clearly trained an SFT+RL model (used for analysis in Section 4.5), so hopefully it would just be a matter of evaluating that.\n- There is some interesting analysis in Appendix A.4 that suggests the DPO better improves pass @ k, but as far as I can see this is never mentioned in the main text (not even a reference to that appendix section itself!)\n- The authors note that the DPO-trained model is more stable during RL training, but don’t provide an explanation as to why. It’d be useful to see some sort of analysis or justification for this - perhaps it suggests the hyperparameters for RL training on the SFT model are suboptimal?\n- The authors do not give enough details on their experimentation. How many training steps / epochs was training run for for DPO/SFT/RL? For RL, what KL penalty was used, and how long were responses allowed to be? It would also be useful to get some indication of how long SFT/DPO/RL training took in GPU-hours.\n\nOverall, I think the work is reasonable, and the finding around using DPO instead of SFT for cold start is interesting. But more justification around decisions are required, and some clearer ablations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YpcXPE4rWd", "reviewer": "Reviewer_8EWE", "summary": "Currently, with the fast development of RL, its application has expanded to MLLM. Most representative paradigms begin with a cold start under supervised fine-tuning (SFT). A challenge here is SFT cold start may induce instruction-style overfitting, weaken out-of-distribution generalization, and ultimately affect downstream RL. In this view, the authors first introduce the Generalization Factor (GF) coefficient to quantify the generalization capability under different methods, showing that preference-based training can do better than SFT. This further brings a Self-distilled, Preference-based Cold Start framework (SPECS) in preference-based training style.", "questions": "The paper has the finding that stage 1 rl can benefit following rl training. The experiments are sound and the discussions are solid. Overall, it is a good paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this paper is sound, and the observation on SFT cold start is fundamental. The GF criteria, though without significant novelty, is straightforward and useful.\n\n2. After raising the problem, the authors propose an intuitive and effective approach to solving the problem. \n\n3. The paper is easy to follow.", "weaknesses": "1. The self-distilled dataset needs further discussion. In Sec. 3.2, the rejected responses are responses that also contain the correct answer, but deviate from the required format. In stage 1 training, is format deviation the other problem we need to consider? What about wrong answers, wrong thinking process, etc? From my perspective, the current design might face the challenge of hard negative mining. More discussions are needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)