Diving into Self-Evolving Training for Multimodal Reasoning
Authors: Wei Liu
Junlong Li
Xiwen Zhang
Fan Zhou
Yu Cheng
Junxian He
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Reasoning
Self-Evolving Training
Large Multimodal Models
Primary area: deep_learning->large_language_models

Abstract: Self-evolving training—where models iteratively learn from their own outputs—has emerged as a key approach for complex reasoning tasks, addressing the scarcity of high-quality chain-of-thought data. However, its effectiveness in multimodal reasoning, a domain more intricate than text-only reasoning, remains underexplored, and the understanding of critical factors in this training paradigm remains limited. Furthermore, a central challenge for this training method is performance saturation, which impedes further improvements and scalability. Inspired by reinforcement learning (RL), in this paper, we reframe self-evolving training for multimodal reasoning through the lens of RL, identifying three pivotal factors: $\textit{Training Method}$, $\textit{Reward Model}$, and $\textit{Prompt Variation}$. Through systematic analysis, we establish relatively optimal design principles that significantly enhance multimodal reasoning capabilities. Moreover, delving deeper into training dynamics, we uncover the roots of saturation and propose a new automatic balancing mechanism to mitigate this limitation. Building on these insights, we propose M-STaR (**M**ultimodal **S**elf-evolving **T**r**a**ining for **R**easoning), a framework that achieves consistent performance gains across models of varying sizes and diverse benchmarks. All resources will be made publicly available.

Diving into Self-Evolving Training for Multimodal Reasoning
Wei Liu 11 Junlong 1i
Xiwen Zhang?
Fan Zhou? Yu Cheng Junxian He
Abstract
autonomous driving (Yang et al., 2023). It requires Large Multimodal Models (LMMs) to understand various modali- ties beyond text. For example, visual mathematical reason- ing (Lu et al., 2023) challenges models to analyze complex figures, diagrams, and charts, leveraging the provided infor- mation to perform reasoning tasks.
models iteratively learn from their own outputs-haa emerged as a key approach for complex reasoning tasks, ad- dressing the scarcity of high-quality chain-of- thought data. However, its effectiveness in mul- timodal reasoning. domain more intricate than text-only reasoning, remains underexplored, and the understanding of critical factors in this train- ing paradigm remains limited. Furthermore, a central challenge for this training method is per- formance saturation, which impedes further im- provements and scalability, Inspired by reinforce- ment learning (RL), in this paper, we reframe self-evolving training for multimodal reasoning through the lens of RL, identifying three pivotal factors: Training Method, Reward Model, and Prompt Variation. Through systematic analysis, we establish relatively optimal design principles that significantly enhance multimodal reasoning capabilities. Moreover, delving deeper into train- ing dynamics, uncover the roots of saturation and propose new automatic balancing mech- anism to mitigate this limitation. Building on these insights, we propose M-STAR (Multimodal Self-evolving Training for Reasoning), a frame- work that achieves consistent performance gains across models of varying sizes and diverse bench- marks. All resources are made publicly available at https:://// http://msttarmmmmeeeeee htt.tiiimm..n............ github. io.
Self-evolving
Despite the critical role of mutlimodal reasoning. the avail- ability of human-annotated thought processes in multimodal scenarios remains limited, hindering the learning of mul- timodal reasoning. Consequently, self-evolving training, which utilizes model's own generation ability to iteratively tune and improve itself without external annotated data, has emerged as an appealing candidate to facilitate reasoning abilities. While research on self-evolving training has pri- marily focused on the text-only settings (Hosseini al., 2024; Sun et al., 2024; Shao et al., 2024), its application in the multimodal domain, especially for reasoning tasks, has been limited with only a few sporadic examples (Fang et al., 2024; Dubey et al., 2024; Deng et al., 2024), and a unified framework has yet to be established.
Inspired by reinforcement learning (RL), in this paper, we reframe self-evolving training through the lens of RL, identi- fying three factors that are critical inside self-evolving train- ing: the training method, the use of reward model, and prompt variation. Through massive controlled studies, we (1) propose a continuous self-evolving training scheme to reduce the gap towards full online learning and outperforms other iterative baselines ($3.2); (2) train the first multimodal, process-based reward model for multimodal reasoning and demonstrate its usefulness in further enhancing performance ($3.3); and (3) find that adding more unlabeled queries helps only when having perfect reward signals (e.g., the oracle groundtruth answers), and it hurts the performance if the re- ward model does not generalize well on unseen data ($3.4), Beyond static design principles, we investigate the train- ing dynamics, revealing how performance saturation stems from diminishing exploration potential during training. To address this, we introduce a new metric that bridges explo- ration and exploitation, and propose an automatic balancing mechanism that dynamically adjusts the sampling tempera- trade-offs.
1. Introduction
Multimodal reasoning is a fundamental skill in many real- world applications, such as intelligent agents (Liu et al., 2024c), robotics (Li et al., 2023; Liu et al., 2024b), and
Equual contribution 'The Hong Kong University of Science and Technology "Shanghai Jiao Tong University 'Helixon Research "The Chinese University of Hong Kong. Correspondence to: Wei Liu, Junxian He <wliuen, junxianh Mensinnh@sse. @eech ust..ss.tttk........... Junlong Li <lockonlvange@@ gomi..com>.
ture to sustain ppploration-exploitaaai
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Combining all the recipes concluded through separate, con- trolled studies, we propose our self-evolving training al-

---

Diving into Self-Evolving Training for Multimodal Reasoning
Data
Generate
G
D
D
7t
R
ABCD Inscribed Circle center is O. Find the angle of ZAMK, Return numeric value.
What the value of the smallest individual bar in the whole chart? Answer
Exploitation Exploration
Improve
Rewarding
I
R
D
Unlabeled Prompt
Labeled Prompt
Figure I: Overview of our self-evolving training framework for multimodal reasoning. We investigate the three essential design components of it, namely Training method (T), Reward model (R), and Prompt variation (P). Orthogonal the static factors, the Dynamics of self-evoloution is also monitered, and provides control signals the training process.
gorithm named as M-STAR (Multimodal Self-evolving Training for Reasoning). Our experimental results on dif- ferent multimodal reasoning benchmarks, including Math- Vista, M3CoT, MMStar, MMBench and AI2D, show that this strategy, which incorporates both optimized static de- sign choices and dynamic adjustments, effectively mitigates exploration loss during training and enhances performance universally for models with varied sizes such as MiniCPM- V-2.5 (8B), Phi-3.5- -iiiio--Visioon (4B) and VL2 (2B).
resembles Rejection Fine-Tuning (RFT, Yuan et al. (2023)) as filters out negative responses in a hard manner. Both steps are performed iteratively to strike a tradeoff between offline and online training. In many tasks such mathe- matical problem-solving. there exists a unique, ground-truth answer a* which is utilized in the reward function, for ex- ample, Singh et al. (2023) directly adopts exact match to compute a binary reward by comparing ú and a". In such an iterative training procedure, the objective at iteration t is to obtain an improved policy model m/ (+1,
2. Overview of Self-Evolving Training for Multimodal Reasoning
(+1 argmax mh Σ D. Ri(a" 9:)), FD
Self-evolving training can be modeled as a general frame- work of reinforcement learning, where various algorithms can be formulated as specific instantiation of RL, such as PPO (Schulman et al., 2017), STaR (Zelikman al., 2022), ReST (Gulcehre et al., 2023) and ReSTEM (Singh et al., 2023). Specifically, given a reward function R. the objective of self-evolving training is to train the policy model ma to maximize expectation of reward R:
(2)
where the ground-truth answer a" can be empty, for example, when dealing with unlabeled inputs, the reward model must be able to score ÿs independently.
The Design Spaces Through the lens of Eq. 2, can identify three dominant factors that influence the training process, (1) training method: the training algorithms to perform this iterative process vary as well. For example, while Gulcehre et al. (2023); Xu et al. (2024b) initialize the model from the last checkpoint at each iteration, Zelik- man al. (2022); Singh et al. (2023) argue that initializing from the beginning checkpoint reduces overfitting and gives better performance empirically. (2) reward model: the de- sign of reward function R. (3) prompt variation: whether to incorporate additional unlabeled inputs without a* into training. Next, we investigate these three design spaces, aiming to summarize the best practices for each factor.
Tg = argmax, Σ D. R((.),, (1)
where x, represent the query and image in the given train- ing data D. while ÿ is response sampled from the current policy model wy. This standard RL objective, however, can be unstable to optimize and difficult scale up, thus a popu- lar algorithm adopted by recent works is to decouple the re- sponse rollout 0; Te|-|xr, of and policy improvement into separate offline stages (Gulcehre al., 2023; Singh al., 2023): (1) Generate: the current policy model generates new responses i4 ~ T |a, o); and (2) Improve: using the rewards to selects certain responses from the Generate step, which are then used to train the policy model with a standard supervised fine-tuning (SFT) loss. This way, the algorithm
3. Diving into Self-Evolving Design Components
In this section, we explore the three key components of self- evolving training, examining various strategies within each.
2

---

Diving into Self-Evolving Training for Multimodal Reasoning
We begin by outlining the general setup ($3.1), followed by comprehensive analysis of each component identify the best practices for multimodal self-evolution ($3.2-$3.4).
interval is small, the checkpoint at each iteration is initial- ized from one from the last iteration, and the optimizer as well as the learning rate scheduler is inherited between iterations, then iterative training becomes an online RL algo- rithm. Therefore, we propose Continuous Self-Evolving, a new iterative self-evolving training variant that represents a smoother interpolation between iterative training and online training. In continuous selffevolving training, we inherit the optimizers well the learning rate schedulers from the last iteration besides inheriting the model checkpoint, so that the optimization is continuous and closer to purely online learning algorithms. This way, we only have a global optimizer and learning rate scheduler essentially across the entire iterative training process. We also analyze the iter- ation interval effect in continuous self-evolving, which is defined as the training queries passed for one iteration we specifically study the effect of having shorter iteration interval, which stands in contrast to the common practice that adopts a long iteration interval to process all the data queries for one iteration.
3.1. General Setup
Models We base our main exploration on V--2.5 (8B) (Yao et al., 2024), and we also validate the final design choice for each component on two extra models with dif- ferent sizes: Phi-3.5- (4B) (Abdin et al., 2024) and InternVL-2 (2B) (Chen et al., 2024c). The details of these models can be found in Appendix A. To make the analysis process easier to understand, we mainly present the results of MiniCPM-V-2..5 in this section, while we include the results of the other models in $4.2.
Datasets We utilize V3OK (Shi et al., 2024), a high- quality and diverse multimodal reasoning dataset as our seed training dataset. Specifically, we downsample half of the examples (180K) from it to serve as our labeled training set, while setting aside the remaining half as a unlabeled training set by not using the answers in it. For evaluation, we split 750 samples from the unlabeled part of M6tK as the in-domain (ID) testset. For our out- of-domain (OOD) testset we use the testmini split of Math Vista (Lu et al., 2023), a comprehensive benchmark encompassing a wide range of multimodal reasoning tasks, including visual question answering, figure-based question answering, science question answering, and more. We also keep an non-overlapping 250 samples from V36K as the global validation set in training.
Setup We perform controlled experiments to study the ef- feet of different training methods, thus in this experiment we use the labeled dataset only and simply adopt the binary exact-match reward between ground-truth answer at and the generated answer. We compare with the most common iterative self-evolving algorithms ReSTEM (Singh et al., 2023) and iterative RFT, which are specific instantiations of our training methods design space. To study the effect of it- eration interval in the proposed continuous self-evolving, we experiment with different percentage of all the queries per iteration, varying from [6.25% 12.5%, 25%, 50%, 100%].
Main Training Settings Before self-evolving training, a warmuu stage (Appendix B) is applied to enhance the CoT generation ability of all models as many LLMs tend to output final answers directly without COT. We adopt most of the training settings from Yao et al. (2024) (see Appendix C), using constant learning rate of 1E - 6 and training for 10K steps across all experiments. During all rollout phases in training, we sample 16 responses per query and set the sam- pling temperature to 1.0. Unless explicitly stated otherwise, we follow existing practices (Singh et al., 2023; Zelikman et al., 2022) and only use the labeled training data.
Results Table 1 presents the experimental results of various training methods. Overall, initializing training from the last policy model checkpoint TĪ and maintaining a contin- uous optimization process contribute most significantly to the effectiveness of self-evolving training, particularly on Math Vista. Continuous self-evolving achieves the best per- formance both on the in-domain MathV360X test set, with 43.1 13.1% 1%, and on the OOD test set, Math Vista, with 57.2%. We also see the importance of maintaining proper interval to traverse the data queries. With a large interval, the training method becomes closer to an offline one, and the model can- not get timely updates on data matching its current output distribution. On the other hand, switching over the Improve and Generate steps too frequently makes the learning pro- cess unstable, leading to a lower score, especially on the in-domain test set. The strategy of continuous self-evolving with proper intervals also works for other smaller models, as shown in Table compared with representative base- lines, indicating its effectiveness and generalizability across different model sizes.
3.2. Training Methods
As described in $2, there are multiple variants on how we would train to update the policy model. Previous works mainly vary the model initialization factor, where at the "Improve" step, the model can be initialized from either the last checkpoint (Xu et al., 2024b; Pang et al., 2024) or the beginning checkpoint before the first iteration (Zelikman et al., 2022; Singh et al., 2023). Besides model initializa- tion, in this work, we introduce new variants of iterative self-evolving through delving into the gap between itera- tive training and online RI concretely, when the iteration
3

---

Diving into Self-Evolving Training for Multimodal Reasoning
Table I: Accuracy results (%) of self-evolving training using various training methods and iteration intervals. Interval (#) stands for iteration interval, the ratio of data we traverse in one iteration, and we also record the number of correspond- ing queries. M represents the policy model from which training is initialized in each iteration. O denotes whether the optimization process is continuous, i.e., the optimizer states and Ir scheduler are inherited from the last checkpoint. Please refer to Table 6 to check the full results on all sub- tasks of Math
Setup We conduct controlled experiments to assess the im- pact of incorporating the Process Reward Model (PRM) into self-evolving training and explore how best to utilize it. No- tably. before applying PRM, responses are pre-filtered based on their final answers to ensure consistency and quality dur- ing training. To train our PRM, we use Monte Carlo rollouts starting from prefixes with partial reasoning steps (Wang et al., 2024) to generate the training data. Specifically, we sample 16 responses per question and complete each step 8 times to obtain step-level annotations and more details can be found in Appendix D. We evaluate two different H opera- tions: (1) Top-K: Pick the top- K correct responses according to their reward scores, and (2) Filtering by a Threshold at Filtering out sampled responses with lower aggregated re- wards than a. The optimal value of is set 0.2 by grid searching on the validation set. Additionally, we investigate how varying the value of K in Top-K affects training, it represents a trade-off between the quality and diversity of the samples. According to $3.2, we fix training methods as continuous self-evolving with 45k interval and set continu- ous self-evolving, with or without randomly selected correct responses as our baselines.
Method
M O Interval (%) Math V360K Math Vista
V-2.5
13.6
52.4
+warmup
38.8
52.6
SFT
44.3
54.8
Iterative RFT
To X
100%
42.3
55.7
Rest RM
B x
100%
42.3
55.1
100%
42.2
56.7
50%
43.1
56.2
Continous Self-Evolving
T/ V
25%
43.1
57.2
12.5%
42.3
56.1
6.25%
41.0
56.8
Results Table 2 presents the results of integrating the PRM into self-evolving training, along with the impact of differ- ent H choices. Continuous Self Exolvvngg with PRM using Top-2 achieves the best performance in both the ID and OOD tests, with scores of 45.3% and 59.2%, respectively. Compared to training without PRM, most instances of self- evolving training with PRM show improved performance, especially in the OOD test. Interestingly, randomly select- ing a subset of correct responses actually leads to worse performance than continuous self-evolving, suggesting that even correct answers can be noisy. Random selection may increase the proportion of these noisy samples, undermining the effectiveness of self-evolving training.
3.3. Reward Models
In self-evolving training, the most common approach to re- ward function design uses a binary reward Rg;)) = (@; =
a"), where à; is the predicted answer inside ÿ and incorrect responses are filtered out to maximize rewards. While effec- tive, this sparse binary reward has limitations. It overlooks the quality of the intermediate reasoning steps within re- sponse. Additionally, reward models trained from equal or higher capacity models than the policy model (Fried et al., 2022; Wang et al., 2024; Sun et al., 2024) can provide richer signals to improve the policy model's learning.
In this section, we introduce a Process Reward Model (PRM) (Lightman et al., 2023; Wang et al., 2024) for multimodal reasoning-thh first of its kind, our knowledge and explore how integrating PRM can enhance reward design and whether it can improve policy model learning in self- evolving training for multimodal reasoning. To incorporate the reward scores into the objective of self-evolving training, the reward function is reformulated as:
In terms of leveraging PRM, we found that using Top-K to select the fixed number of best K responses with high- quality intermediate steps outperforms threshold-based fil- tering. The results also highlight the importance of balanc- ing the quality and diversity of sampled responses. Selecting K = 2 strikes this balance well, ensuring both response di- versity and high-quality reasoning steps for each question. Similar to the results in $3.2, we also see improvement when involving PRM on smaller models in Table 6, Appendix H.
R())) H(1(a" a.) (0.)) (G.) =
(3)
f(;"))
(4)
What makes PRM work for self-evolving training? To pursue deeper insights into the role of PRM in self-evolving training, we conduct an analysis presented in Figure 2. Based on the results from $3.3, we explore PRM's impact from two key perspectives: (1) Can PRM help the model to select out correct responses among multiple rollouts? (2) How different are the Top 2 and the rest correct solutions re-ranked by reward scores? We use the first checkpoint
Here, H is an operation that processes responses based on the final reward scores, where we ensure all responses are correct by matching the ground truths, and (ix) repre- sents the process reward score for each sampled response. The function (s+t) denotes the reward score at each inter- mediate step. Following Lightman et al. (2023), we use the min operation to aggregate stepwise rewards.
4

---

Diving into Self-Evolving Training for Multimodal Reasoning
Table 2:
The results of self-evolving training with PRM and different strategies to leverage reward scores. H is the method to further pick out high-quality responses from the correct rollouts: (1) Top-k is to select K correct responses with highest rewards, and (2) > or is to pick out the correct responses with rewards larger than a. Please refer to Table 6 to check the full results on all sub-tasks of Math Vista.
Bea
Top 2
Accuracy (%)
44
Mejotiy Voling
Weighted Vating
0.50
Rest
Greedy
Portion
42
0.30
40
0.20
0.10
38
0.00
Method
H
PRM Math MathVista
Rollout (K)
2
Relutivity Score
Cont.
Self-Evolving
x
43.1
57.2
(a)
(b)
+ Random
Random-2 x
41.0
55.5
Figure 2: (a): Accuracy on the val. set of greedy decoding and three selection strategy across different numbers of rollouts; (b): Average relativity score annotated by GPT4-o
> a
43.8
57.5
+PRM-based
Top-1
43.0
59.0
Selection
Top-2
45.3
59.2
of Top 2 and the rest responses re-ranked by rewards, we only calculate on correct ones.
Top-4
44.0
58.4
after warmup T as policy model to sample 16 responses for each question in the validation set with temperature-1.0 and reveal the behaviors of PRM in these samples.
the validation set, it tends to either retain or filter out all re- sponses for each query, which reduces diversity and makes the learning process more challenging. This further sup- ports the conclusion that our PRM performs better Reranker than Verifier.
We evaluate the verification ability of our PRM using two metrics, Best-of-N (BoN) and weighted voting (Sun et al., 2024), which are commonly employed to assess the perfor- mance of reward models. Surprisingly, as shown in Fig- ure 2a, our PRM underperforms in both metrics. Notably, BoN and weighted voting yield worse results than vanilla majority voting when N < 16. We speculate that this is due to the lack of high-quality step-level annotations compared to text-only reasoning tasks. These findings suggest that our PRM is not an effective verifier.
3.4. Prompt Variation
In this section, we explore how prompt variation affects self- evolving training. There are two primary types of prompts: labeled prompts and unlabeled prompts. Labeled prompts come with annotated ground truth answers, which can be used to filter out incorrect responses during training. In contrast, utilizing unlabeled prompts in self-evolving train- ing is more challenging due to the absence of ground truth annotations. To maintain the quality of unlabeled prompts in training, surrogates like reward scores pseudo labels must be employed. Meanwhile, unlike labeled prompts, unlabeled prompts are not be trained in SFT period, which increases the difficulty of learning for policy models.
To understand why our PRM can still significantly con- tributes to self-evolving training despite its weaker verifica- tion abilities, we analyzed the distribution of other metrics for the top-2 selected responses compared to other correct responses. We approached this from two perspectives: the average number of reasoning steps, and how much a re- sponse is directly relevant to the question (see Appendix E), since we do not find incorrect steps but find some irrelevant steps after randomly checking some examples. We found the responses re-ranked by our PRM generally have fewer rea- soning steps (Figure 6 in Appendix F) and more relevant to the query (Figure 2b). This highlights the precision of our PRM in recognizing genuinely high-quality responses. Therefore, our PRM acts as an effective reranker to iden- tify top-quality responses. This is especially critical when responses are already filtered by ground-trutt answers, and the ability to accurately assess the quality of reasoning steps beyond accuracy becomes vital.
Skylines: Unlabeled Prompts with Oracle Reward Sig- nals The coupling of these additional factors introduces complexity, making the effective use of unlabeled prompts less predictable. Therefore start by establishing base- line with "skyline" experiments, where both the unlabeled prompts and their ground truth answers are available but not used during the SFT phase. These unlabeled prompts with oracle reward signals serve as an intermediate difficulty between fully unlabeled and labeled prompts, providing insight into the challenges of training with unlabeled data.
Unlabeled Prompts We incorporate unlabeled prompts into self-evolving training. To ensure the quality of sampled responses for these prompts, we use weighted voting to en- semble the predictions from different responses, treating the ensembled prediction as a pseudo label à. This pseudo label then used to filter out responses with conflicting
In addition to the aforementioned analysis, we also investi- gate why leveraging a to filter responses with lower reward scores performs worse than T. The results indicate that, even with the optimal threshold value determined from
5

---

Diving into Self-Evolving Training for Multimodal Reasoning
Table 3: Results of involving unlabeled data. Tiixin denotes when to mixin the unlabeled data. The use of PRM follows $3.3, except we first get a pesudo "ground truth" through weighted voting on unlabeled prompts.
60
Test Set Greedy Ace
55
60
Math V 360K
aal Set PPasSOO
50
Vista Vista
55
Oracle
PRM
Tmixin
V360K
Math Vista
Passo K
45
43.1
57.2
50
45.3
59.2
X
0%
42.5
58.2
40
45
V
0%
42.9
59.1
2000
4000
6000
8000
10000
# Training Steps
X
V
0%
43.3
58.2
V
25%
42.4
57.6
Figure 3: Opposite trend of Greedy Accuracy and Pass@ K.
X
V
50%
42.9
58.2
X
V
75%
45.0
58.8
self-cvolving training can harm the process, potentially caus- ing deviation in the policy model's distribution.
predictions, ensuring consistency. Following the best prac- tices outlined $3.3, we apply PRM as a reranker to select the top-2 responses with the predicted answer à. These unlabeled prompts then mixed with labeled ones for self-evolving training. Additionally, since learning from un- labeled prompts is more challenging for policy models, we investigate the optimal timing to introduce them into train- ing as well. We maintain an interval of 45k prompts and adjust when unlabeled prompts are introduced into the train- ing process. Specifically, we introduce unlabeled prompts after [0%, 25%, 50%, 75%] of the total training process.
4. Dynamics of Self-Evolution & Final Recipe
So far, we have explored the impact of three pivotal factors within our design space, leading to established best practices for learning multimodal reasoning we adopt continuous self-evolving training coupled with a reward model to help data selection as described in $3.3, and we perform the training process on SFT datasets with final answer annota- tions. In this section, we delve even deeper into the current self-evolution strategy to better understand the bottlenecks. Instead of analyzing from design space perspective as previously, we now fix the design parameters and focus exclusively on the training dynamics during the model's self-cvolution. This shift in focus allows us to examine the process from orthogonal angle, providing further in- sights into the underlying mechanisms that drive or impede progress in multimodal reasoning capabilities.
A Glimpse at Unlabeled Prompts: Potential Efforts to Make Them Effective Table 3 presents the results of in- corporating unlabeled prompts with and without oracle reward signals. When training relies solely on oracle re- ward signals without the PRM, continuous self-evolving with unlabeled prompts outperforms standard continuous self-evolving trained only on labeled prompts in the out-of- domain test but underperforms in the in-domain test. This indicates that additional prompts help the model generalize better to underrepresented questions but also increase the risk of forgetting previously learned information. However, after combining with our PRM, all policy models perform worse than our best model trained exclusively on labeled prompts in both benchmarks, even when oracle reward sig- nals are provided. Based on the analysis in $3.3, this occurs since our PRM is unable to verify responses without ground- truth answers, and its generalization remains a concern.
4.1. Monitoring the Training Dynamies
Intuitively, two critical conditions must be met for the suc- cess of self-evolving training: (I) the presence of high- quality candidate responses generated by the model, oth- erwise self-evolving will not work no matter how strong the reward is; and (2) the reward function's ability to effec- tively distinguish and prioritize these high-quality responses. These conditions align with the traditional reinforcement learning concepts of exploration and exploitation. Appar- ently, both exploration and exploitation capabilities are dy- namic targets in self-evolving training, as the policy model evolves and the distribution of rollout responses changes with each iteration. To better understand these training dy- namics, we propose tracking and visualizing three metrics, where we introduce novel metric, Reward-Pass@@r to monitor the ssploration-expooiiatiio trade-offs:
When examining the timing for introducing unlabeled prompts, we find that adding them from the beginning helps mitigate the negative impact on model performance, com- pared to introducing them in midway. However, when unla- beled prompts are introduced later in the training process, they participate less in the overall training, leading to bet- ter results simply due to their limited involvement. This suggests that, without sufficient surrogate supervision (c.g., precise reward signals), introducing unlabeled prompts into
Greedy Accuracy: the model's accuracy with greedy de- coding. We track this metrie for reference to compare
6

---

Diving into Self-Evolving Training for Multimodal Reasoning
55
Table 4: Results on Math Vista with various training strate- gies across multiple model sizes. We highlight the relative improvement of M-STAR over the pre-evolved model, i.e., the "+warmup" row. CPM = MiniCPM-Van V-2.5, Phi Phi- 3.5-vision, and Intern Intern VL2-2B.
50
63
Pass
60
RM Pass@2
46
40
51
20
25
Variant
CPM
Phi
Intern
20
Base
52.4
46.5
46.4
2500
5000
7500
10000
2500
5000
7500
10000
+warmup
52.8
49.3
47.6
Training Steps
Training Steps
SFT
54.7
49.5
41.9
(a)
(b)
Iterative RFT
55.7
50.2
47.5
RestEM EM
55.1
50.5
47.9
Figure 4: (a): @ decreases for all different temper- atures; (b): The Reward-Pass -rrsad- saturates quickly. All metrics are calculated on validation set. with other metrics.
Cont. Self-Evolving
57.2
51.1
48.4
PRR Re-Rank
59.21 6.4
53.21
3.9
48.8
M-STAR (Reward-Pass@2)
59.5, 6.7
54.6 5.2
50.3
2.7
the reward model's capacity to exploit further diminishes as training progresses. This limitation could be due to both the reduced exploration ability and the inherent constraints of the reward model. Next, we fix the reward model as control variable and ask, how can we enhance exploration to allow the reward model to exploit more effectively?.
Pass @ K Accuracy: the percentage of samples for which the model produces least one correct response when sampling K candidates. This metric measures the model's exploration ability.
Reward-Pass @2 2: the ratio (%) of samples for which there exist correct responses among the top 2 responses ranked by the reward model. This metric directly reflects the exploitation efficacy of the reward model for the current policy. We choose @ass@2 since our training strategy involves selecting the top responses using the reward model ($3.3).
4.2. M-STA R- Final Recipe with Optimal Design Choices & Adaptive Explorations
Reward-Pass(e closely relates to the effectiveness of our self-evolving training strategy since our method selects top responses ranked by the reward model, and Reward- @ directly reflects the quality of these 2 responses.? ? While Reward-Pass @2 naturally measures exploitation when the policy is fixed, the absolute value of this metric actually encapsulates both exploration and exploitation - its value would be low if the model fails to explore high-quality candidates. Therefore, we hypothesize that enhancing the newarddPPsssee K scores for the current iteration through varied configurations could potentially improve the efficacy of self-evolving training. We fix reward model as control variable and focus on modifying the model's exploration capabilities to achieve this objective. Analysis in $4.1 sug- gests that the temperature, which is crucial for exploration, may require dynamic adjustment. Thus we propose to adjust the temperature automatically at each iteration based on the validation Reward- Piis-Pass @2 scores. This aims optimize ex- ploration that the selected responses of higher quality, potentially enhancing overall training effectiveness.
Specifically, after each training iteration of our current op- timal strategy, we sample 16 responses from the model checkpoint on the validation set, with the temperature range set to (0.5, 0.7, 1.0, 1.2, 1.5, 1.7, 2.0]. We analyze with varying temperatures as temperature is a key hyperparameter for the generation diversity and model's exploration.
Results Figure 3 shows clear trend where, as training progresses, the Pass@K K metric continuously declines while greedy accuracy improves, This pattern indicates the loss of exploration ability, which hampers the model's 's potential for continuous improvement and may lead to performance sat- uration. These observations are consistent with findings in text-only settings reported by Wu et al. (2024). In Figure 4a we analyze @ass@K K accuracy at various temperatures and observe a significant trend: despite a general decay in exploration ability, larger temperatures tend to resist this decline more effectively, allowing the model to maintain a stronger ability to explore in the mid to late stages of train- ing. This observation suggests that the optimal temperature for training may need to be dynamically adjusted through- out the self-evolving process, rather than being fixed at the outset as is currently common practice.
'While improvements to the reward model could also enhance Reward-Pass@2. we reserve for future work.
"We note that there is a slight mismatch between Reward- @ass@2 and our training strategy, we pre-filter responses using the ground -trth answer before the reward model reranks them. Ideally, a more aligned metric would measure the CoT reasoning quality of the top 2 responses, both containing correct answers. Given that there is no reliable method to score the quality of the thought processes, we consider Reward-Pass @ 2 as reasonable approximation which turns out he effective empirically.
In Figure 4b, we observe that the Reward-Passe Pasidd- metric ini- tially increases but quickly reaches a plateau, indicating that
7

---

Diving into Self-Evolving Training for Multimodal Reasoning
Table 5: Performance of M-STAR compared with baselines. We highlight the relative improvement of M-STAR over the pre-evolved model, ie., the "+warmup" row. For benchmark with suffix "-R", we follow Xu et al. (2024a) to remove some perception sub-tasks in them, to get the subsets that focus more on reasoning.
MathVista
M3CoT
MMSur-R
MMBench-R
A12D
Average
MiniCPM-V.. V-2.5
52.4
41.2
44.6
72.6
64.4
55.0
warmup
52.6
47.8
45.1
76.9
65.9
57.7
M-STAR
59.5, 6.9
48.7, 0.9
50.7, s.6
79.9,
69.1, 3.2
61.6, 3.9
Phi-3.5-vision
46.5
39.4
42.5
56.8
47.5
46.5
warmup
49.3
46.5
44.2
70.9
65.5
55.3
M-STAR
54.5 5.2
51.3, 4.8
48.8 4.6
73.6, 2.7
67.9 2.4
59.2; 3.9
InternVL-221 VL2-2B
46.4
16.7
20.0
14.2
33.5
26.2
warmup
47.6
45.6
41.8
68.8
60.0
52.8
M-STAR 50.3 2.7
47.1, 1.0
42.0, 0.2
67.31 1.5
59.7 0.3
53.3 0.5
Specifically, we adjust the temperature per two iterations, and pick the temperature from 0.3 to 1.6 with interal 0.1 automatically with maximum validation Heward-Pass@er scores. The optimal design choices outlined in $3, com- bined with our adaptive exploration strategy, form fi- nal recipe for multimodal self-evolving training for rea- soning, M-STAR. For experiments on Phi-3.5-vision and
66
55..0
RM Pass @2 Acc
64
Pass Ace
-5.55
62
-50.0
60
.47.5
58
Passfik Phoa@iK-etttic RM Parsi82
VL2, considering the limited capacity of these models and the computational cost, we utilized both the warmup data and multimodal PRM based on MiniCPM-V-2.5
56
-5.00
2000
4000
6000
10000
# Training Steps
Full Results Table presents the results of our final ap- proach well the comparison with representative base- lines. We also demonstrate the scores on all sub-tasks of Math in Table 6, Appendix H. We see that by incor- porating the dynamics of Reward-Pard Pass 2, which balances both exploration and exploitation, our final recipe achieves the highest results for three backbone LMMs. addi- tion overall trend, we observe that self-evolving training based on larger models yields more comprehensive improve- ments. We assume that the smaller model like InterVVL-221 may struggle to generalize its learned abilities across dif- ferent domains as effectively as the larger models, such as MiniCPMV-2.5 and Phi-3.5-vision.
Figure 5: Comparing the smoothed Pass @ K and Reward- Pass @2 curves with the optimal static training progress, which fixs temperature T 1.0.
(Chen et al., 2024a), MMBench (Dev set, v1.1) (Liu et al., 2025), A12D (Kembhavi et al., 2016). For MMStar and MMBench, we remove the perception sub-tasks to construct subsets focus more on reasoning. As shown in Table 5, models self-evolved with M-STAR consistently outperform both the base models and those trained with warmup across nearly all benchmarks. The only exception is Intern VL2-2B, which underperforms on two benchmarks, aligning with the speculations discussed above. Smaller models face chal- lenges in generalizing beyond their training data, particu- larly on perceptioo-inteenivve benchmarks like MMBench-R and AI2D. In contrast, larger models such as Phi-3.5- 3-viinn and MiniCPM-V-2.5 show significantly improved general- ization, despite being trained with the same query set.
We also plot how the @ and Reward-Pann Pass@2 change for M-STAR when trained on MiniCPM-V-2.55. To align with training, show the metrics corresponding the selected temperature in each iteration (see Appendix G for others). Figure 5 shows that compared with choosing fixed temperature over the whole process statically, tuning automatically mitigate the regression of Pass K to avoid the exploration loss. Besides, the Reward- seward-Pass@e is also generally higher than before. These further highlight the necessity monitor the dynamics and adjust accordingly.
5. Conclusion
We dive into the self-evolving training for multimodal rea- soning. Three static components are identified at first, namely the training method, reward model and the prompt variation. Through controlled experiments, we conclude a set of optimal design choices. On the other direction, we also go deeper into the dynamics of self evolvvn training to
M-STAR on More Diverse Benchmarks To further inves- tigate how well M-STAR generalizes to multiple bench- marks, we select four extra multi-modal ones focus on reasoning as well: M3C.T (Chen et al., 2024b), MMStar
8

---

Diving into Self-Evolving Training for Multimodal Reasoning
analyze the trade-off between exploitation and exploration. By balancing the training dynamics, we are able to further improve its performance. We hope our work can provide insights and guidance for future research on self-evolving training for multimodal reasoning.
446. URL https: tttps://aclannthll.. wwwpgy..rrg/2// acl-long. .446.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing. S., Zhong, M., Zhang, Q. Zhu, X., Lu, I.., et al. Internvi: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-241999 2024c.
Acknowledgments
This project is partially supported by NSFC Grant 62306177.
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang. X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z. Li, Z., Gao, Z., Liu, A., Xue, B.. Wang, B., Wu, B., Feng. B., Lu, C., Zhao, C., Deng. C.,, Zhang, C.,, Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F. Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Ding, H., Xin, H., Gao. H., Qu. H., Li, H., Guo, J., Li, J., Wang, J., Chen, J., Yuan, J., Qiu, J., Li, J., Cai, L.,, Ni, J., Liang. J., Chen, J., Dong, K., Hu, K., Gao, K., Guan, K., Huang, K., Yu, K., Wang. L.., Zhang, L., Zhao, L.,, Wang, L.,, Zhang, L., Xu, L., Xia, L., Zhang, M., Zhang, M., Tang, M., Li, M., Wang, M., Li, M., Tian, N., Huang, P., Zhang, P., Wang, Q. Chen, Q. Du, Q. Ge, R., Zhang, R., Pan, R., Wang, R., Chen, R. J., Jin, R. L., Chen, R., Lu, S., Zhou, S., Chen, S., Ye, S. Wanng S., Yu, S., Zhou, S., Pan. S., Li, S. S., Zhou, S. Wu, S., Ye, S., Yun, T., Pei, T., Sun, T., Wang, T. Zeng, W., Zhao, W., Liu, W., Liang, W., Gao, W., Yu, W., Zhang, W., Xiao, W. L., An, W., Liu, X., Wang, X., Chen, X., Nic, X., Cheng, X., Liu, X., Xie, X., Liu, X., Yang, X., X., Su, X., Lin, X., Li, X. Q., Jin, X., Shen, X., Chen, X., Sun, X., Wang, X., Song, X. Zhou, X., Wang, X., Shan, X., Li, Y. K., Wang. Y. Q.. Wei, Y. X., Zhang, Y. Xu, Y., Li, Y., Zhao, Y., Sun, Y., Wang. Y., Yu, Y., Zhang, Y.. Shi, Y., Xiong, Y., He, Y., Piao, Y., Wang, Y., Tan, Y., Ma, Y., Liu, Y., Guo, Y., Ou, Y., Wang, Y., Gong, Y., Zou, Y., He, Y., Xiong, Y., Luo, Y., You, Y., Liu, Y., Zhou, Y., Zhu, Y. X., Xu, Y., Huang, Y., Li, Y., Zheng, Y., Zhu, Y., Ma, Y., Tang. Y., Zha, Y., Yan, Y.. Ren, Z. Z., Ren, Z., Sha, Z., Fu, Z., Xu, Z., Xie, Z., Zhang, Z., Hao, Z., Ma. Z., Yan, Z., Wu, Z., Gu, Z., Zhu, Z., Liu, Z., Li, Z. Xie, Z., Song, Z., Pan, Z., Huang, Z., Xu, Z., Zhang, Z., and Zhang, Z. Deepseek-rl: Incentivizing reasoning capability in llms via reinforeement learning, 2025. URL https: / / arxiv org / 12948.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
References
Abdin, M., Aneja, J., Awadalla, H., Awadallah, A., Awan, A. A., Bach, N., Bahree, A., Bakhtiari, A., Bao, J., Behl, H., et al. Phi-3 technical report: A highly capable lan- guage model locally on your phone. arXiv preprint wrXiv:2404.14219, 2024.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J. Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308.12900 2023.
Cai. Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z.,, Chu, P., et al. Internim2 technical report. arXiv preprint arXiv:2403.17297 2024.
Chae, H., Kim, S. Cho, J., Kim, S., Moon, S., Hwangbo, G., Lim, D., Kim, M., Hwang, Y., Gwak, M., Choi, D., Kang, M., Im, G., Cho, B., Kim, H., Han, J. H., Kwon, T., Kim, M., woo Kwak, B., Kang, D., and Yeo, J. Web-shepherd: Advancing prms for reinforeing web agents, 2025. URL https:/ / /arxivv.. org/ abs/ / 250.2555... 15277.
Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Wang, J.. Qiao, Y., Lin, D., al. Are we on the right way for evaluating large rision-languse models? arXiv preprint
2024a.
Deng, Y., Lu, P., Yin, F., Hu, Z., Shen, S., Zou, J., Chang, K.-W., and Wang, W. Enhancing large vision language models with self self-training on image comprehension. arXiv preprint arXiv:2405.I 19711, 2024.
Chen. Q.. Qin, L., Zhang, J., Chen, Z. Xu, X., and Che, W. CºCOT: A novel benchmark for multi-domain multi-step mulli--odal chain-of- th-tuught.... In Ku, L.W.., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8199-8821 -221,, Bangkok, Thailand, August 2024b. Association for Com- putational Linguistics. doi: 10. 1 12220/////2444//lmmm
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W. Pan, R., Diao, S., Zhang, J., SHUM, K., and Zhang, T. RAFT: Reward ranked finetuning for generative founda- tion model alignment. Transactions on Machine Learn-

---

Diving into Self-Evolving Training for Multimodal Reasoning
ing Research, 2023. ISSN 2835-8856. URL https: //openreeiew net. wwwum? dtw.orrmmmmmmmmmm07m050
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296- 26306, 2024a.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A. et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Liu, J., Li, C., Wang, G., Lee, L., Zhou, K., Chen, S., Xiong, C.,, Ge, J., Zhang, R., and Zhang. S. Self-corrected multimodal large language model for end-to-end robot manipulation, 2024b. URL abs/ 0ass/2005.1744
Fang. Y., Zhu, L., Lu, Y., Wang, Y., Molchanov, P., Cho, J. H., Pavone, M., Han, S., and Yin, H. VILA": Vila augmented vila. arXiv preprint arXiv:2407. 7453, 2024.
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E.,
Liu, X., Zhang, T., Gu, Y., long, I. I.., Xu, Y., Song, X., Zhang. S., Lai, H., Liu, X., Zhao, H., Sun, J., Yang. X., Yang, Y., Qi. Z., Yao, S., Sun, X., Cheng, S., Zheng, Q., Yu, H., Zhang, H., Hong, W., Ding, M., Pan, L., Gu, X., Zeng, A., Du, Z., Song, C. H., Su, Y., Dong, Y. and Tang, J. Visualagentbench: Towards large multimodal models as visual foundation agents, 2024c. URL https: // faraxvvee www.org///aooo abs 2408 06327.
Shi, F., Zhong, R., Yih, S., Zettlemoyer, L., and Lewis, M. Incoder: generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2022.
Gao, Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H., Li, Z., et al. G-llava: Solving geo- metric problem with multi-modal large language model. arXiv preprint 233i: /3770 2023.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mmbench: Is your multi-mon modall model an all-around player? In European Conference Computer Vision, pp. 216-216-33 Springer, 2025.
Gulcehre, C., Paine, T. L.,, Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.0899. 2023.
Lu, P., Bansal, H., Xia, T., Liu, J., Li, C., Hajishirzi, H., Cheng. H., Chang, K.-W,, Galley, M., and Gao, J. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Hosseini, A., Yuan, X., Malkin, N., Courville, A., Sordoni, A., and Agarwal, R. V-star: Training verifiers for self- taught reasoners. arXiv preprint arXiv:2402.064: 06457, 2024.
Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210. 1111111000 2022.
Masry, A., Do, X. L., Tan, Q., Joty, S., and Hoque, ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2279 Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.1883311/2222 findings-ac.1177 URL. https: a http:///ccaaaahlloggo/o/ org 2222 findings ac1.177.
Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV 14, pp. 235-251. Springer, 2016.
Li, X., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y., Zhang, R., Liu, J., and Dong, H. Maniplim: Embod- ied multimodal large language model for object-centric robotic manipulation, 2023. URL https: //axxiv. org/ abs//b////// 23122311222 16217.
Meta. Introducing meta llaaa 3: The most capable openly available 1lm to date, 2024. URL https: / //ai meta. com/ com/blog http//llg//// //eg//eteee
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let's verify step by step. arXiv preprint arXiv:2305.200500 2023.
Pang, R. Y., Yuan, W., Cho, K., He, H.. Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization.
arXiv preprint 2rXiv:2404.1973. 2024.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tuning. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 34892- 34916. Curran Associates, Inc., 2023.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 3748-8765 PML.,, 2021.
10

---

Diving into Self-Evolving Training for Multimodal Reasoning
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1oXiv:1707006445 2017.
Yu, L., Jiang, W., Shi, H., Yu, Liu, Z.,, Zhang, Y., Kwok, J. T., Li, Z.,, Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309, 0 2224,, 2023.
Shao, Z., Wang, P., Zhu, Q. Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint arXiv:2308.01822 2023.
Shi, W., Hu, Z., Bin, Y., Liu, J., Yang, Y., Ng. S.-K., Bing, L., and Lee, R. K.-W. Math-llave llava: Bootstrapping mathe- matical reasoning for multimodal large language models. arXiv preprint ar arXiv:2406. 17294, 2024.
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W. Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuming. arXiv preprint arXiv:2309.0560 2023.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P.. Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:23/2.06588 2023.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning. Advances Neural Information Processing Systems, 35:15476-15444 2022.
Zeng, W. Huang, Y., Liu, Q., Liu, W., He, K., Ma, Z., and He, J. Simplerl-zoos Investigating and taming zero reinforcement learning for open base models in the wild, 2025a. URL https: http///axvvvvv.// abs/ 1503.18892.
Sun, Z., Yu, L., Shen, Y.. Liu, W., Yang, Y., Welleck, S., and Gan, C. Easy-to-hard generalization: Scalable alignment beyond human supervision. arXiv preprint S4ti::2003.09472, 2024.
Wang, P., Li, L.., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y., and Sui, Z. Math-shepherd: Verify and re- inforce LLMs step-by-step without human annotations. In Ku, L..W.,, Martins, A., and Srikumar, (eds.), Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9426 9939,943,,10 Bangkok, Thailand, August 2024. As- sociation for Computational Linguistics. URL https: //aclanthology org/ 2024 acle24.ccl--1 -oellllong.... 510.
Zeng. W., Huang, Y., Zhao, L., Wang, Y., Shan, Z., and He, J. B-STar: Monitoring and balancing exploration and exploitation in self-taught reasoners. In The Thirteenth International Conference on Learning Representations, 2025b. http::/ //pp/ppeeevveeeeeeeeeeeeeeeeeeee net forum? id=P 6dWZJJ.JJ..
Zhai X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sigmoid loss for language image pre-training. In IEEE/CVF International Conference on Computer Vi- sion, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11941-11952. IEEE, 2023. doi: 10.1109//CCV51000. 2023.01100. URL https: /tt///i..og//..111111 ICCV51070.2023 01100.
Wu, T. Li, X., and Liu, P. Progress or regress? self- improvement reversal in post-training. arXiv preprint arXiv:2407.05013. 2024.
Xu, G., Jin, P., Hao, L., Song, Y., Sun, L., and Yuan, L. Llava-cot: Let vision language models reason step-by- step. arXiv preprint arXiv:2411.104
2024a.
Zhang, D., Zhoubian, S., Hu, Z., Yue, Y., Dong, Y., and Tang, J. ReST-MCTS*: LLM self-training via process reward guided tree search. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https: /tpp//eveeeeeeee//////p/p. net/ forum? id-8rcF0qEud5.
Xu, Y., Liu, X., Liu, X., Hou, Z., Li, Y., Zhang, X., Wang. Z., Zeng, A., Du, 7., Zhao, W. et al. Chatglm-math: Improv- ing math problem-solving in large language models with a self-critique pipeline. arXiv preprint 24Xiv2244.002833, 2024b.
Zhang, R., Wei, X., Jiang, D.,, Zhang, Y., Guo, Z., Tong, C., Liu, J., Zhou, A., Wei, B., Zhang, S., et al. Mavis: Mathematical visual instruction tuning. arXiv preprint arXiv:2407.08739, 2024b.
Yang, S., Liu, J., Zhang, R., Pan, M., Guo, Z., Li, X., Chen, Z., Gao, P., Guo, Y., and Zhang, S. Lidar-lim: Exploring the potential of large language models for 3d lidar under- standing, 2023. URL https: //////aaa/ /awxivviviv./////i///a.a//aaaava org/ abs/ 2312. 14074.
Yao, Y. Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Miniepm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800 2024.

---

Diving into Self-Evolving Training for Multimodal Reasoning
A. Details of Selected LMMs
MiniCPM-V-2.5 (Yao et al., 2024) is powerful, openly released LMM. MiniCPM-V-2.5 leverages LLaMA-3-8B (Meta, 2024) for its language model and SigLIP (Zhai et al., 2023) as its vision encoder, resulting in strong multimodal capabilities. Its performance on a wide range of multimodal benchmarks significantly surpasses previous openly released LMMs such as LLaVA (Liu et al., 2023; 2024a) and Qwen- VL (Bai et al., 2023).
Phi-3.5-Vision (Abdin et al., 2024) a multimodal model combining CLIP ViT-L/14 (Radford et al., 2021) image encoder and a Phi-3.5-min transformer decoder. processes interleaved image-text inputs using dynamic cropping for images and is pre-trained on 0.5T tokens from diverse datasets. Post-training via supervised fine-tuning (SFT) and direct preference optimization (DPO) enhances its multimodal reasoning and language understanding capabilities.
Intern VL-2 (Chen et al., 2024c) Intern 2.0 is a multimodal large language model series ranging from IB to 108B parameters. The specific 2B version we use combines Intern ViT (300M) (Chen et al., 2024c), an MLP projector, and InternLM-22-hhatt (1.8B) (Cai al., 2024), showcasing strong vision-language capabilities. Built with progressive alignment training, it efficiently aligns vision and language models while supporting diverse inputs (text, images, video, medical data) and outputs (images, bounding boxes, masks), performing competitively across various vision-language tasks.
B. Warm-Up Phase to Unlock the Chain-of-Thought (CoT) Capability of LMMs
In our preliminary experiments, we found that open-source LMMs would directly output the answer given the query, while struggling to produce detailed chain-of- oftthoughttthtttthh (CoT) reasoning processes. This may originate from the the scarcity of high quality rationales in most existing multimodal SFT training datasets (Masry et al., 2022; Shi et al., 2024), which limits the ability of open-source LMMs to generate detailed, step-by-step reasoning. Self-evolving training, however, requires responses with varying intermediate steps to allow models to learn effectively from on-policy data. To address this issue, we initiate a warm-up phase to collect some CoT data from the model itself as the first step before self-evolving training. Instead of prompting the model to answer questions directly, we prompt it to generate intermediate reasoning steps for a given triplet (question, image, and answer) using the following instruction:
Extra instruction to guide CoT
Offer a comprehensive breakdown of your analytical process, detailing each step, the reasoning behind your decisions, and how you integrated various pieces of information, and put your answer the end.
For each triplet, we ask models to rollout 16 samples with temperature 1.0. We then filter out results where the final answers do not match the ground truth and sample 100K from the generated dataset to create warm-up CoT dataset D,,. with correct answers. Finally, we fine-tune our models on this dataset, treating it as standard RFT process. Our iterative self-evolving training process will then start from this model checkpoint after the warm-up training.
C. Hyper Parameters
We follow the training setup from Yao et al. (2024), using learning rate of lc-6 and batch size of 128. A constant learning rate scheduler with a warmup ratio of 0.1 is applied. Input images are encoded using SigLIP SoVIT-400m/1e 14 (Zhai et al., 2023), and the visual tokens are compressed through a perceiver resampler structure with a single cross-attention layer. Additionally, each input image is sliced into a maximum of 9 segments, with each segment compressed into 96 queries.
D. Training Process Reward Model (PRM)
To train our PRM, first train another checkpoint (denoted as #g) on CoT augmentdd training data for a much longer period to make sure it fully converges.
Based on this model, we leverage Monte Carlo Rollut method (Wang et al., 2024) to collect the training data for PRM. Specially, we randomly pick 50K questions from the full training set, and sample 16 responses for each of them with #g. We de-duplicate these responses, and only keep at most 4 responses for each question. After that we randomly sample 50K question-response pairs from all the pairs, where we control the ratio of correct and wrong responses as 1:1, and the ratio of
12

---

As a complement our analysis in $3.3, additionally plot the number of reasoning steps (split by (n) for the top-2 solutions re-ranked by our process reward model, well as for the other solutions. We observe that the top-2 solutions Diving into Self-Evolving Training for Multimodal Reasoning
multi-choice and free-form question 1:1 well, to keep balanced distribution.
To construct the labels of each step, use #g as the completer to complete the solution from the end of each step in one response. For the hthh step, the step label annotated as N Σラ= (C, = a²), where N(= 16) the number of completion, Cj is the j-th completion.
.V
Based the stepwise annotations, train our PRM from ag. We initialize the linear reward model head the average of the embeddings, and train with MSE loss on all tokens, where the label of each token is identical to the step end token. In experiments we freeze the visual encoder as we find it brings slight improvement.
E. Measuring Response Relativity
To get comprehensive understanding of how our PRM works as a re-ranker, we conduct a quantitative analysis using GPT4-o (gpt-4o) 4ort--40202444 10004-000 00-4-00-000 6) to see how much correct response is directly related to the query, e.g., does not contain irelvaat steps. The prompt we use is as follows:
Prompt for GPT4-o to annotate the relativity score
Given the image and a related question, you need to judge how a candidate solution is directly related to the question. You need to consider all its steps, and return final value bewteen 1-10 as a overall score. Conclude your judgement the end as "So the relativity score is X" where X is the score you give.
[Question] [question}
[Solution] {solution) }
F. Extra Comparison between RM-selected Solutions and Others
Top
0.20
Rost
typically have fewer reasoning steps, indicating that the reward model can effectively identify solutions with fewer irrelevant steps and prioritize more straightforward ones.
Portion
0.15
0.10
G. More Results for M-STAR
0.05
We plot the extra analysis results for M-STAR here. In Figure 7, plot the changes of Pass@k K and Peward-Pass@@2 across different temperatures for M- STAR(Rewand-PPann Pass 2) as a compliment to the adapative adjustion mentioned in $4.2. We can see that acroos all selected temperatures, the exploration ability reflected by Pass does not regress continuously, and Reward-Passe @2 reaches its peak more quickly, compared with training without the monitor of dynamics.
0.00
Steps
Figure 6: Number of reasoning steps for top-2 solutions selected by our reward model and the rest solutions.
H. Full Results of Math Vista
We present the complete evaluation results on Math Vista for all three of our selected models. The scores for each subtask in Vista are reported in Table As shown, M-STAR achieves improvements across all subtasks compared to the pre-evolved models, particularly on geometric problems and math word problems. This demonstrates its enhanced comprehensive multimodal reasoning ability across multiple aspects.
13

---

Diving into Self-Evolving Training for Multimodal Reasoning
66
56
52
63
48
RM Pass
Pass@K
60
44
57
40
36
54
32
0.5
1.5
0.5
1.5
0.7
1.7
28
0.7
1.7
10
24
2500
5000
7500
10000
2500
5000
7500
10000
# Training Steps
# Training Steps
(a)
(b)
Figure 7: (a):Pass@ K changes during the training of M-STAR (Reward-Pass@2)) (b): Rewaard-Pass@) changes during the training of M-STAR (Reward-Passee We pick different temperatures.
Table Full analysis of Vista. Task types: FQA: figure question answering, GPS: geometry problem solving, MWP:
math word problem, TQA: textbook question answering, VQA: visual question answering. We highlight the relative improvement of M-STAR over pre-evolved model, i.e., the "+warmup" row.
Model
ALL
FQA
GPS
MWP
TQA
VQA
MiniCPMV-2.5
MiniCPMV-2.5
52.4
59.2
44.7
50.5
53.8
48.0
+warmup
52.8
58.4
47.1
57.0
53.8
45.8
SFT
54.7
58.7
50.5
56.5
55.7
50.8
Iterative RFT
55.7
59.1
49.5
65.6
55.1
48.0
Rest
55.1
58.0
49.5
64.5
55.1
47.5
Cont. Self-Evolving
57.2
56.3
65.1
$7.0
49.7
+PRM Re-Rank
59.2, 6.4
59.1 0.7
61.1,
68.3 11.3
55.1
51.4, 5.6
M-STAR (Reward- Beaward-Pass@2. 2)
59.5
59.5
59.1,
65.6 8.6
58.9
54.2
Phi-3.5-vision
Phi-3.5-vision
46.5
36.5
36.0
41.9
+warmup
49.3
55.8
42.8
53.2
55.1
38.0
SFT
49.5
53.9
52.9
52.7
49.4
35.8
Iterative RFT
50.2
58.4
41.4
50.0
55.7
43,0
Rest EM
50.5
56.8
46.6
49.5
58.9
39.7
Cont. Self-Evolving
51.1
56.1
48.6
55.9
52.5
40.2
+PRM Re-Rank
53.2 3.9
56.9, 1.1
51.9; 9.1
60.8 7.6
55.1p
39.7, 1.7
M-STAR (Reward-Pass@@2)
54.5- 6.2
56.9, 1.1
56.7, 13.9
57.5; 4.3
55.1o
44.7 6.7
InternVL2-28
Intern VL.2-BB
46.4
53.2
45.2
33.3
50.0
48.0
+warmup
47.6
52.4
54.8
46.2
43.7
36.9
SFT
41.9
37.5
40.4
49.5
50.8
Iterative RFT
47.5
49.8
57.7
52.1
41.8
32.4
EM
47.9
49.4
54.8
51.1
51.3
31.3
Cont. Self-Evolving
48.4
50.5
56.5
40.5
37.4
+PRM Re-Rank
48. 1.2
52.0,
55.8
52.1
45.6r 1.9
35.21 1.7
M-STAR Beeward-Pass@e
50.3 2.7
49.4
57.2
65.0; 18.8
42.4,
35.2,
14

---

Diving into Self-Evolving Training for Multimodal Reasoning
I. More Related Works
In this section we will briefly introduce other works that are related to our study that cannot be elaborated in the main context due page limit.
Self-Evolving Methods The most straightforward and widely-used approach to enhance a model's 's reasoning ability is through supervised fine-tuning (SFT), where models mimic the outputs of highly capable models (Yu al., 2023; Yue et al., 2023). However, as the gap between open-source models and proprietary ones narrows, the performance improvements from SFT tend to plateau. This has led to increased attention self-evolving methods, where models refine and improve themselves without external supervision, as a means to further boost their reasoning abilities.
Some early self-evolving approaches primarily focus on single-round improvements. For instance, LMSI (Huang et al., 2022) leverages CoT prompting combined with self-consistency to generate high-confidence solutions from unlabeled data, which are then used augment the training process. Similarly, RFT (Yuan al., 2023) enriches the training data by filtering solutions using existing labels. Both methods apply this augmentation process in just one iteration.
On the other hand, several works have explored iterative approaches for seff-improvement. Notably, STaR (Zelikman et al., 2022), ReSTEM (Singh et al., 2023), and V-STaR (Hosseini et al., 2024) retrain their models from the original checkpoint after each iteration, while RAFT (Dong et al., 2023), ReST (Gulcehre et al., 2023) and ReST-MCTS" (Zhang et al., (00244))ontinuouss fine-tune models starting from the previous iteration's 's checkpoint, Reinforcement Learning (RL) techniques also fit into this iterative category, offering an online mechanism that tightly couples exploration and exploitation. RL methods, such as PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024), are frequently applied unlabeled data, using an additional reward model to evaluate the quality of generated responses. GRPO, in particular, streamlines the process by removing the value model from PPO and instead leverages in-batch comparison estimate advantages across different rollouts, providing a more stable alternative.
Multimodal Reasoning Currently, the most common approach to improve multimodal reasoning capabilities continues to be supervised fine-tuning (SFT). For example, G-LLaVA (Gao et al., 2023) augments existing geometry question-answeriin datasets to fine-tune the LLaVA-1.5 model (Liu et al., 2024a). Math-LLa LL (Shi et al., 2024) selects and augments data from larger multimodal question-answer datasets, carefully balancing the difficulty of samples. Similarly, MAVIS (Zhang al., 2024b) focuses on the geometry and function domains and generates instruction-bain based tuning data through synthetic data engines.
However, recent works have begun incorporating self-evolving mechanisms into multimodal reasoning. For instance, VII.A (Fang et al., 2024) iteratively improves its image captioning performance by generating increasingly detailed captions, which are subsequently used to retrain the model. LLaMA3-V V (Dubey et al., 2024) employs a reject-sampling strategy to generate missing explanations for question- question-answer pairs that lack intermediate reasoning steps in existing multimodal datasets, thereby enhancing the model's ss reasoning capabilities.
Connecting to Recent Rule-based RL Recent work in online RL with rule-based rewards has demonstrated strong performance on complex reasoning tasks like MATH (DeepSeck-Al et al., 2025; Zeng et al., 2025a), suggesting that using PRMs in GRPO training may constrain improvement. While these approaches leverage base- rewards to enhance the reasoning capabilities of (M)..Ms, our M-STAR framework is compatible with these findings for two key reasons. First, our reward strategy is also rule-based: we filter out responses with incorrect answers before applying PRM, effectively using PRM as a reranker to select trajectories with the highest-quality reasoning steps. Second, our training setup differs significantly from approaches like DeepSeck- R1. While RI applies policy gradients at every step using PRM (as in GRPO), which can introduce reward hacking issues, our STaR-like (Sun et al., 2024; Zeng et al., 2025b) method avoids step-wise policy gradients. Although RL with rule-based rewards provides excellent performance, we believe exploring how to effectively train PRMs tailored to more general scenarios (Chae al., 2025), and integrating PRMs into online RL training, remain valuable directions for achieving even higher performance ceilings.
15


Reviews:
[{"review_id": "wI0lgK1X1I", "reviewer": "Reviewer_grRq", "summary": "The authors reframe self-evolving training for multimodal reasoning through the lens of RL and indentify three factors: training method, the use of reward model, and prompt variation. They train the first multimodal, process-based reward model for multimodal reasoning and\ndemonstrate its usefulness in further enhancing performance; and find that adding more unlabeled queries helps only when having perfect reward signals (e.g., the oracle groundtruth answers), and it hurts the performance if the reward model does not generalize well on unseen data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and effort in reviewing our paper.\n\n# Q1\n> limited technical novelty \n\nOverall we have contributed:  \n- A pilot study to enhance multimodal (MM) reasoning validated by comprehensive studies,  \n- The first self-evolving training recipe that blended online training and PRMs in MM reasoning\n- The new method, monitoring training dynamics, and the experiments to show the effectiveness of it.\n- Significant improvements on various multimodal MM benchmarks\n\nThey span three underexplored areas:\n- A comprehensive RL-based analysis of self-evolving training;\n- A new training framework and PRM tailored to multimodal reasoning;\n- A methodology for tracking and interpreting training dynamics, offering insights into model evolution.\n\n# Q2\n> lack of a hypothesis-driven structure\n\nOur central research question is outlined in the earlier sections of the paper (Abstract, Sec 1 Lines 12-26, 43-48). \n\nWhile writing structures vary across papers, we adopt a component-wise, empirical exploration to derive insights—a format used in prior works like [1,2]. We believe this structure does not diminish the rigour&clarity of our contributions.\n\n[1] Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback, NeurIPS 2024\n\n[2] What Matters When Building Vision-Language Models? NeurIPS 2024\n\n# Q3\n> Why do the authors choose Minicpm, internvl, and phi models ...\n\nWe selected them as they were among the strongest open-source LMMs of their sizes (2B, 4B, 7B) at the time of our experiments, offering solid reasoning abilities necessary for self-evolving training.\n\nWe prioritized:\n- Models with solid baseline reasoning, critical for RL-based methods\n- Stronger open-source models, as improving them is more meaningful than weaker ones.\n\nAlthough LLaVA and Qwen-VL are widely used, they underperformed during our study period. For instance, LLaVA-1.6 achieved only ~20–30 on MathVista, and Qwen-VL-Chat lagged behind MiniCPM-V 7B ( ~40 v.s. 52.8). \n\nOur method is model-agnostic and can be extended to these models in future work.\n\n# Q4\n> Evaluation on more MM reasoning benchmarks\n\nIn Tables 5 and 6, the five benchmarks we use include many diverse subtasks, covering not only math but also tasks such as visual qa, figure qa, logic-qa, scientific reasoning, spatial reasoning, etc, which are very diverse and challenging. And previous works[1,2] often evaluate on just 1-2 benchmarks\n\n[1] Bootstrapping Mathematical Reasoning for Multimodal Large Language Models EMNLP2024\n\n[2] Mathematical Visual Instruction Tuning with an Automatic Data Engine ICLR2025\n\n# Q5\n> baseline comparisons are insufficient\n\nOur work is the first to systematically apply self-evolving training to MM reasoning. In Table 1, we also include the most widely used self-evolution baselines in text-only settings: Iterative RFT (aligned with STaR [1] and ReST [2]) and RestEM—three of the most established methods at the time of submission.\n\n[1] STaR: Bootstrapping Reasoning With Reasoning Neurips2022\n\n[2] Reinforced Self-Training (ReST) for Language Modeling\n\n# Q6\n> Can the proposed method further improve the existing MLLM reasoning methods? Just using a base model (like Minicpm) is not solid.\n\nAs shown in Table 4, we have validated our method on three models of varying sizes, and the conclusions remain consistent. Also, as in many MLLM/LLM reasoning works (e.g. ReST, V-STAR), it's common to mainly conduct experiments on top of general models  rather than stacking different reasoning methods.  \n\n# Q7\n> The ablation studies should delve deeper into algorithmic comparisons…\n\nAt the time of submission, STaR-like methods were among the most effective and compute-efficient for reasoning tasks, especially in MM contexts, as deployed to develop e.g. Llama3, ReST-MCTS* etc. Other RL paradigms like GRPO were less explored in this setting and are beyond our scope.\n\n# Q8\n> Related Work\n\nWe leave related work in **Appendix I**. And we will move them to main body and add more recent papers in the next revision.\n\n# Q9\n> PRM Training Details\n\nWe have provided details in **Appendix D**, but briefly: The PRM is trained using via MC-Rollout, where 50K questions are sampled and each is completed with up to 16 responses using a converged model checkpoint. Stepwise annotations are generated based on completion correctness, and the PRM is trained with token-level MSE loss. The dataset is balanced across correct/incorrect responses and question types.\n\n# Q10\n> Unlabeled Prompts\n\nUnlabeled prompts simulate real-world scenarios where collecting answers is difficult. Since PRM is trained on diverse reasoning steps, we test whether it can generalize to unlabeled data—enabling broader scalability beyond labeled datasets.\n\n# Q11\n> Training Data\n\nIn line 121 of our paper, we use MathV360K, which includes not only MM math problems, but also a diverse range of tasks like function QA, figure-based QA, and more, which covers a broad spectrum of multimodal reasoning scenarios.", "claims_and_evidence": "The authors reframe self-evolving training for multimodal reasoning through the lens of RL and indentify three factors: training method, the use of reward model, and prompt variation. But the experiments do not deeply study how these three factors effect the final results. (i.e., the ablation studies are not enough, the settings of the ablation studies focus primarily on minor hyperparameter adjustments, leading to conclusions that align with conventional expectations.)", "methods_and_evaluation_criteria": "The technical contribution is limited, it seems the authors use some existing LLM  techniques in MLLM. However, what are the new technical challenges induced by MLLM and how do the authors solve them? The paper lacks a hypothesis-driven structure that ties the findings to the central research question, which appears more like a tech report rather than a research paper.", "theoretical_claims": "Not available.", "experimental_designs_or_analyses": "1. Why do the authors choose Minicpm, internvl, and phi models as base models? Can the proposed method use in LLaVA and QwenVL? I think these two models are more common in MLLM research.\n2. The experiments are not solid enough, the proposed method should conduct on more multimodal reasoning benchmarks.\n3. The compared baselines are not enough,  For instance, the author should compare their method with some existing self-evolution methods in LLM and MLLM reasoning methods. \n4. Can the proposed method further improve the existing MLLM reasoning methods? Just using a base model (like Minicpm) is not solid.\n5. The ablation studies should delve deeper into algorithmic comparisons. For instance, contrasting with techniques like RLHF,  DPO, GRPO, as well as exploring different training methodologies (e.g., multi-training stages), the sequence of training stages.", "relation_to_broader_scientific_literature": "The paper lacks a hypothesis-driven structure that ties the findings to the central research question, which appears more like a tech report rather than a research paper.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "1. The paper is well-organized and easy to follow.\n2. The research topic: enhancing reasoning in large multimodal models without external annotated data is a good topic.", "other_comments_or_suggestions": "The related work about Multimodal Reasoning is not enough, the authors should consider more recent papers.", "questions_for_authors": "How do you train the PRM? Can you give me a more clear explanation? The data, training method, etc.\n\nWhat is unlabeled prompt? In line 126, I find the answer is missing. Is it just a math question? \nBesides, why do you use the word 'prompt' to describe this answer missing setting, I think is a little bit weird. Why not use label?\n\nIn Tab5., what is the training data? Just the math data can improve?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CpUkygTubv", "reviewer": "Reviewer_gU79", "summary": "The paper introduces M-STAR—a framework that reframes self-evolving training for multimodal reasoning as a reinforcement learning (RL) problem. It identifies three critical factors (training method, reward model, and prompt variation) and proposes a continuous self-evolving training variant. A novel Process Reward Model (PRM) is designed to assess intermediate reasoning quality, and an adaptive exploration strategy (via automatic temperature tuning) is introduced to mitigate performance saturation. Extensive experiments across multiple benchmarks (e.g., MathV360K, MathVista) and model sizes (MiniCPM-V-2.5, Phi-3.5-Vision, InternVL2-2B) demonstrate consistent gains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Q1\n\n> However, the effectiveness of PRM as a reranker—despite underperformance on standard verification metrics—needs further clarification\n\nThank you for your question.\nFirst, we would like to clarify an important distinction between Best-of-N (BoN) selection and reranking using PRM. When using PRM for BoN, no ground-truth answers are provided—meaning the PRM must identify both correct and incorrect responses independently. In contrast, during our training (excluding the unlabeled prompt setting), the responses are already **filtered using ground-truth answers**, allowing the PRM to focus on reranking correct responses. We then select the top-2 responses as high-quality samples. This is why the BoN results do not align with our usage of PRM during training.\n\nTo better investigate and validate the behaviour of the PRM, we conducted four analyses:\n\n- **Value Function Perspective**:\nOur PRM is trained via MC-rollouts [1, 2], acting as a value function estimating the expected reward (i.e., answer correctness) from an intermediate reasoning step. To assess accuracy, we compare the predicted value score (the PRM score at the step with the lowest value) against an empirical value score (the proportion of 16 rollouts from that step leading to a correct answer).\nWe then compute the mean squared error (MSE) between the PRM-predicted value and this empirical value across the dataset, grouped by whether the original response was ultimately correct or incorrect. \n\n|       |  Correct   |  Wrong   |\n|-----|-----|-----|\n|  MSE   |   0.081  |   0.124  |\n\nThis result indicates two things. \n\n1. It highlights why our PRM here is more suitable as a reranker rather than a verifier—while PRM effectively distinguishes good reasoning trajectories for correct responses, its predictions are less reliable for incorrect responses, where it can be more easily confused.\n\n2. Responses filtered by answers with higher PRM scores are with higher quality, making them better samples to learn.\n\n\nThis suggests future evaluations could separate reranking performance (when answers are known) from value estimation (on unlabeled or incorrect responses).\n\n- **Human Evaluation**:\nTo validate the reliability of the relativity score presented in Figure 2 in our paper and assess whether GPT-4o is unbiased, we conducted a human evaluation to measure the alignment between questions and responses. Specifically, we invited two human annotators to label 50 responses. To facilitate more consistent judgments, we categorized the relativity score into three levels: not relevant (corresponding to original scores 1–4), somewhat relevant (scores 4–7), and very relevant (scores 8–10). We then computed the agreement between the human annotations and GPT-4o's automatic scores. The average agreement exceeded 84%, which, according to prior research on automatic evaluation [3], indicates a strong level of consistency. This result supports the reliability of both our proposed relativity score and the GPT-4o annotations shown in Figure 2.\n\n- **Readability Analysis**:\nWe use the Flesch Reading Ease (FRE) metric, we found that top-2 responses are more consistently readable than other samples with slightly higher mean (64.48 vs. 64.25) and much smaller variance (28.48 vs. 55.97)\n\nThis suggests PRM reranking helps reduce incoherent or erratic language, yielding clearer responses overall.\n\n[1] Math-Shepherd: ... ACL2024\n\n[2] ProcessBench\n- **Qualitative Case Study**:\nThe stepwise reward provides a finer signal of reasoning quality. Even when answers are correct, reasoning may be flawed. For example:\n```\nQuestion: What type of MRI is shown in the image?\\nChoices:\\n(A) T2-weighted  \\n(B) Diffusion-weighted  \\n(C) FLAIR  \\n(D) T1-weighted.\n\nTop-2 Response: The choice is D, because the image is described as having a \\\"very bright signal\\\", ….\\n\\nDark areas on T1-weighted MRI typically …\\n\\n# Answer\\n\\nD\n\nOther Response: The given MRI is indicated as a diffusion-weighted analysis.\\n\\nTherefore, the correct answer to the problem is: \\n\\n# Answer\\n\\nD\n```\nAlthough both predict the right answer, the top-2 response has clearer and more consistent reasoning, while the other includes misleading claims—underscoring the value of stepwise evaluation.\n\n# Q2\n\n> limited diversity in benchmark tasks\n\nAs shown in Tables 5 and 6, the five benchmarks we used include many diverse subtasks, covering not only the math domain but also tasks such as visual qa, figure qa, logic-qa, scientific reasoning, spatial reasoning, etc, which are very diverse and challenging. And they are also common practices to evaluate multimodal reasoning (e.g. math-llava)\n\n# Q3\n\n> How robust is the continuous self-evolving..\n\nFor the robustness of hyperparameters, in table 4, we follow the same parameters as Table 1,2 to train Phi-3 and InternVL models. Considering they are two models with different series and sizes, but the results remain consistent with MiniCPMV, we believe it proves hyperparameters in MSTaR are robust.", "claims_and_evidence": "Claims: \n\n(i) continuous self-evolving training outperforms conventional iterative methods\n\n(ii) integrating a multimodal PRM enhances candidate selection\n\n(iii) adaptive temperature tuning effectively balances exploration and exploitation\n\nEvidence: \n\nThe authors support these claims with controlled ablation studies (e.g., Table 1 and Table 2) and analysis of metrics such as Reward-Pass@2 and Pass@K. However, the effectiveness of PRM as a reranker—despite underperformance on standard verification metrics—needs further clarification.", "methods_and_evaluation_criteria": "The proposed method is well aligned with the multimodal reasoning challenge, specifically addressing the scarcity of high-quality chain-of-thought annotations. The evaluation covers multiple baselines, model sizes, and both in-domain and out-of-domain benchmarks.", "theoretical_claims": "The reformulation of self-evolving training as an RL objective is sound, and the derivations (e.g., Eq. 1 and Eq. 2) appear correct. While the continuous optimization approach is promising, further details on convergence guarantees and stability—especially during adaptive temperature tuning—would enhance the theoretical contribution.", "experimental_designs_or_analyses": "The experimental design is comprehensive, with clear comparisons between various training strategies (iterative, continuous, PRM-enhanced). The analysis of exploration–exploitation dynamics via Reward-Pass@2 is particularly insightful. However, the potential impact of noisy unlabeled data in the PRM setup warrants deeper discussion.", "relation_to_broader_scientific_literature": "The paper is well-situated within recent work on self-training, multimodal reasoning, and RL-based training methods (e.g., references to Singh et al. (2023)[1], Zelikman et al. (2022)[2], Hosseini, Ali, et al.[3] and related chain-of-thought literature). \n\n[1] Singh, Avi, John D. Co-Reyes, and Rishabh Agarwal. \"Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.\" ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models\n\n[2] Zelikman, Eric, et al. \"Star: Bootstrapping reasoning with reasoning.\" Advances in Neural Information Processing Systems 35 (2022): 15476-15488.\n\n[3] Hosseini, Arian, et al. \"V-STaR: Training Verifiers for Self-Taught Reasoners.\" CoRR (2024).", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n- Novel reformulation of self-evolving training as an RL problem. \n\n- Introduction of continuous training and adaptive exploration, supported by thorough empirical evaluations.\n\n- Detailed analysis of reward model dynamics and exploration–exploitation trade-offs.\n\nWeaknesses:\n\n- Some ambiguity remains regarding the PRM’s role as a reranker given its lower performance on standard verification metrics\n- limited diversity in benchmark tasks may restrict claims about generalizability.", "other_comments_or_suggestions": "Minor typos (e.g., “mutlimodal” instead of “multimodal” in Line 013 col 2 word 6) should be corrected. Though not critical, some explanations—especially around the adaptive temperature mechanism—could be clarified for enhanced readability.", "questions_for_authors": "1. Can you clarify the apparent discrepancy between the PRM’s verification metrics (e.g., BoN, weighted voting) and its effectiveness in reranking responses?\n\n2. How robust is the continuous self-evolving training process to variations in iteration interval and temperature adjustment parameters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NSUHk6XkPV", "reviewer": "Reviewer_dYgr", "summary": "This paper identifies three key components in multimodal reasoning models that require further exploration. It systematically analyzes and unveils the critical aspects of training methods, reward models, and prompt design. Additionally, it proposes the use of appropriate temperature adjustment to balance exploration and exploitation. The final approach is validated through extensive experiments.  \n\n\n## update after rebuttal\nThank the authors for the clarifications. I would like to keep my current rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. And we appreciate your recognition of your work. We would address your concerns one by one:\n\n# Q1\n\n> Line 175-178 “...switching over the Improve and Generate steps too frequently makes the learning process unstable, leading to a lower score, especially on the in-domain test set.” lacks of evidence.\n\nThanks for your question. In our table 1., less interval means switching over the Improve and Generate Steps more frequently, and we can found from the results that too small intervals would hurt the perf in in-domain test set (MathV360K testset)\n\n# Q2\n\n> Please re-organize the first paragraph in Section 3.2 to make sure that “iteration interval” is defined first before being used to avoid unnecessary confusion.\n\nThanks for your suggestion. We would follow your suggestion to define the “iteration interval” first and further clarify it.\n\n# Q3\n\n> “…we fix training methods as continuous self-evolving with 45k interval” Please specify which iteration interval the 45k interval refers to. [6.25%,12.5%,25%,50%,100%]\n\n\nWe are sorry for the confusion. Since the total amount of training data is 180K (line 124), [6.25%,12.5%,25%,50%,100%] correspond to 11K, 22K, 45K, 90K, 180K respectively.\n\n# Q4\n\n > In Table 6, continuous evolving performs worst in MiniCPMV2.5 FQA task, can the authors explain this result?\n\n> Also in Table 6, for MiniCPMV2.5, continous evolving with PRM + rerank … on TQA task. Meanwhile, for Phi3.5, continous evolving performs really bad on TQA task but PRM + rerank has a positive influence on this task. What’s the dynamic behind this observation?\n\nThank you for your thoughtful observations regarding the performance dynamics in Table 6. We address both points below in a unified explanation.\n\n- Overall, **M-STaR** consistently outperforms other variants, and significance testing (p < 0.046) supports the robustness of this improvement.\n- Compared to GPS and MWP, which demand more complex multimodal reasoning, tasks like TQA and FQA rely more heavily on perception abilities, such as visual understanding or interpreting structured text layouts. While reasoning is still required, the relative emphasis shifts more toward perception in these tasks.\n- Without dynamic monitoring, the self-evolving training process may saturate, over-optimizing for reasoning ability and neglecting perception skills. This leads to unstable or suboptimal performance on tasks like TQA and FQA.\n- As shown in Table 6, M-STaR effectively mitigates these issues. It adaptively adjusts the evolution process by introducing dynamic control to avoid training saturation. The improvements are especially significant when both the LLM and vision encoder have sufficient capacity, confirming M-STaR’s effectiveness across diverse subtasks.\n\n# Q5\n\n> Can the author explain the motivation to focus on the short iteration interval as mentioned in line 146-149 in section 3.2?\n\nThank you for your question.\nWe focus on short iteration intervals because our study explores optimal training design through the lens of RL. In the context of LLM training, several prior works [1–3] have shown that online training methods with appropriate iteration interval outperform offline ones. In our self-evolving training framework, a shorter iteration interval corresponds to an more online training regime, where the model can adapt more quickly to newly generated samples. Our findings are consistent with existing RL literature, which emphasizes that iteration intervals should be appropriately short—not too large (to avoid stale or offline updates), and not too small (to prevent excessive variance across iterations).\n\n[1] Direct Language Model Alignment from Online AI Feedback ICML2024\n\n[2] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\n\n[3] SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild", "claims_and_evidence": "Most claims made in the submission are supported by experimental evidence. \n\nLine 175-178  “...switching over the Improve and Generate steps too frequently makes the learning process unstable, leading to a lower score, especially on the in-domain test set.” lacks of evidence.", "methods_and_evaluation_criteria": "I believe that the methods and corresponding evaluation metrics in this paper are well-aligned and detailed.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I did check all the experimental designs and analyses.", "relation_to_broader_scientific_literature": "This paper provides practical insights for the multimodal reasoning community, particularly in the design of training methods and process reward models.", "essential_references_not_discussed": "The paper is self-contained.", "other_strengths_and_weaknesses": "- The paper is well-written and easy to follow.\n\n- This paper systematically presents the key factors for improving model performance.", "other_comments_or_suggestions": "1 Please re-organize the first paragraph in Section 3.2 to make sure that “iteration interval” is defined first before being used to avoid unnecessary confusion. \n\n2 “…we fix training methods as continuous self-evolving with 45k interval” Please specify which iteration interval the 45k interval refers to. [6.25%,12.5%,25%,50%,100%]", "questions_for_authors": "1 In Table 6, continuous evolving performs worst in MiniCPMV2.5 FQA task, can the authors explain this result?\n\n2 Also in Table 6, for MiniCPMV2.5, continous evolving with PRM + rerank negatively influence a lot on TQA task. Meanwhile, for Phi3.5, continous evolving performs really bad on TQA task but PRM + rerank has a positive influence on this task. What’s the dynamic behind this observation?\n\n3 Can the author explain the motivation to focus on the short iteration interval as mentioned in line 146-149 in section 3.2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBqJp0S6dk", "reviewer": "Reviewer_HhcC", "summary": "This paper investigates self-evolving training for multimodal reasoning through the lens of reinforcement learning, identifying three key factors: Training Method, Reward Model, and Prompt Variation. The authors propose a continuous self-evolving training scheme that inherits optimizer states between iterations, develop a multimodal Process Reward Model (PRM) for reranking responses, and analyze the impact of unlabeled data. They also examine training dynamics, identifying performance saturation as a key challenge, and propose an automatic balancing mechanism to adjust sampling temperature. These components are combined into a framework called M-STAR (Multimodal Self-evolving Training for Reasoning) which is evaluated across multiple multimodal reasoning benchmarks using different model sizes. The authors report consistent performance improvements, particularly on MathVista where their approach achieves a 6.7% absolute improvement over the pre-evolved model.\n\n## update after rebuttal\nThe authors address most of my questions, e.g. PRM as effective reranker, and statistical significances of results. I still have concerns about the effectiveness of the self-improving methods as the improvements are small in most benchmarks. I am still on the fence, slightly more positive. So I increase my rating. AC, please note that, I think the paper is borderline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reivew.\n\n# Q1\n\n> 1. ...shows that your PRM is…\n\n> 2. How... relate to recent findings in papers like DS-R1…\n\n## Q1.1 PRM Analysis\n\nFirst, we want to clarify an important distinction between BoN selection and reranking using PRM.  But due to the space limit, please refer to **Q1 in Response to Reviewer gU79** for details about this **clarification and analyses** below.\n\nTo better investigate and validate the behaviour of the PRM, we conducted several analyses, which help better understand the PRM and our findings.\n\n**Value Function Analysis**:\n\nPRM serves as a value estimator trained via Monte Carlo rollouts. We show our PRM is with a lower MSE for correct responses (0.081) than incorrect ones (0.124), supporting its strength in reranking valid reasoning paths over verifying incorrect ones.\n\n**Human Evaluation**:\n\nWe validate GPT-4o’s relativity score (Figure 2) via human annotation across 50 examples, showing ~84% agreement and confirming its reliability as an alignment measure.\n\n**Readability Analysis**:\n\nUsing the Flesch Reading Ease score, we find that top-2 reranked responses are more readable and less erratic, indicating PRM’s benefit in producing clearer, more coherent outputs.\n\n**Qualitative Case Study**:\n\n\nEven when final answers are correct, PRM prefers responses with more coherent reasoning, illustrating its advantage in identifying high-quality rationales beyond binary correctness.\n\n\n## Q1.2\nThanks for the insightful question! \n1. As discussed in **Q1.1**, our use of PRM plays a distinct role in our framework. First, our reward strategy also follows rule-based—we filter out responses with incorrect answers before using PRM. In this way, PRM is used more like a reranker, helping us select trajectories with the highest-quality reasoning steps that are also consistent with the original question.\n2. Second, the training setup differs significantly from R1. While in DeepSeek-R1’s discussion, it applies policy gradients at every step using PRM (as in GRPO), which can introduce reward hacking issues, our method is STaR-like and avoids this risk by not using step-wise policy gradients.\n3. While PRMs have higher ceilings, rule-based rewards are more robust and stable—one reason for their success in large-scale setups like R1. Still, we believe exploring PRMs further is valuable for pushing performance even higher.\n\n# Q2\n\n> The improvements reported…. Have you conducted statistical significance…? How do these improvements…?\n\n## Q2.1\n\nIn the context of self-evolving or RL-based methods on reasoning tasks, 3–6% absolute gains are substantial. Prior work such as RestEM reports similar improvements in the 1–6% range. Importantly, our benchmarks span diverse subtasks, and we observe much larger improvements on some of individual benchmarks, for instance, a notable 12% absolute gain on geometry problem-solving. These improvements are consistent across different model sizes and benchmarks.\n\nTo validate the statistical significance of our results, we perform t-test on the predictions, to assess whether the hypothesis that “MiniCPMV with M-STaR in Table 4 is better than Cont. Self-Evolving + PRM-Rerank on MathVista” is statistically significant or not. Our results demonstrate that the result is significant with a **p-value < 0.042**.\n\nRegarding stability, we observe a variance of around **0.04** across three independent runs of our static optimal strategy, which is acceptable given the computational constraints. Hence, we do not perform extensive repeated runs in later experiments due to expensive computation requirements.\n\n## Q2.2\n\n> The paper claims … in Table 1 are small (less than 3% in many cases, e.g. 57.2% vs 55.1%),...\n\nAs mentioned in **Q2.1**, continuous self-evolving training is one part of our overall framework and not expected to drive all performance gains alone. Similar to prior work (e.g. Tulu-2.5), changes to the training algorithm typically yield modest yet meaningful improvements in reasoning tasks. \n\nNonetheless, optimising each component is crucial to fully realising the potential of the complete M-STaR framework. We conduct a significance t-test specifically for cont self-evolving training and Iterative RFT in Table 1 on MathVista, which results in a **p-value < 0.02**—proving the significance. \nAnd the improvement across different models is also consistent according to Table 4.\n\n# Q3\n\n> How does your approach differ …? …novel technical…?\n\nWe will clarify the distinctions between our approach and prior STaR-like methods in the related work section. Briefly, each component of our framework introduces a new design aimed at improving and advancing beyond traditional STaR-like approaches—such as continuous self-evolving training, using PRM as a reranker, dynamic self-evolution, and their integration into a unified framework for multimodal reasoning. \nThese components did not exist in the mentioned previous works, and they lead to significant gains as we reported in the submission.", "claims_and_evidence": "The paper makes several claims that are not fully supported by convincing evidence:\n\n1. The authors claim their PRM is effective as a reranker despite not good as a verifier. While they show some analysis in Figure 2, the explanation lacks depth and rigor. The paper notes that PRM-selected responses have fewer reasoning steps and are more relevant to queries, but doesn't provide a compelling theoretical or empirical explanation for why this makes PRM effective in the self-evolving training context.\n\n2. The improvements reported on benchmarks are modest (mostly 1-6% absolute gains), and it's not clear how significant these improvements are statistically.\n\n3. The paper claims that continuous self-evolving training with proper intervals is better than traditional approaches, but the performance differences shown in Table 1 are small (less than 3% in many cases, e.g. 57.2% vs 55.1%), raising questions about the significance of this contribution.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are generally appropriate for the problem at hand. The authors use established multimodal reasoning benchmarks and conduct controlled experiments to isolate the impact of different components. However, there are some issues:\n\n1. The authors use GPT4-o to measure \"response relativity,\" but don't provide sufficient details about the reliability of this metric or potential biases in this evaluation approach.\n\n2. The paper makes a case of PRM. This seems to be opposite to the findings in the DeepSeek-R1 paper (though the paper is on text based reasoning tasks). Therefore, this deserves more in-depth evaluation.", "theoretical_claims": "The paper makes limited theoretical claims, primarily framing self-evolving training as a reinforcement learning problem. The formulation appears sound, but the paper doesn't develop this theoretical foundation into novel insights that significantly advance our understanding of self-evolving training.", "experimental_designs_or_analyses": "The experimental design has several issues:\n\n1. The analysis of why PRM works as a reranker despite being ineffective as a verifier is insufficient. This is a critical insight that could be valuable to the field, but the paper doesn't explore it deeply enough.\n\n2. The experiments on training dynamics are interesting but preliminary.", "relation_to_broader_scientific_literature": "The paper positions itself within the self-evolving training literature but doesn't clearly differentiate its contributions from prior work. The authors mention related approaches like STaR, ReST, and ReST^EM, but don't provide a detailed comparison of how their approach differs technically or conceptually.\n\nThe paper also doesn't adequately address recent findings that question the effectiveness of PRMs for reasoning tasks, such as those mentioned in the DeepSeek-R1 paper, which found that PRMs have limitations in guiding reasoning tasks due to challenges in defining fine-grained steps and assessing their correctness.", "essential_references_not_discussed": "ReFT: Reasoning with Reinforced Fine-Tuning\nTrung Quoc Luong, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin, Hang Li", "other_strengths_and_weaknesses": "Strengths:\n- The paper provides a systematic exploration of different components of self-evolving training for multimodal reasoning.\n- The introduction of continuous self-evolving training with inherited optimizer states is a potentially useful contribution.\n- The analysis of training dynamics and the proposed automatic balancing mechanism addresses an important challenge in self-evolving training.\n\nWeaknesses:\n- The paper lacks novelty in its core components. Most of the techniques have been explored in prior work, and the paper doesn't clearly articulate what is fundamentally new.\n- The improvements over baselines are modest, raising questions about the practical significance of the approach.\n- The most intriguing finding—that PRM can be effective as a reranker despite being ineffective as a verifier—is not explored deeply enough to provide meaningful insights.\n- The paper doesn't adequately address the limitations of PRMs for reasoning tasks that have been identified in recent literature.", "other_comments_or_suggestions": "- The paper would benefit from a clearer articulation of its novel contributions relative to prior work.\n\n- A more rigorous analysis of why PRM works as a reranker despite being ineffective as a verifier would significantly strengthen the paper.\n\n- The paper should address recent findings that question the effectiveness of PRMs for reasoning tasks and explain why their approach might overcome these limitations.", "questions_for_authors": "1. Your paper shows that your PRM is ineffective as a verifier but effective as a reranker. This is an intriguing finding that contradicts conventional wisdom about PRMs. Can you provide a more rigorous analysis or theoretical explanation for why this is the case? How does this relate to recent findings in papers like DeepSeek-R1 that question the effectiveness of PRMs for reasoning tasks?\n\n2. The improvements reported on benchmarks are relatively modest (mostly 1-6%). Have you conducted statistical significance tests to ensure these improvements are meaningful? How do these improvements compare to the variance observed across different training runs?\n\n3. How does your approach specifically differ from prior work like STaR, ReST, and ReST^EM beyond the application to multimodal reasoning? What are the novel technical contributions that distinguish your work?\n\nI am open to upgrade my rating if important questions are addressed or clarified satisfactorily.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an interesting reframing of self-evolving training for multimodal reasoning through a RL lens, identifying key factors such as training method, reward modeling, and prompt variation. The introduction of the first multimodal, process-based reward model is a notable contribution, demonstrating potential for performance enhancement. However, the evaluation lacks depth in exploring the identified factors through sufficient ablation studies and comparisons with existing state-of-the-art methods and benchmarks, particularly on more common LMM architectures. While the findings regarding the impact of unlabeled data based on reward signal quality are insightful, the overall contribution would be strengthened by more rigorous experimentation and a clearer articulation of the novel technical challenges addressed within the multimodal context.
Decision: Accept (poster)