First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training
Authors: Lai Wei
Yuting Li
Chen Wang
Yue Wang
Linghe Kong
Weiran Huang
Lichao Sun
Venue: NeurIPS 2025 poster ()
Keywords: Multi-modal Large Language Models
Multi-modal Reasoning
Self-Improvement
Unsupervised Post-Training
Primary area: deep_learning
TL;DR: We introduce an unsupervised method for post-training multi-modal large language models using implicit reward signals from majority voting based on GRPO.

Abstract: Improving Multi-modal Large Language Models (MLLMs) in the post-training stage typically relies on supervised fine-tuning (SFT) or reinforcement learning (RL), which require expensive and manually annotated multi-modal data--an ultimately unsustainable resource.
This limitation has motivated a growing interest in unsupervised paradigms as a third stage of post-training after SFT and RL.
While recent efforts have explored this direction, their methods are complex and difficult to iterate.
To address this, we propose MM-UPT, a simple yet effective framework for unsupervised post-training of MLLMs, enabling continual self-improvement without any external supervision.
The training method of MM-UPT builds upon GRPO, replacing traditional reward signals with a self-rewarding mechanism based on majority voting over multiple sampled responses. 
Our experiments demonstrate that such training method effectively improves the reasoning ability of Qwen2.5-VL-7B (e.g., 66.3\%$\rightarrow$72.9\% on MathVista, 62.9\%$\rightarrow$68.7\% on We-Math), using standard dataset without ground truth labels.
To further explore scalability, we extend our framework to a data self-generation setting, designing two strategies that prompt the MLLM to synthesize new training samples on its own.
Additional experiments show that combining these synthetic data with the unsupervised training method can also boost performance, highlighting a promising approach for scalable self-improvement.
Overall, MM-UPT offers a new paradigm for autonomous enhancement of MLLMs, serving as a critical third step after initial SFT and RL in the absence of external supervision.
Our code is available at \url{https://github.com/waltonfuture/MM-UPT}.

First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training
Lai 2e
Yuting Li
Chen Wang?
Yue Wang"
Linghe Kong'
Weiran Huang
Lichao Sun
School of Computer Science, Shanghai Jiao Tong University
Zhongguancun Academy
Shanghai Innovation Institute
Lehiigh University
Abstract
Improving Multi-modal Large Language Models (MLLMs) in the post-training stage typically relies supervised fine-tuning (SFT) reinforcement learning (RL), which require expensive and manually annotated multi-modal data-an ulti- mately unsustainable resource. This limitation has motivated growing interest in unsupervised paradigms as a third stage post-training after SFT and RL. While recent efforts have explored this direction, their methods are complex and difficult to iterate. To address this, we propose MM-UPT, a simple yet effective framework for unsupervised post-training of MLLMs, enabling continual self-immrov improvement without any external supervision. The training method of MM-UPT builds upon GRPO, replacing traditional reward signals with a self-rewarding mechanism based on majority voting over multiple sampled responses. Our experiments demonstrate that such training method effectively improves the reasoning ability of Qwen2.5- VL-7B (e.g., 66.3%-+72.9%% on Math Vista, 12.9%->6..777 on We-Math), using standard dataset without ground truth labels. To further explore scalability, we extend our framework to a data self-generation setting, designing two strategies that prompt the MILM to synthesize new training samples its tional experiments show that combining these synthetic data with the unsupervised training method can also boost performance, highlighting a promising approach for scalable self-improvement. Overall, MM-UPT offers new paradigm for au- tonomous enhancement of MLLMs, serving as critical third step after initial SFT and RL in the absence of external supervision. Our code is available at
Addi-
Introduction
Multi-modal Large Language Models (MLLMs) have achieved remarkable performance on variety of vision- lasiuallanguage tasks, ranging from image captioning to visual reasoning [22, 51, 67, 73, 81, 83]. The dominant paradigm for improving MI.LMss in the post-training stage typically involves supervised fine-tuning (SFT) and reinforcement learning (RI. [2, 42, 47, 59, 65, 77]. However, both SFT and RL. rely on large volumes of high-quality and annotated multi-modal data, such as image captions, visual reasoning traces, verifiable ground truth answers, and human preference signals. As real-world tasks
'Email: waltonfuture sjttttuurr@jjjj.t.... @eje@essssu.... edu.en 'Correspondence to Weiran Huang (weiran @ @outlook.comm..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Estimate Advantage, Update Policy
0₁
112
O2
F'2
Majority Voting
MLLM
Find the perimetor of the Siltringje Siltfiangia A film Vittangie B D, A De55, SC D- 12.1 and $8 C-31 25.
Og
Dg
Data without Labels
Responses
Rewards
Overview of the MM-UPT framework. Given unlabeled multi-modal input, the MLLM samples multiple responses, and uses majority voting to determine the pseudo-label. The MLLM then updated via GRPO, enabling self-improvement without external supervision.
Figure I:
grow in complexity and quantity, critical challenge emerges: curating and annotating high-quality data at scale becomes increasingly impractical. To overcome this data-dependency. paradigm shift required towards third stage post-training beyond SFT and RL, dedicated to the continual self- self-imppovemment of MLLMs through synthetic and unlabeled data. We formalize this third-stage paradigm as Unsupervised Post- Trainnrin....
Previous works have studied the use of MLLMs themselves to generate synthetic instruction data for seff-improvement through offline training techniques like SFT and DPO [6, 23, 39-41, 64, 74]. These approaches typically involve complex pipelines with multiple stages, such as data generation, verification, and filtering, which are hard to iterate online. Fortunately, recent studies demonstrate notable success using online reinforcement learning (e.g. GRPO [37]) with verifiable rewards to enhance the reasoning capabilities MLLMs 31, 53]. A concurrent work, TTRL [84], further extends this line by applying GRPO on test-time scaling of LLMs. It is promising that online RL enables models to continuously improve, thus acquiring novel reasoning abilitics that exceed corresponding base models' capacity.
Motivated by these insights, we propose MM-UPT (Multi-Modal Unsupervised Fost-Training) an easy-to- to--impeemment framework for unsupervised post-training in MLLMs. As illustrated in Figure 1, our approach adapts the online reinforeement learning method GRPO [37], which is known for its stability and scalability. The core challenge in applying GRPO to unsupervised setting is the absence ground-truth labels for reward calculation. To overcome this MM-UPT works by deriving implicit reward signals via majority voting over multiple sampled responses. In particular, majority voting aggregates multiple responses and selects the most frequent one, which has been widely used and shown effective to improve model performance [44, 49, 84]. Thus, adopt the majority answer to serves as a dynamic pscudo-laber in GRPO: responses that align with this consensus receive positive reward, while divergent ones are penalized. This process effectively encourages the model to bootstrap its own tigh-confidence knowledge, promoting the generation stable and consistent answers without relying on any external supervision reward models.
Beyond learning from existing unlabeled data, MM-UPT further extends to a data self- self-generation setting that enhances scalability under the unsupervised paradigm. Specifically, we design two strategies that allow the MI.LM itself to synthesize new training samples: (1) in-context synthesizing, where the model generates new questions conditioned original examples (image, question, and answer); and (2) direct synthesizing. where the model generates questions solcly from the given image. These self- aeff-gneratee samples are then used within the same unsupervised reinforcement learning method, enabling continual self-improvement even in the absence of human-created questions.
In our experiments, we focus on the domain of multi-modal reasoning, which is widely focused and inherently challenging. We explore two key scenarios for constructing unlabeled data assuming that labels are not available: (1) using human-created questions without ground-trutt labels, and (2) employing synthetic questions generated by the model itself, inherently lacking ground-truth labels. This setup allows us to examine both the effectiveness of unsupervised training existing data and self-generated data in enhancing reasoning performance. We evaluate MM-UPT across range of reasoning benchmarks and observe notable performance improvements over the base models

---

(e.g,, 67.3% on MathVista, 62.9%~68..7% on We-Math using Qwen2.5-VL-7B [T] in the first scenario. Our method also outperfonms previous baseline methods, and is even competitive with supervised GRPO, underscoring the effectiveness of MM-UPT as a self-improving training strategy. As for the second scenario, we find that models trained on unlabeled synthetic data achieve performance competitive with those trained on the original unlabeled dataset, revealing a viable path for scalable self-improvement. Additionally, our deeper analysis reveals a clear trade-off MM-UPT: it improves accuracy by reinforcing high-confidence knowledge but reduces response diversity and requires sufficient initial competence to prevent error amplification.
Our main contributions are summarized as follows:
We are the first to formalize three-stage post-training paradigm for MLLMs, with Unsupervised Post- TasinTrainnn (UPT) as the critical third stage enabling continual improvement without external supervision. We instantiate this stage with MM-UPT, a simple and effective framework using majority voting as a self-rewarding signal in online reinforcement learning.
Extensive experiments on multi-modal reasoning tasks demonstrate the effectiveness of majority voting as a pseudo-reward estimation for unsupervised training.
We extend MM-UPT to utilize synthetic data generated by the MLLM itself, and find that training the MLLM on such data leads to notable performance gains. This reveals promising path toward efficient and scalable self-improvement in unsupervised post-training.
2 Related Works
Self Improvement. High-quality data obtained from human annotations has been shown to sig- nificantly boost the performance of LLMs across wide range of tasks [12, 18, 32]. However, such high-quality annotated data may be exhausted in the future. This presents a substantial obstacle the continual learning of advanced models. As result, recent research has shifted toward eeff-improvement, leveraging data generated by the LLM itself without any external supervi- sion [8, 16. 29, 54, 84]. Several following works also explore self-improvement in the multi-modal domain [6, 11, 39, 64, 72, 74]. Genixer [74] firstly introduces self-improvement pipeline including complex data generation and filtering for SFT. STIC [6] and SENA [40] construct preference data pairs for DPO [36] in a self-supervised manner, focusing enhancing perceptual capabilities. In contrast these approaches which are complex and hard scale, the key distinction that our work leverages online reinforcement learning using GRPO [12] with simple yet effective data synthesizing strategies at the post-training stage, which can be more scalable for self cef--improvement without reliance on any external supervision. In addition, none of these previous methods focus multi-modal reasoning tasks, which are considered challenging for current models.
Multi-modal Reasoning, Recently, the reasoning abilities of MLLMs have become a central fo- cus of research [27, 53, 56, 82]. In contrast traditional LLM-based reasoning [12, 28, 62] that primarily relies on text, multi-modal approaches must both process and interpret visual inputs, significantly increasing the complexity of tasks such as geometric problem-solving and chart inter- pretation [3, 30, 75]. Several works in this field have sought to collect or synthesize a large scale of muuliimodaa reasoning data [5, 33, 38, 70]. Notably, the recent emergence of ol-like reasoning models [19] represents an initial step toward activating the slow-thinking capabilities of MLLMs, as demonstrated by several SFT-based methods, such as I.L.VA-Col [55], MAmmoTH- [13), and Mulberry [60]. Moreover, some concurrent works have further explored reinforcement learning approaches, particularly GRPO [37], in the post-training stage of MILMSs to enhance performance on multi-modal reasoning tasks [7. 31, 34, 48, 78]. While these supervised post-training methods have demonstrated promising results, our work explores a different direction by focusing on totally unsupervised post-training of MLLMs to self-improve the reasoning abilities.
3
The Framework of Multi-Modal Unsupervised Post-Traie Post-Training
Existing post-training techniques for MLLMs, such as supervised fine-tuning (SFT) and reinforcement learning (RLHF or RLVR), rely heavily on labeled data or external reward models. While these approaches have proven effective, their reliance on external supervision makes continual improvement unsustainable due to the high cost and limited scalability of manual annotation. To overcome this limitation, we formalize new third-stage post-training paradigm that enables the model to self-
3

---

improve without access to any external supervision. such as ground-truie truth labels or additional reward models. We instantiate this paradigm with MM-UPT Multi-Modie Unsupervised Post- fost-Training). simple yet effective framework designed to operate purely on unlabeled multi-modal data. The overview of our complete framework is shown in Figure I.
3.1 Problem Formulation
Firstly, we formulate the problem of unsupervised post-training for MLLMs as follows: Given a well-trained multi-modal LLM Te and collection of unlabeled multi-modal data Q { ())) where 1 represents an image and qi denotes a corresponding question, our goal is to improve the model'ss 's performance without access to any ground-truth answers external supervision signals. This setting differs significantly from conventional supervised fine-tuning (SFT), reinforcement learning with verifiable rewards (RLVR), reinforcement learning with human feedback (RLHF), which typically rely on labeled data (Tz, qi, Yi) or human preference data (7,, li y Vi ). where yi denotes the answer of and (y," y/ denotes the preference pair of q;- In contrast, we only allow to operate in fully unsupervised manner for this setting, leveraging only the model's own responses to generate training signals. This presents significant challenges, as the model must learn to assess and improve its own outputs without any external guidance.
3.2 Training Method
To achieve the unsupervised training. MM-UPT introduces a self-rewarding mechanism using majority voting as pseudo-labes [49] based on the online reinforcement learning. In particular, MM-UPT is built upon the GRPO algorithm [37]. which is widely used in the post-training stage multi-moda LLMs. GRPO optimizes computational efficiency by eliminating the need for separate value model; instead, directly utilizes group-normalized rewards to estimate advantages. Specifically, for question q and the correlated image / from the training dataset Q, GRPO samples group of responses O { from the old policy Vold and then optimizes the policy model by maximizing the following objective:
[(0) 11 on
(04))))) (Olg.1)
G
lou
(,,(0) (l) -,, +c) Ai. DD [Tal (aelTtees }
min
G
where (0) Teaa (g.ssg. Tref represents the reference model, and the term DEL introduces a KL divergence constraint to limit how much the model can deviate from this reference. The advantage estimate A; measures how much better the response o, is compared to the average response, which is computed using group of rewards fr.,,,...., for the responses in set O: A; "i-mean[[e)a) mot sall(rr:
moloct (g.0u)e)))
In the above standard GRPO formulation [12], the reward is computed in a supervised manner based on labels for each response in O {o: _ Shifting towards unsupervised setting, where no ground-truth labels are available, one feasible way is to construct pseudo-labels to calculate the reward for GRPO. Motivated by [16, 49, 84], we use majority voting over the group of sampled responses O to serve as pscudo-labels. Majority voting selects the most frequent answer among the sampled responses O and has proven to be simple yet effective technique [49, 84], making it suitable for deriving good pseudo-rewart signals. Specifically, we first extract answers from the responses
O = for G using an rule-based answer extractor [15] E(.), resulting in Ý = E(O) = {0.).- Then, the majority-voted answer y" can be obtained by:
y" arg max yEÝ
Σ [[y = 0.. 1=1
(1)
where I[[]] is the indicator function. The reward r, is then determined based on the y":
11
if f/ y
r
0, otherwise.
(2)

---

Algorithm The training method of MM-UPT
1: Input: Current policy тa, old policy Re. unlabeled training dataset Q, Group size G, reference model Wref, clip parameter e, KL penalty coefficient 5, answer extractor E(-).
2: for cach sample (7, q) Q do
3: Sample group of responses: )m1 ~ Tea (o 1,9);
I/ Sample multiple responses
4: Extract answers: y = E(() = N=:;
5: Determine majority vote: y" arg max yax Σ Tly Mt]:
If Select the most frequent answer
6: Compute pseudo-rewards: Ti < Tß: y y']:
// Reward based majority agreement
Compute advantage estimates: A. + ri monn mou roala
7:
ad ad((-1)2---a)))
8:
Compute GRPO objective:
9:
(0) + à - 1o.] Elal min (n,(0)A,, clip (01 -, +) A.] -Dxx
[Dxx[Tool [Tol|
10:
where (B) Tesa (.....)))))
11:
Update policy parameters: 0 + VoJoRPo(0):
12:
Update old policy: Oata + 0:
13:
end for
14:
return T
In this way, compute pseudo-rewards via majority voting and apply standard GRPO update the MILM. This majority- reward encourages the model to converge toward consistent, high- consensus responses, thereby enabling the model further exploit its existing self-knowledge leveraging unlabeled data. The pipeline of our training method in MM-UPT is shown in Algorithm 1.
3.3 Synthetic Data
To further explore the scalability of unsupervised post-training, we extend our framework to data self-generation setting, where the model asked to synthesize new training samples on its own. In particular, we design two simple yet effective data synthesizing strategies as follows.
In-Context Synthesizing. Inspired by Self-Instruc [50], we construct a data generation pipeline that leverages in-context examples to guide the synthesis process, Each original example consists of image, a question, and its corresponding answer. To synthesize new samples, we provide the model with the full triplet and instruct it to generate a new question that is semantically distinct from the original but relevant to the same image. This strategy helps generate task-relevant and meaningful variations of the original question, as well as ensure the quality of synthetic questions. During unsupervised post-training, the model then attempts to answer each newly generated question, and pseudo-labels are derived from the majority vote among its sampled responses, consistent with the mechanism described in Section 3.
Direct Synthesizing. In addition to in-context generation, we also explore a direct synthesizing strategy that further increases diversity. Here, the model receives only the image and is prompted to freely create a new question without any reference to the original question. This open-ended formulation encourages the model generate wider range of diverse and novel questions based solely on the visual input, rather than being constrained by the original task. Similar to the in-context setup, we perform unsupervised post- post-training on these synthetic samples using majority voting to define pewudo-sewards.
This setup enables the MILM expand the training corpus without any human annotations, thus achieving fully autonomous seff-improvement loop.
Experiments
We conduct extensive experiments to evaluate the effectiveness MM-UPT across various multi- modal LLMs, datasets, and benchmarks. Our experiments are designed to explore two key scenarios: (1) using human-created questions without ground-truth labels (Section 4.2), and (2) employing synthetic questions generated by the model itself, inherently lacking ground-truth labels (Section 4.3). Before presenting the experimental results, we first outline the baseline methods, evaluation bench- marks, and implementation details in the experimental setup as follows.
5

---

Table 1: Main results of Scenario 1 on four multi- modal mathematical reasoning benchmarks. We report accuracy (%) for each method on Math Vision. Math Verse, Math Vista, and We-Math. All methods are conducted on the Qwen2.5-VL-77h backbone. MM-UPT outperforms other baseline methods, and is even competitive with supervised methods.
Model and Methods
Unsupervised?
Training Data
Math Vision Matheerse MathVista
We-Math
Avg
Swen2.5-V1.-7h
24.87
43.83
66.30
62.87
49.47
GRPO [37]
Geometry3K
28.32
46.40
69.30
68.85
53.22
GRPO [37]
GeoQA
26.15
46.28
67.50
66.65
51.65
GRPO [37]
MMRI
29.01
45.03
71.40
67.24
53.17
SFT [43]
Geometry3K
25.92
43.73
67.90
64.94
50.63
SET [43]
GooQA
25.72
44.70
67.40
65.10
50.73
SFT [43]
MMR1
26.45
43.53
63.30
64.20
49.37
SRLM [54]
Geometry3K
26.94
44.54
66.90
66.32
51.18
SRLM [54]
GeoQA
25.16
44.62
66,30
65.00
50.27
SRLM [54]
MMRI
25.33
45.08
67.00
64.66
50.52
MSS [16]
GeometryäK
25.10
43.96
65.50
64.43
49.75
LMSI [16]
GeoQA
25.49
43.50
66.60
49.78
+ LMSI [16]
MMRI
24.83
43.76
64,90
66.38
49.97
Genlxer [74]
Geometry3K
26.02
43.15
65,50
62.18
49.22
Genixer [74]
GeoQA
25.30
44.11
66.80
64.25
50.12
Genixer [74]
MMRI
23.68
43.30
65.50
64.66
49.29
STIC [6]
Cieometry3K
25.39
42.92
65.20
62.99
49.13
STIC [6]
GeoQA
23.49
42.87
64.30
63.62
48.57
STIC [6]
MMRI
23.78
42.72
66.10
63.74
49.09
.M-UPP (Ours)
GeometryäK
27.33
42.46
68.50
51,23
+ MMM-UP (Ours)
GeoQA
27.07
43.68
68.90
68.22
51.97
-MM-UPT (Ours)
MMRI
26.15
44.87
72.90
68.74
53.17
4.1 Experimental Setup
Baseline Methods. Several prior works have explored self-improvement in both LLMs and MLLMs. Note that we focus on unsupervised self- eelf-improvement, we do not compare with methods that rely on external models (e.g., GPT-4o [18]) for supervision [17, 64, 71, 79, 80]. Instead, we compare with several totally unsupervised methods: LMSI [16], SRLM [54], Genixer [74], and STIC [6]. In particular, LMSI corresponds to supervised fine-tuning with self-generated content selected by majority voting. SRIM uses the model itself as L--M-as-a-Judge [76] to provide its own rewards during DPO [36] training. Genixer prompts the MLLM first self-generate self-check it. STIC applies DPO where original images and good prompts are used to generate preferred answers, and corrupted images and bad prompts to produce rejected answers. Additionally, we also compare with GRPO [37] and rejection SFT [43], which are two strong supervised methods. The details of these baseline methods shown in Appendix A.I.
answer and then
Benchmarks. We evaluate our method on four popular multi-mode mathematical reasoning bench- marks: Math [45], Math Vista [27], Verse [69], and We-Math [35]. These benchmarks offer comprehensive evaluations with diverse problem types, including geometry, charts, and ta- bles, featuring multi-subject and meticulously categorized visual math challenges across various knowledge concepts and granularity levels. We provide more details in Appendix A.2.
Implementation Details. We adopt the EasyRI [61] framework for multi-modal unsupervised post-training, which is based on GRPO. Specifically, we the training episodes to 15, and use AdamW optimizer [24] with learning rate of 1 x 10-6, weight decay of 1 x 10-2, and gradient clipping maximum norm of 1.0. The KL divergence constraint stabilize the training. The vision tower of the multi modal model is also tuned without freezing. In our training, we use a rollout temperature 0.7, which strikes a good balance: lower temperatures produce low-diversity outputs, while higher temperatures often lead to lower-quality outputs. Within each episode, we perform one rollout group (G=10) per data point. Other hyperparameters follow the default settings provided in the E framework.
GRPO set to 0.01 to

---

Table Performance comparison of MM-UPT using different synthetic data generation strategies in Scenario 2. Both "In- Context Synthesizing" and "Direct Synthesizing" approaches yield significant improvements over the base model and perform competitively with the "Original Questions" on average, demonstrating the effectiveness of synthetic data for unsupervised self-improvement
Model and Methods
Datasct Math
MathVerse
Math
We-Math
Avg
Qwen2,5-VL-7B
24.87
43.83
66.30
62.87
49.47
w/ Original Questions
Gco3K
27.33
42.46
68.50
66.61
51.23 (3.6%))
w/ In- In-Context Synthesizing
Geo3K
26.71
41.24
68.30
67.76
51.00 (3.1
w/ Direct Synthesizing
Gen3K
26.88
43.53
69.90
68.97
52.32 (5.8%
w/ Original Questions
GeoQA
27.07
43.68
68.90
68.22
51.97 (5.1%))
w/ In- In-Context Synthesizing
GeoQA
26.09
42.87
70.60
52.20 (5.5%)
w/ Direct Synthesizing
GenQA
26.25
44.64
71,50
68.28
52.67 (6.5%%)
w/ Original Questions
MMRI
26.15
44.87
72,90
68.74
53.17 (7.5%)
w/ In- In-Context Synthesizing
MMRI
26.15
45.10
71.90
68.62
52.94 (7.0%1)
w/ Direct Syathesizing
MMRI
26.15
44.11
70.40
67.99
52.16 (5.4%))
4.2 Scenario 1: Unsupervised Training on Standard Datasets
For our experiments, we firstly employ standard training datasets with masked labels to simulate the first scenario (i.e., using human-createe questions without ground-truth answers). We conduct MM-UPT on Geometry3k [25]. GeoQA [4], and MMRI [20] using the Qwen2.5-VL-7B [1] model. These datasets cover a diverse set of visual math problems, including geometric diagrams, charts, and structured question formats muuttppe-choocc and fill-i-thhe-blank), serving strong foundation for models to self-improve the mudti-modal mathematical reasoning abilities. More details of these datasets are introduced in Appendix A.3.
Table presents the main results on four challenging multi-modal mathematical reasoning benchmarks. We observe that MM-UPT achieves consistent improvements in average over the base Qwen2.5- VI.-BB model across all datasets, also outperforming other baseline methods such as SRLM, LMSI, Genixer, and STIC. Notably, MM-UPT is able to improve the average score from 49.47 (base model) to 53.17 (with MMR dataset), demonstrating its effectivenesss in leveraging unlabeled data for self-improvement. In comparison, previous baselines provide only marginal gains even degrade performance on certain benchmarks, highlighting the limitations existing methods when applied to already strong models in multi-mode multi-modal reasoning tasks. Furthermore, UPT even competitive with supervised post- pest-training methods, such as rejection sampling based SFT [43] and GRPO [37]. These results underscore the potential of MM-UPT to further exploit the knowledge embedded in multi-modal models for self-imprrvemmen. Further experiments on the training dynamics, generalization and adaptability of our method are shown in Appendix B.
find that MM-
4.3 Scenario 2: Unsupervised Training on Synthetic Datasets
To further explore the potential of MM-UPT, we investigate the use of unlabeled synthetic data (mentioned Section 3.3) to improve MILMs. This aligns with the ultimate goal of MM-UPT: enabling continual self-improvement even after human-created data is exhausted.
In our experiment, we use the previous two methods in Section 3.3 to generate the synthetic data, leveraging Geometry3K 3K [25], GeoQA [4], and MMRI [20] as the seed datasets, and Qwen2.5-VL-7B as the base MLLM for data synthesis. MM-UPT then applied the same base model (i.e.. Qwen2.5- VL-7B) using each of these different synthetic datasets separately. Table 2 presents experimental results using different synthetic data generation strategies. Both in-context and direct synthesizing lead to significant improvements over the base model, achieving performance comparable to training on original human-written questions. This shows that synthetic questions can effectively enhance model's is reasoning ability under MM-UPT, Notably, direct synthesizing even surpasses numan--ritten questions (when applied to Geometry3K and GeoQA) on average, demonstrating the strong ability of the model to generate high-quality textual questions solely based images. This highlights the
7

---

potential for scalable and fully autonomous self self-inprovement in multi-modes domain via visual- centric data synthesis.
Moreover, we manually examine some synthetic data. We observe that in-context synthesizing often produces questions similar the original ones by substituting conditions expressions, resembling data rephrasing. In contrast, direct synthesizing generates more diverse and novel questions. While some the directly synthesized questions still contain hallucinations, many are of high quality and beneficial for unsupervised post-training. This underscores the potential of the direct synthesizing approach as a simple yet effective method for data generation, without the need for textual in-context examples. Below, we present two illustrative examples that showcase the effectiveness and quality of synthetic questions generated through both approaches.
Demo: Examples of synthetic data using different strategies.
Original Question: BCI DF: AB
1, C.F
In-Contest Synthetie Question: Given that BCI DF, AB
BD AC + 7, 7nd CP = 24. Find the value
Directly Synthetic Question: In the given trinngle AAD,, point B lies on A.D and point C7 lies A.,. BC D.,, what the ratio of the ABBC
&ADF?
Original Question:
If ZRST is right sngle, SU RT. UV 1. ST, and LKTS 47", LRSU
In Contest Synthetic Question:
If LRST is right ingle. SU RT, UV L ST, T and LRTS 47% find LVST.
Directly Synthetic Question:
In tringll RST.. lies RT such SU
RT. such
UV is perpendieular to
wniss, units, SV units, the length VT:
4.4 Ablation Study
To evaluate the generality and effectiveness of MM-UPT, conduct ablation study across
range backbone models beyond the primary Qwen2.5-VL-7B [1]. Specifically, we apply MM-UPT to several state-of-the-arr models of varying scales, including
[1], MM-Eureka- 7B [31], and ThinkLite- VL-7B [48]. All models are post-trained using MM-UPT on the Geometry3K dataset [25], without access to any labels (i.e., Scenario 1). As summarized in Table 3, MM-UPT consistently improves the performance of all tested models on average, despite the absence of supervision during post-training. Notably, ThinkLite-VL--77 with MM-UPT achieves the highest average score (54.07), and shows substantial gains on the Math Vista [27] benchmark, reaching a score of 74.70. In addition, Qwen2.5-VL-3BB, the smallest model in study, also benefits well from MM-UPT (+7.4% on average), demonstrating the robustness and adaptability of MM-UPT for performance enhancement. These results collectively reveal that MM-UPT can be easily applied to various multi-modal models to enable consistent self-improvement.
Moreover, results also show that MM-UPT is compatible with supervised GRPO. For instance, MM-Eureka-7B was already tuned with supervised GRPO on the K12 dataset, yet applying MM-UPT unlabeled dataset (Geometry 3K) further improved its average score from 53.10 to 53.78 (Table 3). Similarly, ThinkLite- VL-7B was trained with supervised GRPO on the ThinkLite dataset, and applying MM-UPT again boosted its average Math Vista (74.70). These results highlight the practical value of MM-UPT as lightweight refinement step that remains effective for models already optimized with supervised GRPO, enabling them to leverage new unlabeled data for further improvement.
54.07, including substantial gain
Deeper Analysis
Going beyond standard benchmarking, conduct a deeper analysis investigate MM-UPT's performance boundaries (Section 5.1 and Section 5.2) and tradeoffs (Section 5.3). This helps better understand its behavior and potential applications.

---

Table Ablation study using different models besides Qwen2.5-VL-7B. We conduct this experiment on Geometry3K [25] dataset without labels.
Models
Math Vision
MathVerse
Math
We-Math
Avg
Qwen2.5-VL VL-7BB
24.87
43.83
66.30
62.87
49.47
Qwen2.5-VL-7B MM-UPT
27.33
42.46
68.50
66.61
51.23 (3.6% 1)
MM--urcka-7B
28.06
50.46
69.40
64.48
53.10
MM-Lurcka-7BB MM-UPT
28.95
50.63
69.10
66.44
53.78 (1.3%)
ThinkL.ite-Ve VL.-71
26.94
46.58
69.00
67.99
52.63
ThinkT -te- V1,7B + MM-UPT
26.91
47.26
74.70
67.41
54.07 (2.8% 1)
Dwen2.5-VI.-3B
19.47
33.58
56.30
50.63
39.00
Owen2.5-V1--BB MM-UPT
22.17
32,39
57.10
55.22
41.72 (7.4%
Table 4: Performance of MM-UPT the difficult ThinkLite- 11K dataset. Results show that MM- UPT leads to a decrease in performance when applied to a dataset where the model has limited prior knowledge, highlighting the limitations of majority voting in such scenarios.
Models
Training Data
Vision Math Verse
MathVista
We-Math
Avg
Qwen2.5-VL-77
24.87
43.83
66.30
62.87
Qwen2.5 VL-7B MM-UPT
ThinkLite- 11K
21.12
37.10
59.20
59.02
44.11
5.1 Why Does MM-UPT Work?
Majority voting [49] fundamental ensemble technique that enhances prediction reliability by aggregating multiple independent responses. In our framework, it offers simple yet powerful pseudo-reward signal to help model self-improve, particularly when the model are moderately reliable on the unlabeled datasets. We consider simplified explanation for it using a classical toy example. Suppose that each response hits the correct answer with probability p > 0.5 in a binary question. Then, we sample the model's response times independently. The final answer is determined by a majority vote, that is, the answer that appears more than n/2 times. Let X denote the number of correct predictions among the samples. Since each prediction is correct with probability p, X follows binomial distribution: X ~ Binomiaa(n,, p). The majority vote correct if X >> n/2, and the corresponding probability of this event (denoted F) is:
P(E) = Σ D (1--))) -p))
i= [=//2]]
When p 0.5, it follows that P(E) p. which means that the ensemble outperforms each individual response. For instance. if p = 0.7 and 10. then P(E) so 0.849, demonstrating significant gain over the base accuracy. This analysis reveals the rationality of majority voting serve as the pseudo- label for deriving reliable reward signal in the unsupervised setting. In our experimental setting, we mainly target datasets that are not especially hard, such as Geometry3k [25], GeoQA [4], and MMR1 [20], for unsupervised post-training. Hence, hypothesize that the model has relatively high chance of answering questions in these datasets correctly. This allows the model to yield stable improvements through MM-UPT using majority voting as the pseudo-label.
5.2 When Might MM-UPT Fail?
According the analysis in Section I, it reveals that the effectiveness of MM-UPT diminishes when the model lacks sufficient prior knowledge of the target dataset. To show that, we apply MM-UPT to ThinkLite- [48] dataset using Qwen2.5-VL. V-- [1]. ThinkLite- 11K is collected via difficulty- aware sampling that only retains samples that the model rarely answers correctly. Thus, this setting reflects scenario where the model is more likely to be wrong than right. such cases, majority voting amplifies incorrect answers rather than filtering them, leading to degraded performance. As shown in Table 4, applying MM-UPT to ThinkL ite- 1KK results in a significant drop accuracy across all benchmarks. This suggests that majority voting fails to provide reliable reward signals when

---

Table 5: Comparison of pass@10 across different benchmarks. Results show that MM-UPT improves single-response accuracy (pass@1, Table 1) but reduces response diversity, leading to a drop in pass@10. Supervised GRPO alleviates this issue to some extent.
Model
MathVision
Math Verse
Math
We-Math
Avg
Qwen2.5-VL-7B
0.6556
0.7307
0.8730
0.9420
0.8003
Qwen2,5-VL-TE MM-UPT
0,5661
0.6477 0.6726
0,8240
0.8621
0.7250
Qwen2.5-VL-7B: Supervised GRPO
0.6164
0.8570
0.9075
0.7634
the model has limited prior understanding of the domain. To address this issue, alternative forms of algorithms using more fine- tne-grained and complex rewarding methods, such LLM-as-Judhh [54, 76] and model collaboration [8. 21], may be necessary. Note that our work represents an initial attempt at self-imp self-improvement in MLLMs via GRPO, and we believe that these algorithms complementary to our approach and could be integrated into our framework in the future.
5.3 Trade-Offs in MM-UPT
Although MM-UPT improves passê1 accuracy across multiple benchmarks, we also observe a consistent decrease in passēn (for large n, e.g., pass@10) performance in Table 5. This reflects a common trade-off between accuracy and diversity in reinforcement learning for multi-modal models. Similar findings have been reported in supervised GRPO [66] (also shown in Table 5), where models tend to collapse onto high-consensus reasoning patterns, thereby reducing output diversity. In MM-UPT, this effect is further amplified because the implicit reward signal from majority voting encourages the model to favor dominant responses, which may suppress minority answers that are occasionally correct. That is said, MM-UPT let the model reinforce problems it can already solve, thereby forgoing the opportunity to tackle more challenging ones and abandoning exploration. While pass@1 is typically the most relevant metric in real-world applications, mitigating the decline of passon remains an important open problem for future work.
Another key consideration is the trade-off between training and inference costs. A straightforward alternative to MM-UPT is to apply majority voting directly at inference time, which can also boost accuracy by aggregating multiple outputs. However, inference-tine ensembling incurs substantial computational overhead, as n samples must be generated per query, making it expensive and often impractical scale. By contrast, MM-UPT shifts this cost into one-time training stage: after refinement, the model produces stronger single-pass outputs, avoiding the need for repeated sampling during deployment. This distinction highlights different usage scenarios: MM-UPT is particularly beneficial when inference efficiency and scalability are critical, whereas inference-time ensembling may be preferable when training resources are limited.
In summary, MM-UPT offers simple unsupervised post-training framework but also entails clear trade-offs. Balancing accuracy and diversity, and choosing between training-time and inference-time costs, are crucial aspects to consider and represent promising directions for future exploration.
6 Conclusion
In this work, we formalize a third-stage post-training paradigm for multi-modal large language models after SFT and RI, termed Unsupervised Post-Training (UPT). We instantiate this paradigm through MM-UPT, a simple yet effective framework that leverages majority voting as a self-rewarding mechanism within the GRPO algorithm, guiding models toward consistent and high-confidence responses on multi-modal reasoning tasks. This can be used to further exploit the model's 's internal knowledge by reinforcing consistent predictions, and thus enables celf-improvement without any external supervision. Extensive experiments across multiple benchmarks demonstrate that this method effectively enhances the reasoning performance of strong MLLMs without relying on labeled data or external reward models. Furthermore, we extend the framework to data self-generaiie generation setting, showing that synthetic questions produced by the model itself can further boost performance, revealing a scalable path toward autonomous seff-improvement. Future work could explore other fine-grained methods to provide pseudo-reward signals based on our framework, and investigate the scaling laws of unsupervised post-training using synthetic data.
10

---

Acknowledgements and Disclosure of Funding
This project is supported by the National Natural Science Foundation of China (No. 62406192), Open- ing Project of the State Key Laboratory of General Artificial Intelligence (No. SKLAGI20240P12), Tencent WeChat Rhino-Binn hiro- Focused Research Program, and Doubao LLM Fund.
References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[2] Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip Yu, and Lichao Sun. A survey of ai-generated content (aige). ACM Computing Surveys, 1-(5);1-38, 2025.
[3] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. In Findings of the Association Computational Linguistics: ACL-IJCNLP 2021,
pages 513-523, 2021.
[4] Liang Chen, Lei Li. Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generaliza- tion ability in vision- Laaounl models with less than $3. https: http::///gtttuu com www/Deep Deep-Agent/ R1-V V, 2025, Accessed: 2025-02-02.
[5] Kanzhi Cheng. Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. Vision- language models can self-improve reasoning via reflection. arXiv preprint 2rXiv:241 2024.
[6] Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, Quanquan Gu, James Y Zou, Kai-Wei Chang, and Wei Wang, Enhancing large vision language models with self-training on image comprehension. Advances in Neural Information Processing Systems, 37:131769-131397, 2024.
[7] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement. arXiv preprint arXiv:2503.17352, 2025.
[8] Qingxiu Dong. Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. Self-boosting large language models with synthetic preference data. arXiv preprint arXiv:2410.0696,, 2024.
[9] Bofei Gao, Feifan Song. Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07900 2024.
[10] Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312. 1370, 2023.
[11] Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, al. Infinity-mm: Scaling multimodal performance with scage and high-quality instruction data. arXiv preprint arXiv:2410. 18558, 2024.
[12] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.129448 2025.
[13] Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang. King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vi. Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.052377 2024.
[14] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint axXiv:2103.0387. 2021.

---

[15]
hiyouga. Mathruler. httpt//.....gggggggoo.o.oo/////a 2025.
com/hiyyogg//Machuuler
[16] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei
Han. Large language models can self-improve. arXiv preprint arXiv:2210. 2022.
[17] Xin Huang, Jing Bai, Yeqing Shen, Jia Wang, Zheng Ge, and Osamu Yoshie. Seeking the right question: Towards high-quality visual instruction generation. openreview, 2024.
[18] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
[19] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672) 2024.
[20] Sicong Leng. Jing Wang, Jiaxi Li, Hao Zhang, Zhiqiang Hu, Boqiang Zhang, Hang Zhang, Yuming Jiang, Xin Li, Deli Zhao, Fan Wang, Yu Rong, Aixin Sun, and Shijian Lu. Mmrl: Advancing the frontiers of multimodal reasoning, betps://githue com/LengSccon///ee 2025.
[21] Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, and Semih Yavuz. Im- proving I1m reasoning through scaling inference computation with collaborative verification. arXiv preprint 2400:22400.0331. 2024.
[22] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee, Visual instruction tuning. arXiv preprint 20444:2304.08485, 2023.
[23] Yixin Liu, Yonghui Wu, Denghui Zhang, and Lichao Sun. Agentic autosurvey: Let Ilms survey llms. arXiv preprint arXiv:2509.1866 2025.
[24] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.055 101, 2017.
[25] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu, Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04163. 2021.
[26] Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song- Cann-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110.133. 13214 2021.
[27] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0225: 2023.
(28) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct. arXiv preprint 098553230..09583, 2023.
[29] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint arXiv:2410.1228 12832, 2024.
[30] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[31] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning, arXiv preprint arX1v:2503.07365, 2025.
12

---

[32] OpenAl. Openai gpt-4.5 system card, 2025. URL attps://cdn /ttps://cdn.openai. com/
gpt-4-5-
--ssteemm system--t----- card- 2ooddd227222...m pdf.
[33] Shuai Peng, Di Fu, Liangeai Gao, Xiuqin Zhong, Hongguang Fu, and Zhi Tang. Multi- math: Bridging visual and mathematical reasoning for large language models. arXiv preprint arXiv:2409.0014 2024.
[34] Yi Pong. Xiaokun Wang. Yichen Wei, Jiangbo Pei, Weijie Qiu, Ai Jian, Yunzhuo Hao, Jiachun Pan, Tianyidan Xie, Li Ge, et al. Skywork rlv: Pioneering multimodal reasoning with chain-of- thought. arXiv preprint arXiv:250.00599, 2025.
[35] Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01228 2024.
[36] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[37] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint vrXiv:2402.03300 2024.
[38] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See- KionKiong Ng. Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406. 02.4406.72944 /7294, 2024.
[39] Yucheng Shi, Quanzheng Li, Jin Sun, Xiang Li, and Ninghao Liu. Enhancing cognition and explainability of multimodal foundation models with self-synthesized data. arXiv preprint arXiv:2502. 14044, 2025.
[40] Wentao Tan, Qiong Cao, Yibing Zhan, Chao Xue, and Changxing Ding. Beyond human data: Aligning multimodal large language models by iterative self-evolution. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
[41] Zhiquan Tan, Lai Wei, Jindong Wang, Xing Xie, and Weiran Huang. Can i understand what i create? self-knowledge evaluation of large language models, arXiv preprint 24Xi::2006.06144 2024.
[42] Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models. arXiv e-prints, pages arXiv-2503, 2025.
[43] Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty- aware rejection tuning for mathematical problem-solving. Advances in Neural Information Processing Systems, 37:7821-7844 2024.
[44] Fouad Trad and Ali Chehab. To ensemble or not: Assessing majority voting strategies for phishing detection with large language models. In International Conference on Intelligent Systems and Pattern Recognition, pages 158-173. Springer, 2024.
[45] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95166 2025.
[46] Xiaoxuan Wang, Yihe Deng, Mingyu Derek Ma, and Wei Wang. Entropy-based adaptive weighting for self-training. arXiv preprint arXiv:2503.23913. 2025.
[47] Xidong Wang, Dingjic Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Ilms to 1000 images efficiently via a hybrid architecture. arXiv preprint arXiv:2409.0 02889, 2024.
[48] Xiyao Wang, Zhengyuan Yang. Chao Feng, Hongjin Lu, Linjie Li, Chung- Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mets-guided sample selection for data-efficient visual reasoning self- oof--improvementt arXiv preprint 2004v:2504.079 2025.
13

---

[49] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consisteney improves chain of thought reasoning in language models. arXiv preprini arXiv:2203. //171, 2022.
[50] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions. arXiv preprint arXiv:2212. 2022.
[51] Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-:: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308. /2067, 2023.
[52] Lai Wei, Zhiquan Tan, Chenghai Li, Jindong Wang, and Weiran Huang. Diff-erank: A novel rank-based metric for evaluating large language models. arXiv preprint arXiv:2401.17139, 2024.
[53] Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang. Advancing multimodal reasoning via reinforcement learning with cold start. arXiv preprini WxXiv:2505.22334 2025.
[54] Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-rewarding language models: Self-improving alignment with Ilm-as-a-meta-judee arXiv preprint vrXiv:247..19599 2024.
[55] Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. arXiv preprint 1..ix:2411.1044// 2024.
[56] Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. arXiv preprint urXiv:2411. 2024.
[57] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 2551:2242.111155. 2024.
[58] An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu. Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 12122, 2024.
[59] Zhejian Yang, Yongchao Chen, Xueyang Zhou, Jiangyue Yan, Dingjie Song, Yinuo Liu, Yuting Li, Yu Zhang, Pan Zhou, Hechang Chen, et al. Agentic robot: A brain-inspired framework for vision-language-action models in embodied agents. arXiv preprint arXiv:2505.22 234450 2025.
[60] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang. Shunyu Liu, Yingjic Wang, Yuxin Song, Haocheng Feng, Li Shen, et al. Mulberry: Empowering milm with li-e reasoning and reflection via collective monte carlo tree search. arXiv preprint 1rXiv:2412.18319, 2024.
[61] Zheng Yaowei, Lu Junting, Wang Shenzhi, Feng Zhangehi, Kuang Dongdong, and Xiong Yuwen. Easyrl: An efficient, scalable multi multi-modality rl training framework. https // thub. com/ hiyouga/EasyR1,
2025.
[62] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
[63] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
[64] Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint 2rX.v:2405.17220 2024.
[65] Zhengqing Yuan, Yang Wang, Zhaoxu Li, Yanfang Ye, and Lichao Sun. Tinygpt-moe: Scal- ing multi-modal large language model via advanced vision encoder with mixture-of-experts. Proceedings of Machine Learning Research, 2024.
14

---

[66] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprini a50iv:2504. 2025.
[67] Kai Zhang, Rong Zhou, Eashan Adhikarla, Zhilingg Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision language foundation model for 11:3)12-11111111 2024.
diverse biomedical tasks. Nature Medicine,
[68] Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. Right question already half the answer: Fully unsupervised IIm reasoning incentivization. arXiv preprint arXiv:2504.08812 2025.
[69] Renrui Zhang. Dongzii Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-We Chang. Yu Qiao, et al. Mathverse: Does your multi- Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[70] Renrui Zhang, Xinyu Wei, Dongzhi Jiang. Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, et al. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint arXiv:2407.087. 2024.
[71] Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, al. Multimodal self-instruct: Synthetic abstract image and visual reasoning instruction using language model. arXiv preprint arXiv:2407.07053, 2024.
[72] Xiaoying Zhang, Da Peng. Yipeng Zhang, Zonghao Guo, Chengyue Wu, Chi Chen, Wei Ke, Helen Meng, and Maosong Sun. Will pre-training ever end? first step toward next-generation foundation milms via self-improving systematic cognition. arXiv e-prints, pages arXiv-2503, 2025.
[73] Yu Zhang, Jinlong Ma. Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, and Min Zhang. Evaluating and steering modality preferences in multimodal large language model. arXiv preprini urXiv:2505.20977 2025.
[74] Henry Hengyuan Zhao, Pan Zhou, and Mike Zheng Shou. Genixer: Empowering multimodal large language model as powerful data generator. In European Conference on Computer Vision, pages 129-147. Springer, 2024.
[75] Haojie Zheng, Tianyang Xu, Hanchi Sun, Shu Pu, Ruoxi Chen, and Lichao Sun. Thinking before looking: Improving multimodal IIm reasoning via mitigating visual hallucination. arXiv preprint 2rXiv:2411.1255) 2024.
[76] Lianmin Zheng, Lii Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging au-jde with mt-bench and chatbot arena. arXiv preprint arXiv:2306.. 2006552300.05685 2023.
[77] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang. Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. International Journal of Machine Learning and Cybernetics, pages 1-65, 2024.
[78] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsich. RI-zero's "aha moment" in visual reasoning on a 2b non-sft model. arXiv preprint arXiv:2503.05132. 2025.
[79] Shijie Zhou, Ruiyi Zhang, Yufan Zhou, and Changyou Chen. high-quality text-rich image instruction tuning dataset via hybrid instruction generation. arXiv preprint 2rXiv:2412.1633 2024.
[80] Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self-rewarding vision language models. arXiv preprint at 405. 2024.
15

---

[81] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing languaee understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[82] Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal alignment to enhance mathematical reasoning. arXiv preprint arXiv:2408.086400, 2024.
[83] Fei Zuo, Kehai Chen, Yu Zhang, Zhengshan Xue, and Min Zhang. InImageTrans: Multimodal LLM-based text image machine translation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 20256-20277, Vienna, Austria, July 2025. Association for doi: 10. URL https://aclanthooogg. org 20g/2025..ee..oo wwwdiinnnnniinn a
Computational LingutttiionallLLiiiiiiiiiii ISBN 979-8-999996-7--5555555
[84] Yuxin Zuo, Kaiyan Zhang, Shang Qu, Li Sheng, Xuekai Zhu, Biqing Qi, Youbang Sun, Ganqu Cui, Ning Ding, and Bowen Zhou. Ttrl: Test-time reinforcement learning. arXiv preprint 2504 /6084, 2025.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have mentioned in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have mentioned limitations the "experiment" part our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

Justification:
We provide them in our experiment part.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: We open-sourced models, which are easy reproduce.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We provide codes in our supplemental material. The data and models we used are all open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips... cc/ mublic/guides: /CodeSobmisss colbmmsssionll sssisssoiccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide in experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
Experiments on MLLMs are too expensive to run for many times.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide in our appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpar///////e///e/e/e//ae cc htte/publcceEeeee
httpp//bii/hhhtttttteeleeeee.eee cttphiiccccclleeiececcc ines?
Answer: [Yes]
Justification:
We follow NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license and terms of use in experiments.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: h/tp://ies...//eeeee htt/e//rps.cce/ooeeeeeeeeeeeeeee / for what should or should be described.
23

---

Appendix
A Implementation Details
We provide the implementation details of our experiments as follows.
A.1 Baselines
Here, we explain how we implement different baseline methods in comparison.
LMSI [16] employs the maoority-voted response the target for supervised fine-tuning (SFT). For each question, we generate multiple responses and retain the ones that lead to the majority answer for training.
SRLM [54] studies Self- Rewardidd Language Models, where the model itself is used via LLM-as- a-Judge prompting to provide its own rewards during training. In particular, for each question, we generate multiple candidate responses and use the prompt provided in the original paper have the MILMM score its own outputs. Among the responses, the one with the highest score is selected as the positive example, and the one with the lowest score as the negative example. These pairs are then used to construct preference datasets for Direct Preference Optimization (DPO) [36].
Genixer [74] introduces a comprehensive data generation pipeline consisting of four key steps: (i) instruction data collection, (ii) instruction template design, (iii) empowering MLLMs, and (iv) data generation and filtering. To adapt Genixer in our setting, we remove the first two steps because we already have instruction data. After that, we use Qwen2.5-Ve VL L as the backbone model to self- generate responses times per question for each dataset. In the filtering stage, we use the prompt to let the model self-judge the responses following Genixer:
Here a question-anse answer pair. Is (Q: Xgs A: Xa} true for this image? Please answer this question with Yes or No.
In addition, Genixer calculates the probability of predicting the "Yes" rather than prompt the model to directly output "Yes" or "No" as the filtering label:
Xx i1 P(ullXX, Xa,es),
where Y, is the predicted judge. X, is the image, X, is the question, X, is the self-generated response, and L is the length the total predicted judge. Then, it proposes threshold À control the filtering in the following manner:
True,
if Y,
Yes and P(YP) > A
S
False,
if Y, Yes and P(Y < <
False, Yf
No
where 5" the filter label representing keeping or removing the current sample. P(Y.") denotes the probability of the result "Yes" of n-th candidates. À is set 0.7 following the paper.
STIC [6] proposes a two-stage self-training algorithm focusing on the image comprehension capa- bility of the MILMs. In Stage 1, the base MILMM self-constructs its preference dataset for image description using well-designed prompts, poorly-designe
prompts, and distorted images with dif- fusion noise. In Stage 2, small portion of the previously used SFT data is recycled and infused with model-generated image descriptions to further fine-tune the base MLLM. In particular, since Qwen2.5-Vv does not open-source the SFT data, opt use the model's self-generated responses sampled from different datasets to represent the previously used SFT data.
A.2 Benchmarks
We provide some details about the benchmarks we use to evaluate the models' reasoning ability. Math Vision [45] is a challenging benchmark containing 3040 mathematical problems with visual contexts from real-world math competitions across 12 grades. It covers 16 subjects over 5 difficulty

---

levels, including specialized topies like Analytic Geometry, Combinatorial Geometry, and Topology. Math Vista [27] is a comprehensive benchmark for evaluating mathematical reasoning in visual contexts. It contains 1000 questions featuring diverse problem types including geometry, charts, and tables. Math Verse [69] is an all-around visual math benchmark designed for an equitable and in- depth evaluation of MLLMs. The test set contains 3940 multi-subject math problems with diagrams from publicly available sources, focusing on Plane Geometry and Solid Geometry. We-Math [35] meticulously collect and categorize 1740 visual math problems in the test set, spanning 67 hierarchical knowledge concepts and 5 layers of knowledge granularity.
For all benchmarks, we prompt the models to place their final answers within designated box format. We then employ Qwen2.5-32B -ann255-32-Insstrce [57] to evaluate answer correctness by comparing the extracted responses with ground truth answers, which often contain complex mathematical expressions. Note that our reported benchmark scores may differ from those in the original papers due to variations in evaluation protocols.
A.3 Standard Training Datasets
In our experiments, we use three standard training datasets for multi-modal reasoning: Geome- try3K [25]. GeoQA [4], and MMRI [20]. Geometry3K consists of 2.1K multiple-choice questions in 8K fill-in-the-baant questions sourced from the larger Geol dataset [10]. MMRI consists of 7,000 samples and includes both multiple-choice questions and fill-i-thhe-blane questions. These samples cover a range of tasks, including understanding charts and geometric reasoning.
the training set, covering a wide range of geometric shapes. GeoQA includes
B
Additional Experiments
B.1
Training Dynamics
To better understand the behavior of MM-UPT during training, we monitor several diagnostic metries, including the majority voting reward and entropy, both of which are label-free and provide insights in the absence of ground-truth truth supervision. In particular, majority voting reward calculated following Equation 2. Entropy can be used as an unsupervised objective that measures the uncertainty of the model's generation [46, 52, 68]. For a group of responses O = {ox) sampled from the question and image I, we cluster the responses according to their meaning. That is, if two responses share the same meaning (i.e., extracted answers), they should merged into one same cluster in the semantic The empirical distribution over clusters is
space. This results to K(K < G) clusters C = {e)) ) defined as:
lcy1 p(c, 1)= G
where |cyl denotes the number of responses that belongs to cj. The semantic entropy (denoted as H) over the model's 's response meanings distribution can be estimated as follows:
=
Σ cCE(C)
plesla, I) log
Figure 2 presents the MM-UPT training curves of the key metrics on Qwen2.5-VL-7B using the MMR1 dataset. We observe that the majority voting reward consistently increases over time, accom- panied by a steady decrease in the entropy. This indicates that the model is converging toward more consistent predictions, reflecting improved confidence and stability in its responses.
Additionally, we track the change in average benchmark accuracy and effective rank [52] throughout training. The accuracy exhibits an upward trend, demonstrating that our MM-UPT framework- based on an online reinforcement learning dggrrthm-effecttnn continuously and iteratively. The effective rank [52] further measures the amount of knowledge the model comprehends in the datasets. During training, the internal knowledge of the model exploited, leading to consistent increase in the effective rank on the benchmark.
enables the model to self-improve
25

---

Majority Vokting Beward
0.530
Effective Rank
0,580
0.89
Entropy
Accuracy
0.52
0.500
0.515
0,570
0.330
Staps
Staps
Steps
Steps
Figure 2: Training dynamics of MM-UPT using Qwen2.5-VI.-7B on the MMR1 dataset. We plot the majority voting reward, semantic entropy, and average benchmark accuracy the course of unsupervised post-training.
Table 6: Performance on non non-mathematical VQA benchmarks. We evaluate Qwen2.5-VL-7B before and after applying MM-UPT on the MMRI dataset. Scores reported as accuracy.
Models
ChartQA
IconQA
Qwen2,5-VL-7B
71.96
54.20
Qwen2.5-VL-7B MM-UPT
77.48 (7.7%
56.55 (4.3%))
B.2 Generalization Beyond Multimodal Mathematical Reasoning
A potential concern is that by encouraging convergence toward high- high-consensus answers, MM-UPT might reduce response diversity and overspecialize on mathematical reasoning, potentially harming performance other tasks. To investigate whether this method suffers from a negative impact on broader generalization, we extend our evaluation to two non-mathematical visual question answering benchmarks: ChartQA [30], which tests visual perception over charts, and IconQA [26], which focuses on abstract diagram understanding. In particular, we evaluate the Qwen2.5-VLL 5--L7BBBB model trained with MM-UPT on the MMR1 dataset. As shown in Table 6, our method not only avoids performance degradation but leads to notable improvements on both benchmarks. This suggests that the underlying mechanism of MM- AM--UPT-reinfrccin the generation of more reliable and consistent answers- is a beneficial trait that positively transfers to other domains. The results indicate that the induced consistency does not degrade, and even enhance, the model's general utility on related visual understanding tasks.
B.3 Adaptability to Language Tasks
Furthermore, to assess the adaptability of our MM-UPT framework to purely language-based based domains, we apply MM-UPT to a language model, Qwen2.5-MATH-77 [58]. trained on the DAPO-I7K dataset [63] using the same self-rewarding mechanism. We evaluate the resulting model on two popular pure-math benchmarks: MATH [14] and Omni-MATH [9]. As shown in Table 7, MM- UPT leads substantial improvements on both benchmarks. These results strongly suggest that our framework not limited to multi-modal settings and can be effectively extended to purely banee
language- based reasoning tasks, provided the base model demonstrates sufficient initial competency.
Table Performance on pure-language mathematical reasoning benchmarks. We evaluate Qwen2.5-
MATH-7B before and after applying our unsupervised post-training method.
Models
MATH
Omni-MATH
Qwen2.5-MATHH-72
51.20
18.09
Qwen2.5-MATTH-7B UPT
75.80 (48.0%
32.72 (80.9%
C
Compute Resources
We conduct our experiments using NVIDIA H100-80G and A800-40G GPUs. The experimental time using A800 for training Qwen2.5-VI.-7B [1] on the Geometry3K [25] dataset using GRPO around 10 hours.
26


Reviews:
[{"review_id": "oaNPqCXRDk", "reviewer": "Reviewer_QjgG", "summary": "This paper introduces MM-UPT, a framework for unsupervised post-training of MLLMs that enables self-improvement without external supervision. The method builds on GRPO by replacing traditional reward signals with a self-rewarding mechanism based on majority voting over multiple sampled responses. Given unlabeled multi-modal data, the model samples multiple responses, determines pseudo-labels through majority voting, and updates itself via GRPO. The authors evaluate MM-UPT on multi-modal mathematical reasoning benchmarks, showing that it significantly improves performance compared to base models and outperforms other unsupervised baselines. They also explore synthetic data generation strategies, finding that model-generated questions can achieve competitive results. The paper provides analysis of when this approach succeeds and when if fails.", "questions": "See Weakness", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n- The paper proposes a remarkably simple yet effective approach for unsupervised GRPO based on majority voting, achieving strong results—even outperforming supervised training on some benchmarks.\n\n- It provides a thoughtful analysis of the conditions under which majority-voting-based unsupervised GRPO is likely to succeed—namely, when the training data is relatively easy and the base model already has reasonable initial accuracy. It also clearly acknowledges limitations, such as failure cases on harder datasets where the model lacks sufficient prior knowledge.\n\nWeaknesses\n\n- It seems that the proposed unsupervised GRPO method, MM-UPT, cannot consistently improve performance across all benchmarks. As shown in Table 1, the performance on MathVerse drops modestly in some data settings. Can the authors explain the possible reasons for this inconsistency?\n\n- Based on Table 1, it seems that using different kinds of data brings significantly different improvements on different benchmarks. However, the paper lacks analysis on why different training data improve performance differently and lacks experiments investigating whether combining these datasets could bring additional performance improvements.\n\n- Based on the basic idea of majority voting and the analysis of semantic entropy during training, this method encourages the model to converge toward more consistent predictions. My concern is that this may reduce response diversity. As a result, I'm wondering whether this method affects the model's performance on other non-math benchmarks?\n\n- The method's effectiveness critically depends on the base model's initial accuracy being above chance (>0.5). This is a strong assumption that limits applicability, as the method cannot bootstrap knowledge on genuinely difficult or out-of-domain datasets where the model would initially have low performance.\n\n- Given that evaluation focuses solely on multi-modal mathematical reasoning, can this method transfer to improve pure language math tasks or other non-math tasks?", "was_revised": true, "final_justification": "After reviewing the rebuttals, I keep my initial score.", "rebuttal": "Thank you for highlighting the simplicity and effectiveness of MM-UPT, and for appreciating our analysis of its strengths and limitations. We would like to address your concerns point by point below.\n\n---\n\n**Q1**. It seems that the proposed unsupervised GRPO method, MM-UPT, cannot consistently improve performance across all benchmarks. As shown in Table 1, the performance on MathVerse drops modestly in some data settings. Can the authors explain the possible reasons for this inconsistency?\n\n**A1**: Thanks for your question. **We believe that this inconsistency is a natural and common phenomenon in self-improving settings** [1,2]. For instance, [1] reports that their self-improvement strategy improves LLaMA2’s performance on MMLU but leads to performance drops on ARC and HellaSwag. Similarly, [2] shows that their method enhances Mistral-7B on ARC and TruthfulQA, while slightly degrading its performance on MMLU. This reflects a broad trend in the field: self-training can be hard to guarantee the consistent performance gain across all benchmarks.\n\nIn addition, as our unsupervised MM-UPT relies on a pseudo-reward from majority voting, the reliability of this reward signal is dependent on the model's initial accuracy and consistency on the training data. That said, **this reward signal may be noisy or suboptimal, particularly when the base model’s initial performance is weak on some MathVerse-style questions contained in the datasets like Geometry3k**. In such cases, MM-UPT may reinforce incorrect but confident outputs, leading to the modest performance drop in MathVerse when training on some datasets.\n\n[1] Yuan, Weizhe, et al. \"Self-rewarding language models.\" ICML 2024.\n\n[2] Yang, Haoyan, et al. \"Dynamic Noise Preference Optimization for LLM Self-Improvement via Synthetic Data.\" \n\n---\n\n**Q2**. Based on Table 1, it seems that using different kinds of data brings significantly different improvements on different benchmarks. However, the paper lacks analysis on why different training data improve performance differently and lacks experiments investigating whether combining these datasets could bring additional performance improvements.\n\n**A2**: Thanks for your comment. We agree that different training datasets lead to different improvements across benchmarks, and we provide further explanation and supporting experiments below.\n\nFirst of all, this variation largely stems from the domain and structure alignment between the training data and evaluation tasks. For example, MMR1 provides more diverse multi-modal data formats (including diagrams, tables, and charts), while Geometry3K and GeoQA are more specialized towards geometric data. In addition, the pseudo-reward signal quality also plays a critical role in different datasets as mentioned in Question 1, which can bring significantly different improvements on different benchmarks.\n\nSecond, **training with different datasets separately (rather than combining them) is a standard practice in finetuning**. For example, it is common to train models separately on GSM8K and MATH, as shown in [1]. This strategy allows for controlled analysis of transfer behavior and prevents data imbalance.\n\nMoreover, **we also conduct experiments where we combine Geometry3K, GeoQA, and MMR1 together for training, and observe that the model achieves further improvements in overall performance**. These results suggest that data combination is a viable strategy, especially when the datasets have complementary coverage.\n\n| Different Datsets | We-Math | MathVista | MathVerse | MathVision | **Avg**    |\n| ----------------- | ------- | --------- | --------- | ---------- | ---------- |\n| Geometry3k        | 0.6661  | 0.6850    | 0.4246    | 0.2733     | 0.5123     |\n| GeoQA             | 0.6822  | 0.6890    | 0.4368    | 0.2707     | 0.5197     |\n| MMR1              | 0.6874  | 0.7290    | 0.4487    | 0.2615     | 0.5316 |\n| Combine                  |  0.6970       | 0.7190          |   0.4492        |     0.2757       |  **0.5352**  |\n\n\n [1] Lin, Zhihang, et al. \"Cppo: Accelerating the training of group relative policy optimization-based reasoning models.\" \n \n---\n\n**Q3**. Based on the basic idea of majority voting and the analysis of semantic entropy during training, this method encourages the model to converge toward more consistent predictions. My concern is that this may reduce response diversity. As a result, I'm wondering whether this method affects the model's performance on other non-math benchmarks?\n\n**A3**: Thank you for the question. We agree that our method tends to reduce output diversity by encouraging majority-agreement responses. However, this is not unique to MM-UPT. In fact, **response diversity reduction is a widely observed phenomenon in supervised reinforcement learning (RL) as well**: Some works [1,2] have shown that the reasoning capability boundary of LLMs often narrows as RLVR training progresses. Therefore, it is expected that MM-UPT, which operates under a similar reinforcement learning setup, would exhibit similar behavior.\n\nIn our study, we mainly focus on multi-modal reasoning because it is a very challenging task. To investigate potential side effects on broader generalization, **we also extend our evaluation for the model trained on MMR1 using MM-UPT on two non-math benchmarks: ChartQA [3] and ICONQA [4]**, which test visual perception abilities over charts and natural images respectively. As shown below, MM-UPT does not hurt, and can even enhance performance on tasks outside math, likely due to the reinforcement of more reliable and consistent answer generation in general. This also suggests that the reduced diversity may not degrade model utility on other non-math benchmarks. \n\n\n| Models                    | ChartQA    | ICONQA     |\n| ------------------------- | ---------- | ---------- |\n| Qwen2.5-VL-7B-MM-UPT | **0.7748** | **0.5655** |\n| Qwen2.5-VL-7B             | 0.7196     | 0.5420     |\n\n\n[1] Yue, Yang, et al. \"Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?.\"\n\n[2] Dang, Xingyu, et al. \"Assessing diversity collapse in reasoning.\" SSI-FM@ICLR 2025.\n\n[3] Masry, Ahmed, et al. \"Chartqa: A benchmark for question answering about charts with visual and logical reasoning.\"\n\n[4] Lu, Pan, et al. \"Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.\"\n\n---\n\n**Q4**. The method's effectiveness critically depends on the base model's initial accuracy being above chance (>0.5). This is a strong assumption that limits applicability, as the method cannot bootstrap knowledge on genuinely difficult or out-of-domain datasets where the model would initially have low performance.\n\n**A4**: Thank you for pointing out this. We agree that MM-UPT relies on the assumption that the base model achieves above-chance performance, which is necessary for majority voting to serve as a reliable pseudo-reward signal. **We have explicitly acknowledged and analyzed this limitation in Section 5.3. This is also a fundamental limitation shared by most unsupervised or self-improving frameworks**. Without a reasonably competent starting point, self-generated signals are likely to reinforce noise rather than useful knowledge. In this work, **we position MM-UPT as an effective post-training enhancement technique rather than a substitute for pretraining or supervised fine-tuning**. Nonetheless, we view this as a promising direction for future work. One avenue is to incorporate more robust reward estimation strategies (e.g., LLM-as-a-Judge, collaborative verification) that go beyond majority voting, allowing self-improvement even in low-confidence regimes\n\n---\n\n**Q5**. Given that evaluation focuses solely on multi-modal mathematical reasoning, can this method transfer to improve pure language math tasks or other non-math tasks?\n\n**A5**: Thank you for the question. As discussed in A3, we have already evaluated MM-UPT on non-math multi-modal benchmarks (ChartQA and ICONQA), and observed consistent performance gains, suggesting that MM-UPT does not harm generalization beyond multimodal math tasks.\n\nTo further assess transferability to pure language math tasks, we apply MM-UPT to a language-only mathematical model, Qwen2.5-MATH-7B [1], trained on the DAPO-17K dataset [2] using the same self-rewarding mechanism. We evaluate the resulting model on two popular benchmarks: MATH [3] and Omni-MATH [4]. As shown below, **MM-UPT leads to substantial improvements on both benchmarks for Qwen2.5-MATH-7B**. These results suggest that MM-UPT is not limited to multi-modal settings, and it can also be effectively extended to purely language-based reasoning tasks, provided the base model demonstrates sufficient initial competency. \n\n| Models                    | MATH     | Omni-MATH |\n| ------------------------- | -------- | --------- |\n| Qwen2.5-MATH-7B + MM-UPT    | **0.7580** | **0.3272** |\n| Qwen2.5-MATH-7B           | 0.5120   | 0.1809    |\n\n\n[1] Yang, An, et al. \"Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.\" \n\n[2] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" \n\n[3] Hendrycks, Dan, et al. \"Measuring mathematical problem solving with the math dataset.\" \n\n[4] Gao, Bofei, et al. \"Omni-math: A universal olympiad level mathematic benchmark for large language models.\""}, {"review_id": "mDyuosoNyH", "reviewer": "Reviewer_nZ3M", "summary": "The paper proposes an unsupervised RL training framework, MM-UPT, to improve the reasoning capability of MLLMs, which uses the majority voting as a proxy for the ground truth answer. The authors conduct extensive experiments and demonstrate the effectiveness of their method. They also provide interesting discussions and analysis", "questions": "How does the weight beta of the KL divergence affect the training result?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The topic is very related. The proposed method, MM-UPT, is simple but effective. The ground truth answer is replaced with a majority voted answer, so GRPO-like training can be done in an unsupervised way.\n- MM-UPT outperforms prior unsupervised methods and approaches fully‑supervised GRPO on several benchmarks.\n- The authors investigate synthetic question generation, and demonstrate that synthetic questions can be useful to improving model reasoning ability. The proposed method works on smaller scale models (3B) too.\n- The paper is clearly written, and the authors provide open-sourced code.\n\nWeaknesses:\n- If the starting accuracy is lower than 0.5, this method may not work, so it depends on the quality of the pre-trained model. However, I understand this is a basic requirement for almost all unsupervised methods.", "was_revised": true, "final_justification": "After reviewing the rebuttals, I keep my original score of 5.", "rebuttal": "Thank you for your positive and encouraging feedback, and for recognizing the effectiveness and clarity of our work. We would like to respond to your questions below.\n\n---\n\n**Q1**. If the starting accuracy is lower than 0.5, this method may not work, so it depends on the quality of the pre-trained model. However, I understand this is a basic requirement for almost all unsupervised methods.\n\n**A1**: Thank you for highlighting this important observation. We agree that MM-UPT requires the base model to exhibit non-trivial initial accuracy (i.e., >0.5) for majority voting to yield meaningful pseudo-rewards. This dependency on pre-trained model quality is indeed a limitation. However, as you rightly notes, it is a fundamental assumption shared by most unsupervised or self-improving frameworks. Without a reasonably competent starting point, self-generated signals are likely to reinforce noise rather than useful knowledge. In this work, we position MM-UPT as a post-training enhancement technique rather than a substitute for pretraining or supervised fine-tuning. MM-UPT is best used to amplify existing knowledge in models that already demonstrate basic competency on the target domain.\n\n---\n\n**Q2**. How does the weight beta of the KL divergence affect the training result?\n\n**A2**: Thank you for the question. The KL divergence weight $\\beta$ plays a critical role in stabilizing MM-UPT training. If $\\beta$ is set too small, the policy may deviate too far from the reference model, leading to instability and training collapse. Conversely, setting $\\beta$ too large overly constrains the policy, resulting in minimal updates. Following prior works [1], we adopt an empirical value of **1.0e-2** as the default setting. To further validate this choice, **we additionally conduct an ablation study on the KL coefficient using the MMR1 dataset**. As shown below, setting $\\beta$ = 1.0e-2 achieves a good balance between stability and learning performance on average. \n\n| KL Coefficient ($\\beta$) | We-Math    | MathVista  | MathVerse  | MathVision | **Avg** |\n| ------------------------ | ---------- | ---------- | ---------- | ---------- | ------------------ |\n| 2.0e-2                   | 0.6943     | 0.7160     | 0.4462     | 0.2572     | 0.5284             |\n| 5.0e-3                   | 0.6960     | 0.6920     | 0.4368     | 0.2635     | 0.5221             |\n| **1.0e-2**                   | 0.6874     | 0.7290     | 0.4487     | 0.2615     | **0.5317**             |\n\n[1] Deng, Yihe, et al. \"Openvlthinker: An early exploration to complex vision-language reasoning via iterative self-improvement.\""}, {"review_id": "u7fvJfWopH", "reviewer": "Reviewer_XWR7", "summary": "This paper introduces MM-UPT for unsupervised post-training of MLLMs using the GRPO RL algorithm without any human-annotated supervision. MM-UPT utilizes pseudo label based on majority voting over multiple sampled responses to guide learning. Experiments show that the proposed framework improves the reasoning ability of Qwen2.5-VL-7B and other base models on benchmarks like MathVista and We-Math, outperforming prior unsupervised baselines and even approaching supervised GRPO performance.", "questions": "Same as stated in weaknesses. If the authors can answer my questions, I am willing to raise my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The performance of the proposed method surpasses the previous unsupervised baselines, indicating the potential of unsupervised RL post-training.\n2. The analysis about the failure cases and reasons why MM-UPT can work is insightful.\n3. The paper is easy to follow with clear structures.\n\n**Weaknesses**:\n1. Besides what have been analyzed for MM-UPT, another question is the upper bound of MM-UPT. More specifically, what is the relationship between the amount of the unsupervised data and the performance gain. Can the model benefit from unsupervised training endlessly?\n2. For the comparison with supervised baseline like GRPO, I am wondering the epoch setting for supervised GRPO, as after training for more epochs, the sampled responses would be more consistent for supervised GRPO as well. If both of them use the same training data as the seed, will them have similar upper bound after convergence.", "was_revised": true, "final_justification": "I have read the rebuttal from authors and raise my score accordingly.", "rebuttal": "Thank you for recognizing the strengths of our method and analysis. We would like to address your questions regarding the upper bound of MM-UPT and its relationship with supervised GRPO below.\n\n---\n\n**Q1**. Besides what have been analyzed for MM-UPT, another question is the upper bound of MM-UPT. More specifically, what is the relationship between the amount of the unsupervised data and the performance gain. Can the model benefit from unsupervised training endlessly?\n\n**A1**: Thank you for this question. We agree that understanding the upper bound of MM-UPT is crucial for assessing its scalability. To explain this, we conduct additional experiments analyzing the relationship between the size of the unsupervised training data and the resulting performance. **The results below show that the performance gradually improves as the amount of unsupervised data increases within the tested range**.\n\nNotably, the general scaling law [1] rules state that under a fixed model size, performance improvements from additional data will eventually saturate. Therefore, **while we have not yet observed an obvious plateau in our current setting, we believe the performance of MM-UPT is indeed upper-bounded by the model’s capacity**. However, this further exploration would require significantly more computational resources and access to much larger, more diverse high-quality unsupervised datasets. Due to the resource constraints, currently we are unable to exhaustively explore this saturation point. We are actively working to scale up both data and training resources to better explore this interesting question.\n\n\n| Data Size | MathVision | MathVerse | MathVista | We-Math | Avg    |\n|:--------- |:---------- |:--------- |:--------- |:------- |:------ |\n| 0K        | 0.2487     | 0.4383    | 0.6630    | 0.6287  | 0.4947 |\n| 2K      | 0.2740     | 0.4459    | 0.6970    | 0.6730  | 0.5225 |\n| 4K      | 0.2733     | 0.4467    | 0.7010    | 0.6816  | 0.5257 |\n| 6K        | 0.2615     | 0.4487    | 0.7290    | 0.6874  | 0.5317 |\n| 8K          |  0.2763          |  0.4490         |     0.7120      |    0.6989     | 0.5341       |\n\n\n\n[1] Kaplan, Jared, et al. \"Scaling laws for neural language models.\" \n\n---\n\n**Q2**. For the comparison with supervised baseline like GRPO, I am wondering the epoch setting for supervised GRPO, as after training for more epochs, the sampled responses would be more consistent for supervised GRPO as well. If both of them use the same training data as the seed, will them have similar upper bound after convergence.\n\n\n**A2**: Thank you for the question regarding the training settings and the convergence bounds of our method compared to supervised GRPO.\n\n\nFirstly, we would like to clarify that all comparisons were conducted under fair conditions. As stated in Section 4 of our paper, **both our unsupervised MM-UPT and the supervised GRPO baselines were trained using the exact same hyperparameter settings**, including the number of training episodes (15). Both methods have converged by the end of training. Note that in our experiment, **we mask the labels of supervised data to perform unsupervised GRPO, simply to simulate a scenario where annotated labels are unavailable**.\n\n\nSecondly, **if both methods use the same seed data, they will indeed have a similar upper bound**. Practically, a supervised reward signal is superior to the pseudo-reward from majority voting. Therefore, **once a model has been fully trained with supervised GRPO on a given dataset, its performance is already near its peak capacity and the sampled responses would become consistent for that specific data distribution**. Applying MM-UPT subsequently on the same data (with labels masked) would not be expected to yield further significant improvements, as the potential for gains has already been exhausted by the more accurate supervised signal. To empirically validate this, we conduct an additional experiment. The results below show that after training with supervised GRPO on the MMR1 dataset, further applying MM-UPT on the same unlabeled MMR1 data results in a nearly identical performance upper bound. Thus, **in practical application scenarios, if the dataset has human-annotated labels, one can simply use supervised GRPO for training**.\n\n| Training Method | MathVision | MathVerse | MathVista | We-Math | Avg |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Supervised GRPO (on MMR1 dataset) | 0.2928 | 0.4690 | 0.6960 | 0.6839 | 0.5354 |\n| Supervised GRPO (on MMR1 dataset) + MM-UPT (on unlabeled MMR1) | 0.2924 | 0.4685 | 0.7000 | 0.6787 | 0.5349 |\n\n\nInspired by your question, we think that a more practical and important question is whether our unsupervised method is compatible with supervised GRPO. Specifically, **after conducting supervised GRPO on the model using one dataset, can MM-UPT further improve this model using a new, unlabeled dataset?** Actually, our paper has already provided evidence that the answer is **yes**. As shown in Table 3 in our paper (also the table below), our method can further improve the MM-Eureka-7B model. Note that the MM-Eureka-7B model was already tuned with supervised GRPO on the K12 dataset. **By applying MM-UPT with a new, unlabeled dataset (Geometry3k), we are able to further boost MM-Eureka-7B's average performance from 53.10 to 53.78**. This demonstrates the key value of our method: it serves as an effective tool for continual refinement, allowing already powerful models to keep improving by leveraging new and unlabeled data. \n\n| Training Method | MathVision | MathVerse | MathVista | We-Math | Avg |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Supervised GRPO (on K12 dataset) | 0.2806 | 0.5046 | 0.6940 | 0.6448 | 0.5310 |\n| Supervised GRPO (on K12 dataset) + MM-UPT (on unlabeled Geometry3k) | 0.2895 | 0.5063 | 0.6910 | 0.6644 | **0.5378** |\n\nIn summary, while the performance upper bound is similar when applying both methods to the same dataset, the true strength of MM-UPT lies in its ability to effectively and compatibly refine models using new, unlabeled data, showcasing a promising path for effective self-improvement."}, {"review_id": "FrKa3yrANj", "reviewer": "Reviewer_eY5v", "summary": "The paper proposes Multi-Modal Unsupervised Post-Training (MM-UPT), which uses a binary reward signal based on majority voting instead of the usual ground-truth answer in RLVR. This new reward signal is then applied using the GRPO algorithm.", "questions": "1. I saw in Figure 2 that the training is only shown for 80 steps. Would the proposed reward scale well over more training steps compared to the baseline with a ground-truth reward? \n\n2. What is the relationship between \"training episodes\" (line 173) and \"steps\"? How many rollout is done per training step?\n\n3. Would dropping the temperature to 0 or using majority voting during inference be sufficient to achieve the same performance increase?\n\n4. Would pass@n decrease significantly as every response converges to the majority answer? Such trade-off should be clearly explained in the paper.", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The proposed learning rule is simple, consisting of a change to the reward function.\n- The experimental results are promising, showing that the new method is on par with standard GRPO that uses ground-truth answers.\n\nWeakness\n\n- I am concerned with the scalability of the method across different training data and longer training durations. I understand that the method is essentially decreasing the policy's entropy in a specific way—by steering all responses toward the same answer from majority voting. The positive results in the experiments may not scale as the training data size increases or over more training steps. For instance, standard GRPO with ground-truth answers can still learn from incorrect answers across diverse questions. In contrast, with majority voting, the network might learn to decrease its entropy to an extent that it achieves high rewards on every prompt simply by being more deterministic, rather than by being more correct.\n- The paper has insufficient empirical analysis. The theoretical analysis in Section 5.2 is somewhat obvious. More detailed empirical analysis is needed—for example, how do pass@n and majority-voting@n change during training? What was the temperature used in the experiments and evaluation, and how does temperature affect the results?", "was_revised": true, "final_justification": "I recommend borderline acceptance. The main concern lies in the insufficient analysis presented in the paper—the writing tends to be biased:\n\n- A model with majority voting already achieves similar performance to the fine-tuned model with a majority voting reward. This trade-off between inference cost and training cost should be discussed in the main paper. Which is more important depends on the use case and should be left to the user's judgment.\n- The sacrifice in pass@n is not adequately explored, particularly in comparison to the baseline. I do not believe that simply stating the result is consistent with prior literature is sufficient.\n\nThat said, given that the main results are solid, I still recommend marginal acceptance—assuming that the above issues will be addressed. A paper should provide a balanced evaluation of a method—not read like a sales pitch that highlights only its strengths.", "rebuttal": "Thank you for your review and detailed feedback. We would like to address your concerns below.\n\n---\n\n**Q1**. I am concerned with the scalability of the method across different training data and longer training durations. I understand that the method is essentially decreasing the policy's entropy in a specific way—by steering all responses toward the same answer from majority voting. The positive results in the experiments may not scale as the training data size increases or over more training steps. For instance, standard GRPO with ground-truth answers can still learn from incorrect answers across diverse questions. In contrast, with majority voting, the network might learn to decrease its entropy to an extent that it achieves high rewards on every prompt simply by being more deterministic, rather than by being more correct.\n\n**A1**: Thanks for your questions. Firstly, we would like to clarify that **entropy reduction is a common outcome in RL, including in supervised GRPO and other RL methods [1]**. The key is whether convergence improves correctness or merely confidence.\n\nTo investigate this, we show that majority voting can offer a rational and effective pseudo-label as analyzed in Section 5.2. Our results in Table 1 and Figure 2 also show that **MM-UPT improves both accuracy and consistency, not just determinism**. Additionally, scaling more training steps (see A3) further reveals that the performance gains can plateau in an expected way similar to supervised GRPO, with no evidence of overconfidence.\n\nBesides, we conduct additional experiments to study the performance when data scales. As shown below, **MM-UPT steadily improves with more unlabeled data, suggesting its scalability within the tested range**.\n\n|Data Size|MathVision|MathVerse|MathVista|We-Math|Avg|\n|-|-|-|-|-|-|\n|0K|0.2487|0.4383|0.6630|0.6287|0.4947|\n|2K|0.2740|0.4459|0.6970|0.6730|0.5225|\n|4K|0.2733|0.4467|0.7010|0.6816|0.5257|\n|6K|0.2615|0.4487|0.7290|0.6874|0.5317|\n|8K|0.2763|0.4490|0.7120|0.6989|0.5341|\n\nWe think that fully investigating the scalability of our method is an another important and challenging problem, requiring large-scale computation resources. We are actively working on this direction as part of our future work.\n\n---\n\n**Q2**. The paper has insufficient empirical analysis. The theoretical analysis in Section 5.2 is somewhat obvious. More detailed empirical analysis is needed—for example, how do pass@n and majority-voting@n change during training? What was the temperature used in the experiments and evaluation, and how does temperature affect the results? \n\n**A2**: Thank you for the questions. We would like to clarify that Section 5.2 serves as an easy-understanding and necessary justification for the rationality of reward design in MM-UPT. Beyond that, we provide sufficient empirical analysis across different benchmarks, models, datasets, and ablations. These collectively form a comprehensive empirical foundation. Moreover, **we also conduct the following study regarding the behavior of pass@n and majority-voting@n (maj@n) for different training steps, and the impact of different temperatures in training**.\n\nAs shown below, we observe that maj@10 increases and stabilizes during training, indicating that MM-UPT helps the model consolidate its reasoning, and make more consistent and correct predictions. That said, majority voting benefits are baked into the model. In addition, pass@n tends to increase when n is small (e.g., n=1, shown in Figure 2 in our paper) while decline when n is large (e.g., n=10 shown below). This is a well-known tradeoff in RL training. More details are shown in A5 and A6.\n\n|maj@10|0|10|20|30|40|50|60|70|80|90|\n|-|-|-|-|-|-|-|-|-|-|-|\n|MathVista|0.7050|0.7000|0.7230|0.7280|0.7420|0.7260|0.7410|0.7240|0.7310|0.7210|\n|We-Math|0.6948|0.7098|0.7103|0.7184|0.7224|0.7126|0.7069|0.7080|0.7167|0.7075|\n\n|pass@10|0|10|20|30|40|50|60|70|80|90|\n|-|-|-|-|-|-|-|-|-|-|-|\n|MathVista|0.8730|0.8680|0.8670|0.8630|0.8590|0.8590|0.8480|0.8270|0.8240|0.8150|\n|We-Math|0.9420|0.9178|0.9126|0.9086|0.9000|0.8902|0.8753|0.8631|0.8621|0.8420|\n\nIn our training, we use a rollout temperature of 0.7. This strikes a good balance: lower temperatures produce low-diversity outputs, while higher temperatures often lead to lower-quality outputs. We further conduct an ablation on the MMR1 dataset. As shown below, **temperature = 0.7 outperforms both higher and lower settings**, confirming that it helps generate effective pseudo-reward.\n\n|Different rollout temperatures for training|We-Math| MathVista|MathVerse|MathVision|**Avg**|\n|-|-|-|-|-|-|\n|temperature=1.0|0.6851|0.7060|0.4431|0.2763|0.5276|\n|temperature=0.4| 0.6523 |0.6850|0.4338|0.2641|0.5088|\n|temperature=0.7|0.6874|0.7290|0.4487|0.2615|**0.5317**|\n\n---\n\n**Q3**. I saw in Figure 2 that the training is only shown for 80 steps. Would the proposed reward scale well over more training steps compared to the baseline with a ground-truth reward?\n\n**A3**: Thank you for your question. Our primary goal in this setting was to confirm that the proposed unsupervised method can indeed lead to accuracy gains over steps. To further verify scalability, we extend training to 110 steps. **As the table shown below, MM-UPT continues to improve slightly and eventually converges, with performance plateauing around step 90. This convergence behavior is expected when given a fixed model and dataset, and it also aligns with the supervised GRPO baseline which converges at a similar pace (around step 100)**. While we do not aim to outperform supervised GRPO in absolute terms, our results show that MM-UPT sustains effective training and demonstrates comparable convergence behavior to supervised GRPO. \n\n|Steps|80|85|90|95|100|105|110| \n|-|-|-|-|-|-|-|-| \n|MM-UPT|0.5317|0.5250|**0.5325**|0.5324|0.5259|0.5263  |0.5250| \n|Supervised GRPO|0.5329|0.5344|0.5330|0.5354| **0.5358**|0.5341|0.5239|\n\n---\n\n**Q4**. What is the relationship between \"training episodes\" and \"steps\"? How many rollout is done per training step?\n\n**A4**: Thank you for raising this clarification point. In our implementation, a training episode is a full pass over the unlabeled dataset. Within each episode, we perform one rollout group (G=10) per data point. These groups are used to compute pseudo-rewards and perform a single policy update step per batch. Therefore, **the total number of training steps equals the number of batches per episode times the number of episodes (e.g., 15 episodes × batches per episode)**. Besides, each training step processes a batch of examples sampled from the training dataset. In our experiments, the batch size is set to 128, so each training step involves **128 × 10 = 1280** rollouts. \n\n---\n\n**Q5**. Would dropping the temperature to 0 or using majority voting during inference be sufficient to achieve the same performance increase?\n\n**A5**: Thanks for your question and we conduct these follow-up experiments to help explain.\n\nFirstly, our results show that temperature=0 inference (using Qwen2.5VL, i.e., the base model) alone fails to match MM-UPT. This is because **temperature=0 yields a single, deterministic output that may reflect model biases or local optima, while majority voting aggregates multiple diverse samples to reduce randomness and improve robustness** [4].\n\nBesides, we find that **majority voting during inference does bring performance improvements, but it requires sampling multiple outputs per query, which is computationally expensive and impractical at scale**. Importantly, **MM-UPT allows us to achieve strong performance on pass@1, meaning that the majority voting benefits are baked into the model itself, without needing to rely on expensive sampling-based inference**, making the model more efficient and practical for deployment. Furthermore, we observe that applying majority voting at inference after MM-UPT training can lead to better performance, suggesting that MM-UPT improves both the model’s reasoning consistency and accuracy. \n\n| |We-Math|MathVista|MathVerse|MathVision|**Avg**|\n|-|-------|---------|---------|----------|-|\n|Base (pass@1 with temprature=0)|0.6471|0.6740|0.4353|0.2573|0.5034|\n|+ MM-UPT (pass@1)|0.6874|0.7290|0.4487|0.2615|0.5317|\n|Base (maj@10)|0.6948|0.7050|0.4802|0.2832|0.5408|\n|+ MM-UPT (maj@10)|0.7075|0.7310|0.4772|0.2836|0.5498|\n\n---\n\n**Q6**. Would pass@n decrease significantly as every response converges to the majority answer? Such trade-off should be clearly explained in the paper.\n\n**A6**: Thank you for the question. Encouraging consistency can indeed reduce diversity, lowering pass@n for large n. **This is a well-known trade-off in many RL methods no matter supervised or not [2,3]**. For example, [2] observes that while RL-trained models outperform their base models on pass@n at smaller values of n (e.g., n=1), base models achieve higher pass@n score when n becomes larger.\n\nTo further investigate this, we evaluate pass@10 (meaning a larger n) in our case. As shown below, MM-UPT leads to a modest drop in pass@10 compared to the base model. However, a similar drop is also observed with supervised GRPO, suggesting that this is a general consequence of RL training, rather than a specific flaw in MM-UPT. Importantly, **MM-UPT improves pass@1 significantly (i.e., at smaller values of n) according to Table 1 in our paper, showing that the model is becoming more accurate, not merely more deterministic**. \n\n|pass@10|We-Math|MathVista|MathVerse|MathVision|**Avg**|\n|-|-|-|-|-|-|\n|Base|0.9420|0.8730|0.7307|0.6556|0.8003|\n|+ MM-UPT|0.8621|0.8240|0.6477|0.5661|0.7250|\n|+ Supervised GRPO|0.9075|0.8570|0.6726|0.6164|0.7634|\n\n---\n\n[1] Cui et al. \"The entropy mechanism of reinforcement learning for reasoning language models.\" \n\n[2] Yue et al. \"Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?\" \n\n[3] Dang et al. \"Assessing diversity collapse in reasoning.\"\n\n[4] Wang et al. \"Self-consistency improves chain of thought reasoning in language models.\""}]

Author rebuttal:

Meta-review: The paper presents a simple method of performing a form of self play, wherein a policy rewards itself if multiple sampled trajectories agree via a majority vote. 

- While the reviewers agree that the idea shows improvements in the order of ~3-5 absolute percentage points across a few visual math-based benchmarks, the empirical evidence does not appear to match the claims made. 
- E.g. "a novel framework for continuous self-improvement without external supervision" is not backed up and is a *very strong* claim as even within their benchmarks, this method results in drops in some settings (this is also noted by reviewer QjgG) but the author's response that "this inconsistency is a natural and common phenomenon in self-improving settings" is directly at odds with the claims initially made.

If the claims are toned down to what the results actually show, then I would be ok with recommending acceptance as the execution of the idea has merits that other works could build off of, but in its current form - the paper promises more than it delivers.
Decision: Accept (poster)