Diving into Self-Evolve Training for Multimodal Reasoning

Authors: Wei Liu, Junlong Li, Xiwen Zhang, Fan Zhou, Yu Cheng, Junxian He
Primary area: foundation or frontier models, including LLMs
Keywords: Large Multimodal Models, Large Language Models, Reinforcement Learning, Multimodal Reasoning, Self-Evolve

Abstract:
Reasoning ability is essential for Large Multimodal Models (LMMs). 
In the absence of multimodal chain-of-thought annotated data, self-evolving training, where the model learns from its own outputs, has emerged as an effective and scalable approach for enhancing reasoning abilities. 
Despite its growing usage, a comprehensive understanding of self-evolving training, particularly in the context of multimodal reasoning, remains limited. In this paper, we delve into the intricacies of self-evolving training for multimodal reasoning, pinpointing three key factors: $\textbf{Training Method}$, $\textbf{Reward Model}$, and $\textbf{Prompt Variation}$. We systematically examine each factor and explore how various configurations affect the training's effectiveness. Our analysis leads to a set of best practices for each factor, aimed at optimizing multimodal reasoning.
Furthermore, we explore the $\textbf{Self-Evolution Dynamics}$ during training and the impact of automatic balancing mechanisms in boosting performance. After all the investigations, we present a final recipe for self-evolving training in multimodal reasoning, encapsulating these design choices into a framework we call M-STAR ($\textbf{M}$ultimodal $\textbf{S}$elf-evolving $\textbf{T}$r$\textbf{a}$ining for $\textbf{R}$easoning), built on MiniCPM-V 2.5. 
M-STAR achieves 59.5% accuracy on MathVista, surpassing the pre-evolved model by 6.9% absolutely without using additional human annotations. 
We believe this study fills a significant gap in the understanding of self-evolving training for multimodal reasoning and offers a robust framework for future research. Our policy and reward models, as well as the collected data, will be released to facilitate further investigation in multimodal reasoning.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730308094995

thread_type:
official_review

<thread_object>
creator: reviewer_wmnx

<content>
summary:
The paper analyses the key components of self-evolving training procedure for Multimodal Large Language Models (MLLMs) with the aim of getting new insights into the reasoning capabilities of MLLMs. In particular, three fundamental aspects of self-evolving learning, i.e., training method, reward model, and prompt variation are examined in a series of experiments involving MiniCPM-V 2.5 model. Furthermore, the authors delve into the dynamics of the self-evolution process by means of monitoring four metrics, representative for the analysed process.

strengths:
1) The paper is clearly written, easy to follow, and its underlying theme is well motivated.
2) The experiments are well-designed and formally correct.
3) The considered topic is related to MLLMs, and as such is potentially of interest to the broad subset of the ICLR community.

weaknesses:
1) The main problem is limited experimental evaluation with respect to the number of MLLMs employed in the study. The authors present the results for one particular MLLM with not discussion about how the outcomes generalize to other models. This generalization is obviously a critical issue. 
2) Also, the authors consider only one dataset in their experiments. There are quite many datasets devoted to verifying abstract reasoning abilities of ML models, including MLLMs. Having a more diverse selection of these dataset (problem types) would be beneficial.

questions:
1) Are the presented observations/conclusions related to MiniCPM-V 2.5 also valid for other MLLMs? If so, what is the foundation of such a claim. 
2) A similar question regarding the validity of conclusions for other types of problems / other reasoning domains.
3) In Table 1 the results for In-Domain test samples are lower than those for OOD, which is surprising. What is the reason for a better OOD than ID performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730555082769

thread_type:
official_review

<thread_object>
creator: reviewer_pdvc

<content>
summary:
This work focus on self-evolving training for multimodal reasoning. They mainly research on three components in self-evolving training, that is training method, reward model and prompt variation. By comparing performance of various configuration, which can be considered as a kind of grid-search, they finally determine a set of optimal design. Also, along the process, they gave some in-depth analysis and insight of different topics.

strengths:
1. The layout of this paper is clear, the authors sperated the process into determination of three static components and stick to this layout by presenting each part in a reasonable order.
2. This work provide a comprehensive comparison of various configuration, which can serve as a reference for others working in this field. This work complement a study on self-evolving training method in multimodal reasoning area.
3. Along the process, the authors also provide some in-depth anasis in terms of diversed topic.

weaknesses:
1. This work can be considered as a kind of grid-search process, and doesn't proposed new techniques in terms of methodology.

questions:
1. Why is the order of focusing component be studied in the presented way? In the paper, authors first study in training method and determine a best configuration. Then they directly use this configuration for the study of reward model, finally the similar operation is performed to prompt variation. This way can be seen as a incomplete grid-search process, and the final optimal configuration could be different when they study in a different order. So it would be good for the authors to provide a reasonable explanation of this point.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730599482298

thread_type:
official_review

<thread_object>
creator: reviewer_rzp2

<content>
summary:
This paper explores self-evolving training for enhancing multimodal reasoning in large multimodal models (LMMs), focusing on training without chain-of-thought annotations. The authors investigate three primary factors influencing the effectiveness of this training approach: training methods, reward model design, and prompt variation. They introduce a dynamic framework, M-STAR, built on MiniCPM-V 2.5, to optimize these factors. Key contributions include establishing best practices for each component, implementing a reward model to enhance response selection, and proposing an automatic temperature adjustment mechanism to balance exploration and exploitation during training.

strengths:
1. Relevance: The paper addresses an important and timely problem—enhancing reasoning in large multimodal models without chain-of-thought annotations.

2. Quality: The paper demonstrates high technical quality in its systematic breakdown of training configurations, use of an adaptive exploration-exploitation strategy, and ablation studies.

3. Clarity: The paper is well-organized, with each section logically following from the last, making it easy to understand the flow from problem identification to solution.

weaknesses:
1. Claim: The CoT warm-up phase conflicts with the paper’s definition of annotation/CoT-free self-evolving training. It is essential to clarify this reliance and provide a stronger rationale for the warm-up setting.

2. Empirical results: The paper primarily evaluates MathVista, a single multimodal reasoning benchmark focused on math-based tasks. To show broader applicability, the paper would benefit from additional benchmarks such as VQA or other scientific QA. Besides, there is only one LLM used in all experiments, so it's hard to justify whether the benefit of the method can be transferred to other architectures.

3. Theoretical analysis: Since self-evolving training shares similarities with RL, a theoretical analysis comparing these methods could clarify the unique aspects of M-STAR, such as optimality, stability, and guarantees. Also, the paper lacks a hypothesis-driven structure that ties the findings to the central research question, which appears more like a tech report rather than a research paper. 

4. Contribution: While the paper has explored different configurations exhaustively, the overall contribution is vague given its lack of theoretical grounding and limitations in the empirical study.

questions:
1. Ablation studies showing the model’s performance with and without the CoT phase to quantify its impact on results.

2. What's the computation cost of the proposed method compared to baselines?

3. Can the best configurations explored in this study be applied to other task domains without losing their effectiveness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704379769

thread_type:
official_review

<thread_object>
creator: reviewer_tyha

<content>
summary:
The paper, "Diving into Self-Evolving Training for Multimodal Reasoning," explores the enhancement of reasoning abilities in Large Multimodal Models (LMMs) through self-evolving training, a method where models iteratively improve by learning from their own outputs. The absence of multimodal chain-of-thought annotated data has led to this innovative approach. The study identifies and systematically examines three critical factors—Training Method, Reward Model, and Prompt Variation—that influence the effectiveness of training. The authors present a comprehensive analysis and establish best practices for each factor within a newly proposed framework named M-STAR (Multimodal Self-evolving Training for Reasoning), built on MiniCPM-V 2.5. This framework achieved a significant improvement in accuracy on the MathVista dataset, demonstrating its efficacy. The paper also explores the dynamics of self-evolution and introduces an automatic mechanism to balance model exploration and exploitation, further enhancing performance.

strengths:
(1) The paper introduces an original framework, M-STAR, for self-evolving training in multimodal reasoning. This approach is particularly innovative as it leverages the model's own outputs for iterative improvement, a method relatively underexplored in the context of multimodal reasoning. Additionally, the focus on three specific components (Training Method, Reward Model, and Prompt Variation) for optimizing training presents a novel angle for investigation.

(2) The paper is well-structured and clearly written. The authors effectively communicate complex ideas, such as the dynamics of self-evolution and the implementation of an automatic balancing mechanism during training. The systematic breakdown of each key factor and the subsequent analysis make the paper accessible to readers with varying levels of expertise in the field.

weaknesses:
(1) The motivation or evidence behind the importance of the three components: the training method, the use of the reward model, and the prompt variation is insufficiently substantiated. The authors need to provide more detailed justification or empirical evidence to support the significance of these components in the context of multimodal reasoning.

(2) The ablation experiments are solely based on a single model: MiniCPM-V-2.5, and two datasets from the Math domain. It would be beneficial to explore the effects of different model sizes (understanding the constraints of increased training time with larger models, experimenting with smaller models could be insightful) and datasets from varied domains such as code generation to generalize the findings.

(3) The settings of the ablation studies focus primarily on minor hyperparameter adjustments, leading to conclusions that align with conventional expectations. It is recommended that the authors delve deeper into algorithmic comparisons. For instance, contrasting with techniques like Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or Differentiable Prompt Optimization (DPO), as well as exploring different training methodologies (e.g., multi-training stages) or network architecture designs (e.g., with different multimodal encoders) could provide more robust insights.

questions:
(1) Could the authors elaborate on the specific motivations or additional evidence that underscore the criticality of the training method, reward model, and prompt variation in enhancing multimodal reasoning? A deeper understanding or empirical backing could significantly strengthen the paper's foundation.

(2) Have the authors considered expanding the ablation studies to include a broader range of model sizes, including smaller ones, despite the acknowledged increased training time with larger models? Additionally, could the use of datasets from different domains, such as code generation, provide more comprehensive insights into the model's capabilities and limitations?

(3) In terms of algorithmic comparisons and training methodologies, could the authors provide a comparative analysis with other prevalent techniques like SFT, RLHF, DPO, or different training stages and network structures? Such comparisons could offer a clearer differentiation and possibly highlight the advantages or limitations of the proposed M-STAR framework in a broader context.

(4) In the section "MONITORING THE TRAINING DYNAMICS," it is observed that nearly all training reaches its peak performance quickly (within < 2500 steps), after which the model's performance tends to decline as training progresses. Does this suggest that the base self-evolving training configuration might not be optimally set? For instance, issues such as an overly small dataset, an excessively large model, or inappropriate regularization settings could be contributing factors. How do these factors influence the conclusions drawn from the training baseline, and could this impact the accuracy of the study's outcomes?

(5) Beyond assessing the correctness of results in the Math domain, should the evaluation of the model's outputs also consider other dimensions? For instance, evaluating aspects such as the interpretability, robustness, or even the creativity of the responses could provide a more holistic view of the model's capabilities in multimodal reasoning. How do the authors envision incorporating these additional evaluation metrics into their framework?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732380534335

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to the Reviewers (1/2)

comment:
We thank all the reviewers for the comments! We have revised the PDF to reflect the reviewers' comments, and responded to each reviewer separately in the respective thread. Here we summarize the main revisions of the manuscript.

Adding new models (Appendix F): We introduce two additional models of different sizes: **Phi-3.5-vision** (4B)[1] and **InternVL2-2B** (2B)[2]. To provide a more comprehensive analysis, we leverage the full results from MathVista to study various training recipes in greater depth. It is important to note that MathVista is not solely a benchmark for math word problems; it is a comprehensive multimodal benchmark encompassing a wide range of reasoning tasks, including **visual question answering**, **figure-based question answering**, **science question answering**, **and more**. The results are briefly shown below and the detailed results can be found in Appendix F. (addressing Reviewer tYHA, rZP2, wMNX)

Adding new benchmarks (Appendix G): To further validate the effectiveness of our **M-STAR** recipe, we evaluate it on four additional benchmarks: **M3CoT**[3], **MMStar-R**[4], **MMBench-R**[5], and **AI2D**[6] (-R indicates the the selected reasoning subset. The results are briefly shown below and detailed in Appendix G as well, demonstrating that M-STAR significantly enhances the multimodal reasoning abilities of models across different sizes and benchmarks. (addressing Reviewer  tYHA, rZP2, wMNX)

More comprehensive report on a breakdown of different datasets in MathVista (here we report MiniCPMV-2.5, full results of other models are in Table 5, Appendix F):

| **Model**            | **ALL**        | **FQA**        | **GPS**          | **MWP**          | **TQA**        | **VQA**        | **ALG**        | **ARI**        | **GEO**          | **LOG**        | **NUM**        | **SCI**        | **STA**        |
|----------------------|----------------|----------------|------------------|------------------|----------------|----------------|----------------|----------------|------------------|----------------|----------------|----------------|----------------|
| MiniCPMV-2.5         | 52.4           | 59.2           | 44.7             | 50.5             | 53.8           | 48.0           | 42.7           | 46.5           | 46.0             | **29.7**       | 36.1           | 56.7           | 60.1           |
|     +warmup          | 52.8           | 58.4           | 47.1             | 57.0             | 53.8           | 45.8           | 45.5           | 49.6           | 48.5             | 16.2           | 31.9           | 53.3           | 62.8           |
| SFT                  | 54.7           | 58.7           | 50.5             | 56.5             | 55.7           | 50.8           | 47.0           | 49.0           | 51.0             | 18.9           | 43.1           | 58.2           | 57.5           |
| Iterative RFT        | 55.7           | 59.1           | 49.5             | 65.6             | 55.1           | 48.0           | 47.3           | 53.8           | 50.6             | 16.2           | 37.5           | 55.7           | 65.1           |
| RestEM    | 55.1           | 58.0           | 49.5             | 64.5             | 55.1           | 47.5           | 47.7           | 53.8           | 50.2             | 16.2           | 38.2           | 56.6           | 63.5           |
| Cont. optim.         | 57.2           | 57.6           | 56.3             | 65.1             | 57.0           | 49.7           | 52.0           | 54.4           | 56.1             | 10.8           | 36.1           | 60.7           | 65.5           |
|     + **PRM Re-Rank**     | 59.2 (**+6.4**)| 59.1 (**+0.7**)| **61.1 (+14.0)** | **68.3 (+11.3)** | 55.1 (**+1.3**)| 51.4 (**+5.6**)| 54.8 (**+9.3**)| 55.2 (**+5.6**)| **60.3 (+11.8)** | 10.8 (**-5.4**)| 43.1 (**+11.2**)| 59.0 (**+5.7**)| 66.5 (**+3.7**)| 
| **M-STAR**           | **59.5 (+6.7)**| **59.5 (+1.1)**| 59.1 (**+12.0**)| 65.6 (**+8.6**)| **58.9 (+5.1)**| **54.2 (+8.4)**| **54.5 (+9.0)**| **56.7 (+7.1)**| 58.2 (**+9.7**)| 10.8 (**-5.4**)| **43.1 (+11.2)**| **61.5 (+8.2)**| **69.1 (+6.3)**|
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521753796

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Kind Reminder for Reading the Response

comment:
Dear Reviewers,

We have carefully revised the paper and included additional results to address the valuable feedback you provided. As the rebuttal period is coming to a close, we kindly request if you could review our responses to ensure they adequately address your concerns. We would greatly appreciate that!

Thank you,
The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733042218853

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Further Clarification on Our Motivation and Contributions

comment:
## **General Response for Motivation and Contributions**
	
We thank all the reviewers once again for taking the time to review our work and for providing valuable feedback. We appreciate your insights and suggestions. However, we believe there are still some areas of misalignment regarding the motivation and contributions of our work. To address this, we would like to provide further clarification and highlight the key points.
The motivation behind our work lies at the intersection of two key areas: **multimodal reasoning** and **self-evolving training**. However, we noted that our contributions to multimodal reasoning is not  fully recognized.
Below, we detail our contributions within each domain:

### **Multimodal Reasoning**

1. This is the first work to equip Large Multimodal Models (LMMs) with CoT reasoning for a broad set of multimodal reasoning tasks via self-generation. Given the scarcity of high-quality CoT data in MM reasoning, many popular open-source LMMs, such as LLaVa, MiniCPM, and InternVL, do not demonstrate CoT reasoning abilities for reasoning tasks – they often directly produce the answer even though we used CoT prompts in our preliminary experiments. 
2. We propose a novel and effective self-evolving training framework for multimodal reasoning, relying solely on the base model itself. This is the first work to study a complete self-evolving framework for multimodal reasoning blended with online learning, unlabeled prompts, and process reward models.
3. We train the first multimodal process reward model (PRM), which has not been explored in previous literature, and integrate it into self-evolving training. We will open-source all data collection and training pipeline

Finally, we emphasize that the effort behind this work is nontrivial. Unlike text-only LLMs, MM reasoning lacks adequate support and resources. We invested significant computation, effort, and iterative trials to develop a robust and effective pipeline in this domain, while also analyzing the impact of various contributing factors.

### **Self-Evolving Training**
1. Self-evolving training is effective but far from full explorations. So in this work, we
summarize key components for self-evolving training through the lens of RL and study them in a detailed manner and lead to a successful recipe.
2. We study the self-evolution dynamics from the “exploration & exploitation” trade-off and propose a novel dynamic monitor to enable adaptive exploration during the training process.

------

Overall, in this paper, we have contributed:  
- A pilot study to enhance multimodal reasoning validated by comprehensive studies,  
- Improved methodologies for self-evolving training,  
- Systematic investigations,  
- Constructed resources (e.g., CoT data, PRM and automatically annotated data).

Our empirical experiments substantiate the effectiveness of these contributions stey by step in detail.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes M-STAR, used for training multi-modal reasoning into language models. 
  * Assuming that the reader is familiar with basic RL-HF methods, the main new contributions consist of applying well-known techniques (e.g. Process-based Reward models, variants of self-training over correct synthetic data) over multi-modal benchmarks
  * Ablations were performed over different hyperparameter settings (e.g. sampling methods, temperature).
  * Results are shown for MathVista, although post-rebuttal, results over more multi-modal benchmarks and base models were provided. 

The core issue of this paper is that we don't find any new or surprising conclusions, other than validating that previous techniques also work for multi-modal reasoning. This issue is nearly unanimously agreed by the reviewers, and it's not clear how the setting being multimodal makes the RL training any different from regular text-only training, i.e. all the results are what we'd expect from general RL for LLMs.