Venue: NAACL 2025

# Vision-Language Models Can Self-Improve Reasoning via Reflection

Kanzhi Cheng\({}^{1}\)   Yantao Li\({}^{1}\)  Fangzhi Xu\({}^{2}\)    Jianbing Zhang\({}^{1}\),

Hao Zhou\({}^{2,3}\)   Yang Liu\({}^{2,3}\)

\({}^{1}\)National Key Laboratory for Novel Software Technology, Nanjing University

\({}^{2}\)Shanghai Artificial Intelligence Laboratory

\({}^{3}\)Institute for AI Industry Research (AIR), Tsinghua University

{chengkz,li_yantoao}@smail.nju.edu.cn fangzhixu98@gmail.com zjb@nju.edu.cn zhouhao@air.tsinghua.edu.cn liuyang2011@tsinghua.edu.cn

Correspondence to: Jianbing Zhang Equal contribution.Our code is available at [https://github.com/](https://github.com/) njucckevin/MM-Self-Improve.

###### Abstract

Chain-of-thought (CoT) has proven to improve the reasoning capability of large language models (LLMs). However, due to the complexity of multimodal scenarios and the difficulty in collecting high-quality CoT data, CoT reasoning in multimodal LLMs has been largely overlooked. To this end, we propose a simple yet effective self-training framework, \(R^{q}V\), which iteratively enhances the model's **V**ision-language **R**easoning by **R**eflecting on CoT **R**ationales. Our framework consists of two interleaved parts: (1) iteratively bootstrapping positive and negative solutions for reasoning datasets, and (2) reflection on rationale for learning from mistakes. Specifically, we introduce the self-refine and self-select losses, enabling the model to refine flawed rationale and derive the correct answer by comparing rationale candidates. Experiments on a wide range of vision-language tasks show that \(R^{q}V\) consistently improves multimodal LLM reasoning, achieving a relative improvement of 23% to 60% over GPT-distilled baselines. Additionally, our approach supports self-reflection on generated solutions, further boosting performance through test-time computation. +
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8876–8892

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8876–8892

## 1 Introduction

Humans often rely on intuitive Chain-of-Thought (CoT) to perform complex reasoning [4]. Previous studies have shown that this CoT capacity also emerges in Large Language Models (LLMs) [22]. Through simple prompting or fine-tuning [11, 12, 13], CoT enhances the reasoning performance of LLMs while providing insights into their decision-making process. Recently, OpenAI o1 further advances reasoning by producing long internal CoT sequences, taking LLMs intelligence to a new level.

While CoT reasoning has significantly advanced LLMs in textual domains, extending CoT to multimodal settings remains an open problem. Unlike the abundant, unsupervised text-based CoT in pre-training corpora [11, 22], multimodal CoT resources are scarce in the text-dominated internet collections [14], hindering the full realization of Multimodal LLMs' (MLLMs) reasoning potential.

Recent studies show that open-sourced MLLMs struggle to integrate visual cues into their reasoning process, resulting in weak CoT performance [15, 16]. Consistent with our observations in Figure 1, CoT prompting provides minimal gains over direct prediction [13] and falls far behind GPT-4o. One potential solution is to construct multimodal CoT annotations for post-training; however, manual annotation is prohibitively expensive and hard to scale. This raises our first research question: _can MLLMs self-improve the reasoning capabilities through bootstrapping on CoT samples?_

Orthogonal to fine-tuning on curated CoT annotations, relying solely on positive samples can

Figure 1: Results of Qwen-VL on TabMWP, a visual mathematical reasoning dataset. Qwen-VL exhibits weak zero-shot CoT reasoning performance, while our \(R^{q}V\) iteratively self-improves, surpassing the GPT-distilled baseline by a large margin.



lead to suboptimal policy due to insufficient exploration of reasoning paths. Inspired by human thinking, another promising direction involves learning from trial-and-errors Yuan et al. (2024); Song et al. (2024), where mistakes are not failures but key opportunities to enhance reasoning. A few multimodal approaches use corrupted prompts to create negative samples for preference learning, aiming to improve image comprehension Wang et al. (2023); Deng et al. (2024). However, these methods fail to generate reasoning-aligned positive and negative CoT solutions, making them unsuitable for complex multimodal reasoning tasks. Thus, it remains unaddressed: _how can MLLMs efficiently learn from mistakes to improve their reasoning skills?_

To address the above two questions, this paper proposes \(R^{3}V\), a self-training framework that enables the model to **R**eflect on bootstrapped CoT **Ra**tionales, thereby strengthening its **V**ision-Language **R**easoning. Firstly, we leverage MLLM's pre-existing but weak CoT ability to bootstrap both rationales and answers for a given question, enabling the collection of a large number of positive and negative solutions based on answer correctness. Secondly, we introduce a reflection mechanism on negative solutions to help the model learn from mistakes. Specifically, we design self-refine and self-select losses that guide the model to correct flawed rationales and derive the correct answer by comparing rationale candidates, respectively. The above synergistic process can be repeated, with improved samples boosting MLLM's reasoning and the enhanced model further improving rationale generation. Additionally, through self-select training, our model can derive the superior solution from multiple samples, further boosting performance via test-time computation.

We conduct experiments across a wide range of multimodal reasoning benchmarks, including charts, geometry, commonsense, science, mathematics, etc. \(R^{3}V\) progressively enhances the reasoning ability of MLLMs, delivering a 23%-60% relative accuracy improvement compared to GPT distillation, and consistently outperforming the strong self-training baseline, STaR Zelikman et al. (2022). Moreover, our test-time selection is robust and effective, consistently surpassing Pass@1 and majority voting, even in OOD scenarios.

Our main contributions are as follows:

* We introduce an iterative self-training framework \(R^{3}V\) that leverages CoT bootstrapped by MLLM itself for self-improvement. To our knowledge, this is the first attempt to apply self-training in vision-language reasoning.
* We propose learning from mistakes through self-reflection, with support for test-time computation to further improve reasoning performance.
* We perform extensive evaluations across 6 different multimodal domains to validate the effectiveness of \(R^{3}V\). Our analysis reveals the key factors driving the success of multimodal self-training.

## 2 Related Work

**Vision-Language Reasoning** Beyond the extensively studied unimodal reasoning Cobbe et al. (2021); Sun et al. (2023), multimodal reasoning has recently attracted significant interest as an essential part of human intelligence Yue et al. (2024); Lu et al. (2023). Although MLLMs perform well on general vision-language tasks Liu et al. (2024); Chen et al. (2024b), integrating visual cues into the reasoning process poses unique challenges, especially for open-source models Zhang et al. (2024a); Chen et al. (2024a); Sun et al. (2024b). Several studies have explored using rationale datasets to fine-tune models and enhance multimodal reasoning capabilities Gao et al. (2023); Zhang et al. (2024b) augmented existing mathematical datasets with rationales using GPT distillation, while Yang et al. (2024) enhanced performance through manually collected CoT annotations. In this work, we advocate for MLLMs to self-improve, reducing reliance on resource-heavy rationale annotations.

**Self-Training Methods** Self-training helps the model learn from its own generated outputs, reducing the need for labor-intensive human annotations Yuan et al. (2024); Chen et al. (2024c). Prior works have focused on enhancing the reasoning capacity of LLM. The typical approach involves sampling multiple rationales and filtering positive and negative solutions based on the answers. The LLM is then fine-tuned on the positive samples Zelikman et al. (2022); Hosseini et al. (2024); Yuan et al. (2023) or improved using preference learning Wang et al. (2024b); Mitra et al. (2024), such as DPO Rafailov et al. (2024). Recent advances have also extended self-training to agents Song et al. (2024) and neural symbolic Xu et al. (2024) scenarios. Video-STaR Zohar et al. (2024) is a recent video instruction tuning method that synthesizes question-answer pairs from labeled video datasets. In this paper, we pioneer the exploration of self 

training in vision-language reasoning, investigate the failure of DPO in multimodal settings, and address these challenges with our \(R^{3}V\) framework.

## 3 Methodology

Our self-training framework consists of two alternating components: (1) bootstrapping a large number of positive and negative CoT solutions for multimodal questions (Section 3.1); (2) using the above-sampled solutions to reflect on rationales and learn from mistakes (Section 3.2). This iterative process turns the MLLM from weak to strong. The overall framework is illustrated in Figure 2.

### Preliminaries

In visual-language reasoning, given an image \(I\) and a question \(x\), a multimodal large language model is required to integrate information from both the image and the question for reasoning, generating a CoT rationale \(r\) and then deriving the final answer \(a\). However, due to the difficulty in collecting high-quality rationale data, constructing large-scale \((I,x,r,a)\) pairs presents significant challenges. This hinders the enhancement of MLLM reasoning capacities through fine-tuning. To overcome this limitation, we propose leveraging the MLLM's pre-existing but weak CoT capability to iteratively augment \((I,x,r,a)\) pairs from the widely available vision question answering data \((I,x,a)\), enabling the model to self-improve.

Following STaR Zelikman et al. (2022), the MLLM self-training process involves iteratively fine-tuning on its self-generated rationale data. In each iteration \(t\), given a question \(x\) from training set \(\mathcal{D}=\{(I,x,\hat{a})\}\), the MLLM \(\mathcal{M}\) first generate a CoT rationale \(r\) along with an answer \(a\) , formulated as \(\{(r_{i},a_{i})\}_{i=1}^{|\mathcal{D}|}\). These intermediate outputs are then combined with the original training set, resulting in an augmented dataset that includes rationales:

\[\mathcal{D}_{r}=\{(I_{i},x_{i},r_{i},a_{i})\}_{i=1}^{|\mathcal{D}|} \tag{1}\]

Assuming that rationales leading to correct answers are of higher quality compared to those that do not, we can divide \(D_{r}\) into positive and negative sample sets based on the correctness of the answers:

\[\mathcal{D}_{r}^{+} =\{(I_{i},x_{i},r_{i},a_{i})~{}|~{}a_{i}=\hat{a}_{i}\}_{i=1}^{| \mathcal{D}|} \tag{2}\] \[\mathcal{D}_{r}^{-} =\{(I_{i},x_{i},r_{i},a_{i})~{}|~{}a_{i}\neq\hat{a}_{i}\}_{i=1}^{| \mathcal{D}|} \tag{3}\]

We then fine-tune model \(\mathcal{M}\) on the filtered positive CoT samples \(\mathcal{D}_{r}^{+}\) using supervised fine-tuning (SFT) with a negative log-likelihood objective:

\[\mathcal{L}_{SFT}=\underset{(I,x,y)\sim\mathcal{D}_{t}}{-\!\!\!\!\sum}\!\! \log\mathcal{M}(y~{}|~{}x,I), \tag{4}\]

where the \(y=(r,a)\) is the solution generated by the model. We continue repeating the above process, generating new rationales with the newly finetuned model, until performance plateaus.

### _R\({}^{3}v\)_: Reflection on Rationales

The above self-improvement process strengthens the model using positive solutions, while negative ones are typically discarded. However, negative samples comprise a large portion of the sampled solutions and offer valuable insights for further model enhancement An et al. (2023); Hosseini et al. (2024). In our preliminary experiments, we found that the noisy nature of CoT in multimodal scenarios leads to suboptimal performance when using DPO Rafailov et al. (2024). Inspired by the error-driven learning of humans, we introduce reflection on rationales, teaching the model to correct its own mistakes and reflect on multiple reasoning paths to identify the correct solution. Specifically, we propose additional self-refine (Section 3.2.1) and self-select (Section 3.2.2) losses for multitask learning. Our framework harnesses the continuous production of positive and negative samples in self-training, offering a robust and effective solution for learning from mistakes. Appendix E provides examples of different components in \(R^{3}V\).

#### 3.2.1 Self-Refine

Upon failing to solve a problem, human students will analyze the errors in their solutions and reflect on how to correct them. Inspired by this, we designed the self-refine mechanism to encourage the model to correct flaws in its generated solutions. Multiple positive and negative solutions sampled during self-training can be viewed as the model's repeated reasoning on the same problem, making them well-suited for self-refine training. Specifically, we construct dataset for self-refine as follows:

\[\mathcal{D}_{REF}=\{(I_{i},x_{i},y_{i}^{+},y_{i}^{-})~{}|~{}\exists y_{i}^{+}, y_{i}^{-}\}_{i=1}^{|\mathcal{D}|}, \tag{5}\]

where \(y_{i}^{+}\) and \(y_{i}^{-}\) are positive and negative samples obtained from preceding iterations. Next, the self-refine loss is employed to guide the model in 

correcting errors in its self-generated answers:

\[{\cal L}_{REF}=\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \! 



## 4 Experiments

In our experiments, we focus on a diverse and comprehensive set of vision-language reasoning tasks to demonstrate the effectiveness of \(R^{3}V\). We begin by outlining the benchmarks (Section 4.1) and experimental setup (Section 4.2), followed by the main results of \(R^{3}V\) on six widely used datasets (Section 4.3). We also evaluated the improvements achieved by our framework in out-of-distribution (OOD) scenarios (Section 4.4).

### Datasets

We validate our framework's self-improvement on six vision-language reasoning benchmarks, which require integrating visual information into complex, multi-step reasoning. Refer to Appendix B for detailed information of these benchmarks.

**TabMWP**[50]: A dataset for table-based math word problems requiring reasoning and numerical calculation.

**ChartQA**[69]: Focuses on reasoning and calculations within real-world charts.

**CLEVR-Math**[78]: Compositional reasoning over abstract figures.

**MiniWob**[104]: A widely-used multimodal web navigation benchmark requiring models to generate multi-step actions.

**GeoQA**[12]: A geometry problem benchmark requiring complex reasoning.

**M\({}^{3}\)CoT**[12]: A recently introduced dataset featuring multi-domain, multi-step multi-modal reasoning problems.

### Experimental Settings

We primarily compare our framework with three categories of methods to comprehensively assess its effectiveness. All experiments are conducted under the same parameters to ensure a fair comparison.

**Zero-shot Methods.** We evaluated the MLLMs' zero-shot performance under the direct prompt (where the model tends to provide an immediate answer [50]) and the CoT prompt using "Let's think step by step." GPT-40 was also chosen as a strong baseline for comparison.

**Supervised Fine-tuning Baselines.** Since the self-training requires existing \((I,x,a)\) datasets, we provide the results of fine-tuning MLLMs using direct prompts on these question-answer pairs. We also include a GPT distillation baseline, where GPT-40 annotates CoT rationales for a small subset of each dataset, and then the open-source MLLMs are fine-tuned for CoT reasoning.

**Self-Training Methods.** We employ the aforementioned GPT-distilled, warmed-up MLLM as the starting point for self-training, iteratively sampling

\begin{table}
\begin{tabular}{l|c|c c c|c|c|c|c} \hline \multirow{2}{*}{**Methods**} & \multicolumn{2}{c|}{**Is CoT?**} & \multicolumn{2}{c|}{**Logical and Numerical reasoning**} & \multicolumn{2}{c|}{**Agentie**} & \multicolumn{2}{c|}{**Geometry**} & \multicolumn{2}{c|}{**Multi-Domain**} \\  & \multicolumn{1}{c}{**TabMWP ChartQA**} & \multicolumn{1}{c}{**CLEVR-Math**} & \multicolumn{1}{c|}{**MiniWob**} & \multicolumn{1}{c|}{**GeoQA**} & \multicolumn{1}{c|}{**M\({}^{3}\)CoT**} & \multicolumn{1}{c}{**Avg**} \\ \hline GPT-4o & ✓ & 94.47 & 67.00 & 70.60 & 98.50 & 55.17 & 65.85 & 70.62 \\ \hline \hline \multicolumn{8}{c}{**Qwen-VL**} \\ \hline Zero-shot QA & ✗ & 29.50 & 38.56 & 17.32 & - & 15.14 & 31.28 & 26.36 \\ Zero-shot CoT & ✓ & 31.50 & 37.59 & 12.61 & - & 16.58 & 30.73 & 25.80 \\ \hline
* & 33.03 & 48.96 & 51.97 \\ GPT Distill & ✓ & 62.30 & 46.72 & 51.83 & 51.11 & 31.43 & 47.41 & 48.47 \\ \hline \hline \multicolumn{8}{l}{**Self-Train Based**} & & & & & & & & \\ STaR & ✓ & 77.84 & 53.60 & 61.45 & 78.22 & 34.08 & 50.47 & 59.28 \\ \(R^{3}V\) & ✓ & **83.27** & **57.36** & **68.81** & **82.89** & **39.25** & **54.66** & **64.37** \\ \hline \hline \multicolumn{8}{c}{**LLAVA-L5**} \\ \hline Zero-shot QA & ✗ & 17.66 & 13.04 & 19.04 &
- & 26.92 & 36.63 & 22.66 \\ Zero-shot CoT & ✓ & 15.33 & 8.39 & 13.87 &
- & 23.47 & 35.81 & 19.37 \\ \hline
* & 42.17 & 52.63 & 49.03 \\ GPT Distill & ✓ & 44.63 & 28.48 & 56.52 & 60.44 & 33.81 & 47.54 & 45.24 \\ \hline
**Self-Train Based** & & & & & & & & \\ StaR & ✓ & 56.67 & 33.44 & 73.46 & 76.00 & 41.25 & 54.06 & 55.81 \\ \(R^{3}V\) & ✓ & **59.30** & **33.92** & **79.01** & **80.11** & **45.76** & **56.08** & **59.03** \\ \hline \end{tabular}
\end{table}
Table 1: Main results on six vision-language reasoning benchmarks. _Is CoT?_ column indicates whether a CoT or a direct answer was generated. _Avg._ column reports the average performance across all tasks (- indicates MiniWob is not applicable to this setting and is excluded from the average). \(R^{3}V\) significantly improves upon the GPT-distilled baseline without additional annotation costs, and surpasses the strong baseline StaR by a large margin.



positive and negative rationales from training samples for continuous self-improvement. We then compare \(R^{3}V\) with the well-known self-training baselines STaR Zelikman et al. (2022), which iteratively fine-tunes on self-generated positive solutions for model improvement.

We use two established MLLMs, Qwen-VL Bai et al. (2023) and LLaVA-1.5 Liu et al. (2024), as base models for self-training. We sample three solutions per sample in each iteration by default. The total number of iterations for these tasks is set to 4-5, depending on the convergence speed. Further details can be found in Appendix C.

### Main Results

Table 1 presents the evaluation results of \(R^{3}V\) on various multimodal reasoning tasks, including logical and numerical reasoning, agentic tasks, geometry, and multi-domain scenarios. The evolution progress of self-training is illustrated in Figure 3.

Self-training effectively converts MLLMs from weak to strongOpen-source MLLMs struggle with complex vision-language reasoning tasks. CoT reasoning with the Let's think step by step prompt Zero-shot CoT proves ineffective, with performance even worse than direct prompting Zero-shot QA). In this situation, the self-training method leverages MLLMs pre-existing but weak CoT capabilities to bootstrap multimodal CoT data for self-improvement. This process progressively elevates MLLMs' CoT reasoning, as shown in Figure 3, taking it to the next level on top of the GPT-distilled baseline. As an example with Qwen-VL, our self-training framework \(R^{3}V\) delivers an average 32.8% relative performance improvement over the GPT-distilled baseline (48.47 \(\rightarrow\) 64.37). This result highlights the remarkable potential of MLLMs to enhance their reasoning capabilities through self-training on synthetic data.

\(R^{3}v\) further enhances self-training efficiency by learning from mistakesInstead of discarding valuable negative samples, our \(R^{3}V\) framework leverages carefully designed self-refine and self-select mechanisms to learn from negative solutions, surpassing the strong self-training baseline STaR by a large margin Average 59.28 \(\rightarrow\) 64.37 on Qwen-VL). As shown in Figure 3, \(R^{3}V\) demonstrates swift adaptation across different multimodal scenarios, achieving notably higher gains in the first iteration compared to the STaR baseline, highlighting the efficiency of our method. These results underscore the value of learning from mistakes in multimodal reasoning and demonstrate the effectiveness of our reflection-based methodology.

### Out-of-Distribution Evaluation

Beyond the success of the \(R^{3}V\) framework on in-domain benchmarks, we are curious whether its reasoning improvements can generalize to out-of-distribution OOD and more difficult vision-language tasks. To this end, we aggregated the CoT rationales self-generated by \(R^{3}V\) across in-domain benchmarks and constructed positive and negative pairs for continual training on Qwen-VL. For a fair

Figure 3: Comparison of the iterative self-training process between \(R^{3}V\) and STaR on Qwen-VL across four benchmarks. Full results are provided in Appendix F. \(R^{3}V\) demonstrates higher efficiency in evolution and superior final performance.

 

comparison, we also included a baseline that uses only GPT-distilled positive CoT annotations. We conducted evaluations on three challenging benchmarks: (1) **MMMU**[24], a multi-discipline dataset designed to evaluate various aspects of multimodal reasoning; (2) **MathVista**[30], which focuses specifically on mathematical reasoning in multimodal contexts; (3) **VCR**[75], a cognition-level visual understanding benchmark that requires reasoning based on common sense and visual content.

\(R^{3}v\) also strengthens multimodal reasoning in OOD scenarios.As shown in Table 2, after incorporating \(R^{3}V\)'s self-generated CoT reasoning data, Qwen-VL significantly outperforms both the zero-shot and GPT-distilled baselines. This demonstrates that the CoT annotations synthesized by our framework not only enhance MLLM in-domain reasoning but also generalize to OOD and more challenging vision-language tasks.

Test-time selection generalizes to unseen tasks.Somewhat surprisingly, we found that the test-time selection ability does generalize to unseen tasks. For example, on MMMU, sampling three times during inference combined with our self-select mechanism (see Section 3.2.3) led to further improvement (35.63 \(\rightarrow\) 38.48). This suggests that through our self-select training, the MLLM has learned to compare multiple reasoning paths, identify errors (e.g., recognition or calculation mistakes), and eliminate incorrect options to arrive at the correct answer.

## 5 Analysis

This section analyzes the key factors behind the success of the \(R^{3}V\), as well as the potential challenges of self-training in multimodal reasoning tasks.

### Ablation Studies

Reflection on self-generated CoT facilitates learning from mistakes.To validate the effectiveness of each part of our framework, we independently ablated the self-refine and self-select losses, denoted as _w/o self-refine_ and _w/o self-select_. As shown in Table 3, both self-refine and self-select play a crucial role in improving performance. This highlights the value of negative samples, while our \(R^{3}V\) framework's reflection mechanism (i.e., self-refine and self-select losses) serves as an effective method for learning from mistakes.

Iterative training process is crucial for self-improvement.Next, we ablated iterative training as _w/o iteration_: instead of iteratively sampling and training, we sampled a large batch at once. For example, iterative self-training samples three times per round over four rounds, while _w/o iteration_ samples \(3\times 4=12\) times in a single pass. This approach is similar to Rejection Sampling Fine-tuning (RFT; Yuan et al. (2023)), but includes our self-refine and self-select losses. The results in

Figure 4: Performance comparison of different test-time methods. Our test-time selection is robust and effective, consistently outperforming Test@1 and majority voting.

 

Table 3 demonstrate the importance of iteratation. Although _w/o iteration_ produces a large number of positive and negative samples (comparable to \(R^{3}V\) by our statistics), the progressive training process yields higher-quality, more diverse samples, which boosts self-training performance.

### Test-time Compute

Test-time self-selection boosts performance through sampling.One key advantage of \(R^{3}V\) framework lies in its capacity to enhance performance by scaling test-time computation: during inference, we sample multiple candidate solutions and apply self-select to choose the answer. Figure 4 compares self-selection with Test@1 and majority voting with a sample size of 3. Our self-selection method consistently outperforms Test@1 and majority voting across all tasks. While majority voting reduces noise by aggregating results, self-selection goes further by deeply comparing reasoning paths, eliminating incorrect options, and ultimately analyzing to reach the correct answer.

\(R^{3}v\) consistently benefits from the scaling of sampling size.An open question is the scalability of our test-time selection. We conducted experiments with Qwen-VL on the TabMWP and CLEVR-Math benchmarks, comparing the performance of self-select and majority voting as the sample size increases. As shown in Figure 5, scaling the sample size consistently improves the performance of test-time selection, achieving both higher efficiency and accuracy compared to majority voting. Due to limitations in input length and capability of current MLLMs, performance plateaus with excessive sample size, which we believe stronger base models could address.

Our self-training framework requires no manual annotation, instead synthesizing large-scale positive and negative CoT rationales through sampling, equipping the model with the capacity for self-reflection during reasoning. It also opens up new opportunities for boosting MLLM reasoning performance by scaling test-time computation.

### The Noisy Nature of Multimodal CoT

In our preliminary study, we found that the widely-used preference learning method DPO Rafaulov et al. (2024) struggles to leverage positive and negative solutions for further improvement in multimodal settings. As shown in Table 4, equipping STaR with DPO training yields minimal improvement and falls short of our \(R^{3}V\).

To investigate DPO's failure, we closely examined the positive and negative samples self

Figure 5: Comparison of scalability between test-time selection and majority voting.

\begin{table}
\begin{tabular}{l|c c c|c|c|c|c} \hline \multirow{2}{*}{**Methods**} & \multicolumn{3}{c|}{**Logical and Numerical reasoning**} & \multicolumn{1}{c|}{**Agenti**} & \multicolumn{1}{c|}{**Geometry**} & \multirow{2}{*}{**Multi-Domain**} \\  & **TabMWP** & **ChartQA** & **CLEVR-Math** & **MiniWob** & **GeeQA** & \multicolumn{1}{c|}{**M\({}^{3}\)CoT**} \\ \hline STAR & 56.67 & 33.44 & 73.46 & 76.0 & 41.25 & 54.06 & 55.81 \\ STAR+DPO & 57.61 & 32.64 & 73.27 & 75.33 & 44.03 & 52.98 & 55.90 \\ \(R^{3}V\) & **59.30** & **33.92** & **79.01** & **80.11** & **45.76** & **56.08** & **59.03** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison between \(R^{3}V\) and the reinforced baseline (DPO). Due to the noisy nature of CoT in multimodal scenarios, the DPO method struggles to efficiently learn from mistakes and improve performance.

Figure 6: Proportion of correct rationales in solutions with correct answers. Multimodal CoT contains substantially more noise than text-based CoT.

 

generated by the MLLM (see details and case study in Appendix G). For each task, we randomly selected 100 positive solutions based on answer correctness and manually categorized their CoT fidelity as correct, partially correct, or incorrect. As shown in Figure 6, unlike natural language reasoning tasks (e.g., Logic, Math), multimodal CoT contains significant noise, with the proportion of fully correct CoT ranging from 8% to 70%. This stems from MLLM's limited recognition capabilities, leading to flawed CoT despite correct answers, such as OCR errors. As a result, faulty reasoning in noisy CoT is often misjudged as better solutions, making it challenging for DPO to distinguish between correct and incorrect reasoning paths and ultimately reducing performance Chowdhury et al. (2024). In contrast, our reflection method avoids encouraging the generation of faulty solutions, instead guiding the model to select the correct answer through elimination, demonstrating greater efficiency in noisy multimodal CoT scenarios.

### Generalization to Stronger Backbone

To demonstrate generalizability, we applied \(R^{3}V\) to the latest advanced MLLM, Qwen2-VL Wang et al. (2024a) , evaluating its ability to self-improve in solving geometric problems Chen et al. (2021). As shown in Figure 5(a), even without GPT-distilled warmup, \(R^{3}V\) achieves significant self-improvement by leveraging the model's pre-existing CoT abilities, demonstrating the \(R^{3}V\)'s generalizability across backbones. More impressively, we found that test-time selection demonstrates superior scalability on Qwen2-VL, markedly outperforming majority voting, as illustrated in Figure 5(b). We hypothesize that the enhanced general capabilities of the base model further amplify the effectiveness of self-select, which we leave for future exploration.

## 6 Conclusion

The scarcity of multimodal CoT data limits the reasoning capabilities of current MLLMs. In this paper, we take the first step toward enabling MLLMs to self-improve for better vision-language reasoning. We propose an iterative self-training framework, \(R^{3}V\), which continuously bootstraps positive and negative solutions and improves reasoning through reflection on self-generated CoT rationales. Meanwhile, \(R^{3}V\) enables MLLMs to self-reflect on their generated solutions, offering new opportunities for boosting performance through test-time computation. Extensive experiments and analyses demonstrate the effectiveness of our framework and the key factors behind its success.

### Limitations

As discussed in Section 5.3, due to the limitations of current MLLMs, the CoT annotations generated by \(R^{3}V\) often contain noise. While our framework can self-improve performance on noisy multimodal CoT, we believe that higher-quality CoT will further enhance reasoning ability. Due to computational constraints, our main experiments were conducted on two well-known MLLMs, LLaVA and Qwen-VL. Expanding to larger and more advanced MLLMs could yield interesting results, which we plan to explore in future work.

#### Acknowledgments

This work is supported by the National Key R&D Program of China (2022ZD0160501) and the Natural Science Foundation of China (NSFC, Grant No. 62176115). We thank Yihan Wang and Muye Huang for useful help.

Figure 7: Evaluation result of Qwen2-VL on GeoQA. (a) shows that our self-training approach significantly enhances performance without GPT-distilled warmup. (b) demonstrates the superior scalability of test-time selection, which boosts performance through sampling.

 



## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* An et al. (2023) Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, Jian-Guang Lou, and Weizhu Chen. 2023. Learning from mistakes makes llm better reasoner. _arXiv preprint arXiv:2310.20689_.
* Bai et al. (2023) Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. _arXiv preprint arXiv:2308.12966_.
* Chen et al. (2022) Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. 2022. UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 3313-3323, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Chen et al. (2021) Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. 2021. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. _arXiv preprint arXiv:2105.14517_.
* Chen et al. (2023) Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. 2023. Shikra: Unleashing multimodal llm's referential dialogue magic. _arXiv preprint arXiv:2306.15195_.
* Chen et al. (2024a) Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. 2024a. M\({}^{3}\)CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, Bangkok, Thailand.
* Chen et al. (2024b) Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024b. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. _arXiv preprint arXiv:2404.16821_.
* Chen et al. (2024c) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024c. Self-play fine-tuning converts weak language models to strong language models. _arXiv preprint arXiv:2401.01335_.
* Cheng et al. (2023) Kanzhi Cheng, Wenpo Song, Zheng Ma, Wenhao Zhu, Zixuan Zhu, and Jianbing Zhang. 2023. Beyond generic: Enhancing image captioning with real-world knowledge using vision-language pre-training model. In _Proceedings of the 31st ACM International Conference on Multimedia_, pages 5038-5047.
* Cheng et al. (2024) Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. _arXiv preprint arXiv:2401.10935_.
* Chowdhury et al. (2024) Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. 2024. Provably robust dpo: Aligning language models with noisy feedback. _arXiv preprint arXiv:2403.00409_.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, et al. 2021. Training verifiers to solve math word problems. _arXiv preprint arXiv:2110.14168_.
* Dai et al. (2023) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. _Preprint_, arXiv:2305.06500.
* Deng et al. (2024) Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. 2024. Enhancing large vision language models with self-training on image comprehension. _arXiv preprint arXiv:2405.19716_.
* Ericsson and Simon (1980) K Anders Ericsson and Herbert A Simon. 1980. Verbal reports as data. _Psychological review_, 87(3):215.
* Gao et al. (2023) Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. 2023. G-llava: Solving geometric problem with multi-modal large language model. _arXiv preprint arXiv:2312.11370_.
* Gulcehre et al. (2023) Caglar Gulcehre, Tom Le Paine, Srivaratsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddharn, Alex Ahern, Maiosen Wang, Chenjie Gu, et al. 2023. Reinforced self-training (rest) for language modeling. _arXiv preprint arXiv:2308.08998_.
* Gunjal et al. (2024) Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
* Han et al. (2023) Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023. Chartllama: A multimodal llm for chart understanding and generation. _arXiv preprint arXiv:2311.16483_.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. _arXiv preprint arXiv:2103.03874_.
* Hosseini et al. (2024) Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. 2024. V-star: Training verifiers for self-taught reasoners. _arXiv preprint arXiv:2402.06457_

Cheng-yu Hsieh, Chun-liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
* Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. _arXiv preprint arXiv:2210.11610_.
* Huang et al. (2024) Muye Huang, Lai Han, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu. 2024. Evochart: A benchmark and a self-training approach towards real-world chart understanding. _arXiv preprint arXiv:2409.01577_.
* Jia et al. (2024) Mengzhao Jia, Zhihan Zhang, Wenhao Yu, Fangkai Jiao, and Meng Jiang. 2024. Describe-then-reason: Improving multimodal mathematical reasoning through visual comprehension training. _arXiv preprint arXiv:2404.14604_.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International conference on machine learning_, pages 12888-12900. PMLR.
* Lindstrom and Abraham (2022) Adam Dahlgren Lindstrom and Savitha Sam Abraham. 2022. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. _arXiv preprint arXiv:2208.05358_.
* Liu et al. (2023) Fangyu Liu, Guy Emerson, and Nigel Collier. 2023. Visual spatial reasoning. _Transactions of the Association for Computational Linguistics_, 11:635-651.
* Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 26296-26306.
* Liu et al. (2020) Jian Liu, Leyang Cui, Hanneng Liu, Dandan Huang, Yile Wang, and Yue Zhang. 2020. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. _arXiv preprint arXiv:2007.08124_.
* Lu et al. (2023) Pan Lu, Hertik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannenhe Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. _arXiv preprint arXiv:2310.02255_.
* Lu et al. (2022) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2022. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. _arXiv preprint arXiv:2209.14610_.
* Masry et al. (2022) Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. _arXiv preprint arXiv:2203.10244_.
* Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. Orca-math: Unlocking the potential of slms in grade school math. _arXiv preprint arXiv:2402.14830_.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36.
* Shi et al. (2017) Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In _International Conference on Machine Learning_, pages 3135-3144. PMLR.
* Shi et al. (2024) Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. 2024. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. _arXiv preprint arXiv:2406.17294_.
* Song et al. (2024) Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. _arXiv preprint arXiv:2403.02502_.
* Sun et al. (2024a) Qiushi Sun, Zhriui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. 2024a. A survey of neural code intelligence: Paradigms, advances and beyond. _arXiv preprint arXiv:2403.14734_.
* Sun et al. (2024b) Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yan Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoulamaze Liu, et al. 2024b. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. _arXiv preprint arXiv:2412.19723_.
* Sun et al. (2023) Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. 2023. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration. _arXiv preprint arXiv:2310.00280_ 

Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. 2023. Evaluation and analysis of hallucination in large vision-language models. _arXiv preprint arXiv:2308.15126_.
* Wang et al. (2024a) Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024a. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. _arXiv preprint arXiv:2409.12191_.
* Wang et al. (2024b) Tianduo Wang, Shichen Li, and Wei Lu. 2024b. Self-training with direct preference optimization improves chain-of-thought reasoning. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 11917-11928.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Xu et al. (2024) Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. 2024. Interactive evolution: A neural-symbolic self-training framework for large language models. _arXiv preprint arXiv:2406.11736_.
* Yang et al. (2024) Zhen Yang, Jinhao Chen, Zhengxiao Du, Wenheng Yu, Weihan Wang, Wenyi Hong, Zhibuan Jiang, Bin Xu, Yunxiao Dong, and Jie Tang. 2024. Mathglm-vision: Solving mathematical problems with multimodal large language model. _arXiv preprint arXiv:2409.13729_.
* Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset requiring logical reasoning. _arXiv preprint arXiv:2002.04326_.
* Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. _arXiv preprint arXiv:2401.10020_.
* Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chenpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. _arXiv preprint arXiv:2308.01825_.
* Yue et al. (2024) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoiqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 9556-9567.
* Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. Star: Bootstrapping reasoning with reasoning. _Advances in Neural Information Processing Systems_, 35:15476-15488.
* Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_, pages 6720-6731.
* Zhang et al. (2024a) Renrui Zhang, Dongzhil Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. 2024a. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? _arXiv preprint arXiv:2403.14624_.
* Zhang et al. (2024b) Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhou Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. 2024b. Mavis: Mathematical visual instruction tuning. _arXiv preprint arXiv:2407.08739_.
* Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. 2023. Multimodal chain-of-thought reasoning in language models. _arXiv preprint arXiv:2302.00923_.
* Zhao et al. (2023) Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoj Dong, Jiaqi Wang, and Conghui He. 2023. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization. _arXiv preprint arXiv:2311.16839_.
* Zhou et al. (2024) Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated self-rewarding vision language models. _arXiv preprint arXiv:2405.14622_.
* Zohar et al. (2024) Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, and Serena Yeung-Levy. 2024. Video-star: Self-training enables video instruction tuning with any supervision. _arXiv preprint arXiv:2407.06189_.

 

Additional Related Work

**Multimodal Large Language Models and Multimodal Reasoning** Driven by the advancement of Large Language Models (LLMs), the multimodal research community has recently witnessed a domain shift from Vision-Language Models (VLMs) Radford et al. (2021); Li et al. (2022); Cheng et al. (2023) to Multimodal Large Language Models (MLLMs) Achiam et al. (2023); Liu et al. (2023); Chen et al. (2023); Cheng et al. (2024). Unimodal reasoning has a strong research foundation, such as in mathematics Hendrycks et al. (2021) and code generation Sun et al. (2024). Multimodal reasoning requires models to integrate visual cues into the reasoning process Zhang et al. (2023), presenting new challenges. Recent studies have explored synthesizing table or chart data and leveraging GPT to annotate CoT, aiming to enhance MLLM reasoning capabilities Han et al. (2023); Jia et al. (2024). For instance, Huang et al. (2024) utilizes GPT to generate chart code and render it to obtain diverse chart reasoning samples. In this work, we do not rely on stronger models to synthesize new reasoning samples; instead, we enable MLLMs to achieve self-improvement from self-generated CoT data.

**Self-Training Methods** Self-training, especially integrated with reinforcement learning from its own outputs, offers a promising avenue for model self-improvement Huang et al. (2022); Gulcehre et al. (2023). Recent studies have applied self-training to MLLMs with the goal of enhancing image comprehension, particularly in mitigating hallucinations Zhou et al. (2024); Gunjal et al. (2024); Zhao et al. (2023). Deng et al. (2024) proposes constructing positive and negative sample pairs by perturbing images and prompts, and enhances alignment through DPO training. In contrast, this work focuses on complex reasoning in multimodal scenarios, which requires integrating visual cues to generate step-by-step reasoning CoT. To our knowledge, we are the first to explore self-training in the context of vision-language reasoning.

## Appendix B Vision-Language Reasoning Benchmarks

TabMWP Lu et al. (2022) Tabular Math Word Problems (TabMWP) is benchmark containing open-domain grade-level problems that require mathematical reasoning and calculation on table figures. We use the standard train/test split provided by the author.

ChartQA Masry et al. (2022) We used the humanwritten version as the self-train benchmark, which contains more reasoning-intensive questions compared to the augmented split. This subset contains 7,398 chart figures and question pairs, comprising both free-text and multiple-choice questions.

CLEVR-Math Lindstrom and Abraham (2022) The CLEVR-Math dataset consists of multimodal math word problems that combine text and images, where questions are posed about the state of the scene after a sequence of actions (like addition or subtraction of objects) have been applied. We randomly sampled 10000 instances for training and used the original test set.

MiniWob Shi et al. (2017) MiniWeb asks MLLM to interact with a simulated Web environment. As shown in Figure 8, the model is provided with an image of the web interface along with the html as input. It is then asked to generate Python code to simulate keyboard and mouse actions and complete the given task.

GeoQAChen et al. (2021) GeoQA contains 4,998 multiple-choice geometric problems from Chinese middle school exams and annotated with solving programs. We use human translated English version provided by UniGe Chen et al. (2022).

\(\mathbf{M^{3}CoT}\)Chen et al. (2024a) M\({}^{3}\)CoT is a manually verified multimodal, multi-domain, multi-step visual-language reasoning dataset. We use the official train/test splits in our \(R^{3}V\) self-training process.

EvaluationFor structured outputs like GPT-distilled and self-train methods, we use the benchmark's default evaluation script to calculate metrics. For free-form outputs like the zero-shot CoT baseline, we employ GPT-4o-mini as the evaluator to assess accuracy. For MiniWob, the simulated web environment provides an automatic reward of

Figure 8: Example for MiniWob.

 

0 or 1, which we use to determine the solution's correctness.

## Appendix C Training Details

We use the Qwen-VL and LLaVA-1.5 as the base model and conducted experiments on an MLLM training infrastructure *. We show the number of training and testing samples for each dataset, along with the amount of GPT annotations in Table 5. Our self-training process begins with the GPT-distilled warmup, where we fine-tune the base model using the training dataset augmented with GPT-4o CoT annotations. After this warm-up, the fine-tuned MLLM is employed to sample from the training dataset to build SFT, Self-Refine, and Self-Select data for our training in the \(R^{3}V\) framework. We performed self-training in either four or five iterations, depending on performance saturates.

Footnote *: [https://github.com/TideDra/VL-RLHF](https://github.com/TideDra/VL-RLHF)

The same training hyperparameters are used across all experiments, as detailed in Table 6. We employ DeepSpeed to train MLLM using the Zero2 strategy, maintaining a global batch size of 64.

## Appendix D Algorithms

Algorithm 1 describes the overall process of \(R^{3}V\). The inner for-loop describes how we sample instances to build the proposed dataset, where we always select the most recent data. It is important to note that the sampled instances must be formatted as the data examples shown in Appendix E later.

## Appendix E Examples of \(R^{3}v\) Multi-task Learning

We illustrate examples of SFT, Self-Refine, and Self-Select in Figure 10. The input part in each subfigure shows the contexts such as question-choices pair and prompts used to guide the MLLM, and the output part shows the expected response from the MLLM. CoT prompt "Let's think step by step." will always be appended to the question-choices pairs. The prompts for self-refine and self-select vary slightly between multiple-choice and short-answer tasks. Note that only the self-select prompt will be used for test-time reflection.

We add the sample generated by MLLM into both self-refine and self-select contexts using "Model Prediction" to divide with the questionnaires part. As illustrated in the figure, we use a green checkmark to indicate the positive solutions and a red cross to mark the negative ones. It highlights that \(R^{3}V\) successfully builds negative-positive rationales pairs, from which the model can learn from mistakes in negative demonstrations. Additionally, \(R^{3}V\) also builds diverse reasoning paths, ranging from completely wrong to correct rationales for the MLLM learning to choose from like human.

## Appendix F Evolution Progress

Figure 11 shows the evolution progress of our \(R^{3}V\) framework.

## Appendix G Noisy Nature of Multimodal CoT

We manually reviewed the positive solutions generated by the Owen-VL in our self-training process and evaluated the quality of its CoT reasoning. The CoT error in multimodal setting is significantly higher than samples from logical reasoning datasets Liu et al. (2020); Yu et al. (2020) and math datasetsCobbe et al. (2021); Hendrycks et al. (2021) in natural language setting. Multimodal CoT has considerable noise, such as visual perception error and symbol misinterpretation. We highlight this issue with case studies on the M\({}^{3}\)CoT and GeoQA dataset in Figure 9.

\begin{table}
\begin{tabular}{l r c} \hline \hline
**Hyperparameter** & **Qwen-VL** & **LLaVA-1.5** \\ \hline Batch Size & \multicolumn{2}{c}{64} \\ LR & \multicolumn{2}{c}{3e-5} \\ Epochs & \multicolumn{2}{c}{3} \\ LR Schedule & Constant with warmup \\ LR Warmup Ratio & \multicolumn{2}{c}{0.1} \\ Weight Decay & \multicolumn{2}{c}{0} \\ LoRA Rank & \multicolumn{2}{c}{64} & 128 \\ LoRA alpha & \multicolumn{2}{c}{16} & 256 \\ LoRA Dropout & \multicolumn{2}{c}{0.05} \\ Optimizer & \multicolumn{2}{c}{AdamW} \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameter Settings

\begin{table}
\begin{tabular}{l l r r} \hline \hline
**Dataset** & **Train/Test** & **GPT Anno.** & **Iter#** \\ \hline TabMWP & 23059 / 7686 & 1000 & 4 \\ CLEVR & 10000 / 7955 & 1000 & 4 \\ ChartQA & 7398 / 1250 & 800 & 5 \\ MiniWeb & – & 550 & 4 \\ M\({}^{3}\)CoT & 7973 / 2359 & 936 & 4 \\ GeoA & 3499 / 754 & 536 & 5 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Dataset Statistics 

```
0: Training QA datasets \({\cal D}_{\text{QA}}\), subset with GPT-distilled CoT annotations \({\cal D}^{w}_{\text{SFT}}\), model \({\cal M}\), number of iterations \(T\)
1: Initialize \({\cal D}_{0}={\cal D}_{\text{QA}}\cup{\cal D}^{w}_{\text{SFT}}\) , \({\cal D}_{\text{pos}}=\emptyset\), \({\cal D}_{\text{neg}}=\emptyset\)
2:for each iteration \(t=1,2,\ldots,T\)do
3:\({\cal D}_{\text{SFT}}={\cal D}_{\text{REF}}={\cal D}_{\text{SEL}}=\emptyset\)
4:\({\cal M}_{t}\leftarrow\text{SFT}({\cal M},{\cal D}_{t-1})\)  # sample 3 times on training set
5:\({\cal S}\leftarrow\text{sample}({\cal M}_{t},{\cal D}_{\text{QA}},n=3)\)
6:\({\cal D}_{\text{pos}}\gets{\cal D}_{\text{pos}}\cup\text{eval\_pos}({\cal S },{\cal D}_{\text{QA}})\)
7:\({\cal D}_{\text{neg}}\leftarrow{\cal D}_{\text{neg}}\cup\text{eval\_neg}({ \cal S},{\cal D}_{\text{QA}})\)
8:for item index \(i=1,2,\ldots,|{\cal D}_{\text{QA}}|\)do
9:\(\text{$\#$ all solutions for sample $i$}\)
10:\(\text{$\textbf{d}^{i}_{+}$}=\left\{s^{i}_{1},s^{i}_{2},\dots,s^{i}_{k}|s^{i}_{ k}\in{\cal D}^{i}_{\text{pos}}\right\}\)
11:\(\text{$\textbf{d}^{i}_{-}$}=\left\{s^{i}_{1},s^{i}_{2},\dots,s^{i}_{k}|s^{i}_{ k}\in{\cal D}^{i}_{\text{neg}}\right\}\)  # the \(r\)-th latest pos/neg solution
12:\(s^{+/-}_{r}\leftarrow\text{$\textbf{d}^{i}_{\neq}[-r]$}\)  # SFT
13:if\(\exists~{}s^{+}_{1}\)then
14:\({\cal D}_{\text{SFT}}\leftarrow{\cal D}_{\text{SFT}}+s^{+}_{1}\)
15:endif
16:\(\text{$\#$ self-refine}\)
17:if\(\exists~{}(s^{+}_{1},s^{-}_{1})\)then
18:\({\cal D}_{\text{REF}}\leftarrow{\cal D}_{\text{REF}}+(s^{-}_{1},s^{+}_{1})\)
19:endif
20:\(\text{$\#$ self-select}\)
21:if\(\exists~{}(s^{+}_{1},s^{-}_{1},s^{-}_{2})\)then
22:\({\cal D}_{\text{SEL}}\leftarrow{\cal D}_{\text{SEL}}+(s^{+}_{1},s^{-}_{1},s^{-}_ {2})\)
23:endif
24:endif
25:\({\cal D}_{t}\leftarrow{\cal D}_{0}\cup{\cal D}_{\text{SFT}}\cup{\cal D}_{\text {REF}}\cup{\cal D}_{\text{SEL}}\)
26:endfor
```

**Algorithm 1**\(R^{3}V\)
 

* [15] M. C. B. K. Wilson, and R. J. S. Thorne, _Theoretical Computer Science_**2006**, 379-482 (1999)
* [16] A. D. Martin, P. F. Nason, and G. E. Hinton, _A Practical Guide to Machine Learning: An Introduction_. Cambridge University Press, Cambridge, MA, USA, 1999.
* [17] T. W. Tehgal, _Learning Representations for Neural Networks_. MIT press, Cambridge, MA, USA, 1999.
* [18] H. Zhang, Y. Li, and Z. Chen, _Deep learning with deep convolutional neural networks_. arXiv preprint arXiv:1503.03665v1 (2015)
* [19] L. Zhu, X. Liu, and Q. Sun, _Deep residual learning for image recognition_. IEEE Transactions on Image Processing, Man, and Cybernetics, vol. 10, no. 1, pp. 1-10, Jan. 2015.
* [20] I. Goodfellow, I. Guyon-Abadie, and J. Shlens, _arXiv preprint arXiv:1406.0366v1 (2014)
* [21] V. Khoslavi, A. Mirza, and A. Ozair, _Image classification via generative adversarial nets_. Advances in Neural Information Processing Systems, Vol. 1, No. 1, pp. 1-10, Jan. 2015.
* [22] A. Karpathy, S. Ramanan, A. Girshick, and J. Malik, _Very Large Scale Convolutional Networks_. Springer, New York, NY, USA, 2015.
* [23] A. Karpathy, S. Ramanan, A. Girshick, and J. Malik, _Deep Residual Learning for Deep Representation_. ACM, 2016.

[MISSING_PAGE_POST]

 

Figure 11: The evolution progress of \(R^{j}V\).