Improve Vision Language Model Chain-of-thought Reasoning

## Abstract

Abstract Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. However, current training recipes lack robust CoT reasoning data, relying on datasets dominated by short annotations with minimal rationales. In this work, we show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses. To address this, we propose a two-fold approach. First, we distill rationales from GPT-4o model to enrich the training data and fine-tune VLMs, boosting their CoT performance. Second, we apply reinforcement learning to further calibrate reasoning quality. Specifically, we construct positive (correct) and negative (incorrect) pairs of model-generated reasoning chains, by comparing their predictions with annotated short answers. Using this pairwise data, we apply the Direct Preference Optimization algorithm to refine the model’s reasoning abilities. Our experiments demonstrate significant improvements in CoT reasoning on benchmark datasets and better generalization to direct answer prediction as well. This work emphasizes the importance of incorporating detailed rationales in training and leveraging reinforcement learning to strengthen the reasoning capabilities of VLMs.

## 1 Introduction

Chain-of-thought (CoT) reasoning is essential for improving the interpretability and trustworthiness of VLMs (Li et al., ; Liu et al., ; Chen et al., ; Liu et al., ; ; Bai et al., ). As VLMs are increasingly applied to more difficult tasks, the ability to reason through complex problems becomes essential. However, current training approaches for VLMs often rely on datasets dominated by short answers with limited rationales, which may restrict the models’ ability to generalize to tasks with comprehensive reasoning. In this work, we aim to address these limitations by providing distilled CoT data, introducing supervised finetuning (SFT) and reinforcement learning (RL) strategies to improve VLM reasoning performance.

An example in [fig. 1](#S1.F1) asks for the number of food items in a bar graph. When answering this question, a human would typically enumerate the bars and then calculate the total. However, writing out this enumeration process is far more cumbersome than simply providing the short answer of “14.” Consequently, the annotated training data is predominantly composed of short answers, with minimal rationale provided. This raises a critical research question: Does training on direct prediction implicitly teach the model to perform chain-of-thought reasoning to derive correct answers? Our findings indicate that after training on 26k direct predictions from ChartQA, the accuracy of direct predictions increased by 2.9 (70.2 to 73.1), while CoT prediction accuracy improved by only 0.6 points (71.2 to 71.8), with CoT under-performing direct prediction as a result. This suggests that current training approaches have limited effectiveness in enhancing CoT reasoning.

We hypothesize that developing CoT reasoning capabilities requires explicit training on data that includes detailed reasoning steps. To address the scarcity of high quality CoT reasoning data, we propose leveraging datasets with short ground truth annotations and employing the GPT-4o model to generate reasoning paths that lead to the correct answer. Our approach encompasses a diverse range of tasks, utilizing 9 datasets that demand different reasoning skills, including common world knowledge (A-OKVQA), chart interpretation (ChartQA), document information localization (DocVQA, InfoVQA), real-world text extraction (TextVQA), scientific reasoning (AI2D, SQA), and mathematical reasoning (MathVision, G-LLaVA). We distilled a total of 193k CoT examples for supervised fine-tuning (SFT) and the model, LLaVA-Reasoner-SFT, demonstrates significant improvements in VLM chain-of-thought reasoning performance.

In the lower part of [fig. 1](#S1.F1), we propose further refining SFT model reasoning through model-generated signals (Sun et al., ; Setlur et al., ). Specifically, the model generates multiple CoT steps to derive final predictions, which are then compared to the provided short annotation. Rationales that lead to correct predictions are more likely to be accurate, and vice versa. By optimizing positive (correct) and negative (incorrect) pairs of rationales with Direct Preference Optimization (DPO), we align the VLM toward more accurate reasoning process.
The aligned model, LLaVA-Reasoner-DPO, shows improved performance across all three domains as well as better out-of-domain generalization. Additionally, we demonstrate that DPO model can serve as a verifier to assign appropriate rewards, facilitating effective credit assignment (Rafailov et al., ; Lu et al., ).

Our contributions can be summarized as follows:
(A) We release a comprehensive CoT dataset ShareGPT-4o-Reasoning containing 193k examples, covering various VQA tasks.
(B) We demonstrate the effectiveness of SFT in improving CoT reasoning using this dataset.
(C) We show that reinforcement learning with DPO can further improve model reasoning using model-generated signals, without requiring additional human-labeled data.

Figure: Figure 1: The upper figure questions whether training exclusively on direct-answer datasets can effectively teach CoT prediction. In the lower figure, generating CoT for prediction provides the additional benefit of reasoning alignment, allowing the model to improve by leveraging self-generated data.
Refer to caption: /html/2410.16198/assets/x1.png

## 2 Related Work

#### VLM Reasoning

Previous work has evaluated the reasoning capabilities of vision-language models (VLMs) across various domains, including mathematics (Lu et al., ; Wang et al., ), college-level questions (Yue et al., ), and science (Kembhavi et al., ; Lu et al., ). Studies such as Zhang et al. (; ); Gao et al. () focus on training VLMs to generate step-by-step solutions for math problems or chart-based calculations, while Shao et al. () trains VLMs for chain-of-thought (CoT) reasoning in object localization tasks.

#### VLM/LLM Alignment

VLM alignment has utilized preference modeling techniques, such as Direct Preference Optimization (DPO)(Ouali et al., ; Deng et al., ; Yu et al., ; Li et al., ; Gunjal et al., ; Sun et al., ), and Proximal Policy Optimization (PPO)(Sun et al., ), to improve factual accuracy and reduce hallucination. To improve reasoning capabilties in LLM, Sun et al. (); Setlur et al. (); Lu et al. (); Pang et al. (); Xie et al. () use iterative or step DPO to improve math CoT reasoning capabilities.

Figure: Figure 2: Workflow diagram showing: a) the use of GPT-4o to generate rationale given short annotations; b) SFT of open-source VLM for CoT reasoning; c) Build preference dataset for reinforcement learning with DPO to enhance reasoning.
Refer to caption: /html/2410.16198/assets/x2.png

**Table 1: Data statistics of CoT distilled on different dataset**
| Dataset | Dataset Size |
| --- | --- |
| A-OKVQA | 16.9k |
| ChartQA | 26.0k |
| SQA | 6.1k |
| AI2D | 11.9k |
| InfoVQA | 22.4k |
| DocVQA | 37.3k |
| TextVQA | 29.7k |
| MathVision | 11.0k |
| G-LLaVA | 30.3k |
| Total | 193k |

## 3 Method

As shown in [fig. 2](#S2.F2), our pipeline consists of three stages: (A) CoT data distillation from GPT-4o ([section 3.1](#S3.SS1)), (B) SFT with CoT (and direct) data to enable VLM CoT reasoning, and (C) RL for further enhancement of CoT reasoning. The RL stage involves generating positive (correct) and negative (incorrect) reasoning data pairs sampled from SFT, as detailed in [section 3.3](#S3.SS3).

### 3.1 Reasoning Data Distillation

To mitigate the limited availability of high-quality CoT data, we leverage VQA datasets with short annotations and augment them with rationales generated by the GPT-4o model. We collect 193k visual CoT instances to create the ShareGPT-4o-Reasoning dataset, which we plan to release for public use. We focus on the following reasoning types as demonstrated in [fig. 4](#S3.F4):

Figure: Figure 3: The distribution of word counts for CoT and direct answer.
Refer to caption: /html/2410.16198/assets/x3.png

Real-World Knowledge includes the A-OKVQA dataset (Schwenk et al., ), which covers a broad range of commonsense reasoning and real-world knowledge for answering questions.

Chart Understanding includes the ChartQA dataset (Zhang et al., ), which involves tasks like item comparison, counting, and numerical computation.

Document Understanding/Real-World Text includes DocVQA (Mathew et al., ), InfoVQA (Mathew et al., ), and TextVQA (Singh et al., ), focusing on information localization and extraction in industrial documents and real-world image comprehension.

Math and Science includes MathVision (Wang et al., ), G-LLaVA (Gao et al., ), SQA (Lu et al., ), and AI2D (Kembhavi et al., ), focusing on scientific knowledge and mathematical reasoning.

After distillation, we filtered out examples whose answer predicted by GPT-4o is different from ground truth. The data statistics are presented in [table 1](#S2.T1), and a comparison of answer lengths is shown in [fig. 3](#S3.F3), highlighting that CoT responses peak around 100 tokens, while direct answers are typically under 5 tokens. The exact distillation prompt is provided in [appendix A](#A1).

Figure: Figure 4: Distillation of examples from various VLM task domains, highlighting the specific reasoning capabilities required.
Refer to caption: /html/2410.16198/assets/x5.png

### 3.2 Supervised Fine-Tuning for Chain-of-Thought Prediction

We choose LLaMA3-LLaVA-NeXT-8B as our base architecture, whose weight is initialized with the Open-LLaVA-NeXT weights(^1^11https://github.com/xiaoachen98/Open-LLaVA-NeXT). To ensure the model handles both direct and chain-of-thought (CoT) predictions, we implement two types of prompts during training.

Direct Prediction: For direct prediction tasks, we use the prompt “Answer the question with a short answer” for short-answer questions, and “Answer with the option’s letter from the given choices directly” for multiple-choice questions.

CoT Prediction: For CoT prediction tasks, we use the prompt “Generate a reason first and then output a letter answer” for multiple-choice questions, and “Generate a reason first and then output a short answer” for short-answer questions. In the model’s response, the rationale is followed by the answer, which is formatted as “### Answer: ” to enable answer extraction during evaluation.

### 3.3 Reinforcement Learning for Enhanced Reasoning

To further improve the quality of reasoning chains, we apply RL using the DPO algorithm to better align the model’s reasoning process toward more accurate predictions. The DPO algorithm requires both positive and negative responses. To generate these, we use the SFT model as the policy model (i.e., generator), producing 32 candidate predictions per question (temperature 1.0 for short answer and 1.2 for multiple-choice questions). Each prediction is compared with the ground truth to determine its correctness ([fig. 2](#S2.F2)). Following the approach in Dubey et al. (), we select instances with an accuracy between 0.25 and 0.85. From these, we randomly pair positive and negative responses, creating up to three pairs per question.

Formally, the dataset is denoted as $\mathcal{D}_{DPO}=\{(\mathcal{V},x,y_{w},y_{l})\}$, where $\mathcal{V}$ is the image, $x$ is the question, $y_{w}$ and $y_{l}$ are the positive and negative responses. The DPO objective is defined as below:

$$ $\mathcal{L}_{\mathrm{DPO}}\left(\pi_{\theta};\pi_{\mathrm{ref}}\right)=-\mathbb{E}_{\left(\mathcal{V},x,y_{w},y_{l}\right)\sim\mathcal{D}_{DPO}}\Bigg{[}\log\sigma\Bigg{(}\beta\log\frac{\pi_{\theta}\left(y_{w}\mid x,\mathcal{V}\right)}{\pi_{\text{ref }}\left(y_{w}\mid x,\mathcal{V}\right)}-\beta\log\frac{\pi_{\theta}\left(y_{l}\mid x,\mathcal{V}\right)}{\pi_{\text{ref }}\left(y_{l}\mid x,\mathcal{V}\right)}\Bigg{)}\Bigg{]}\,,$ $$

where $\pi_{\theta}$ is the policy model to be optimized and $\pi_{\text{ref }}$ is the base reference model, both models are initialized with SFT weights. $\sigma$ is the logistic function and $\beta$ is set to $0.1$.

**Table 2: SFT experiments with data composition in [fig. 5](#S3.F5): ① format alignment only, ② direct responses only, ③ CoT responses only and ④ both direct and CoT responses. Inference is performed using both direct and CoT templates. The best CoT prediction result is highlighted in orange, while the best direct prediction result is marked in blue. The results demonstrate that combining CoT and direct responses during training leads to the best performance across both types of prompts. Refer to [section 4](#S4) for detailed analysis.**
| Methods | Prompting | A-OK | ChartQA | DocVQA | InfoVQA | TextVQA | AI2D | SQA | MathVista | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-Next | direct | 85.8 | 70.2 | 75.7 | 37.7 | 68.2 | 71.5 | 75.4 | 39.3 | 65.5 |
| + Format ① | CoT | 84.3 | 71.2 | 67 | 34.9 | 62.2 | 67.4 | 74.4 | 40.3 | 62.7 |
| LLaVA-Next | direct | 86.4 | 73.7 | 78 | 45.4 | 71.9 | 78.8 | 91.5 | 43.2 | 71.1 |
| + Direct ② | CoT | 85.7 | 71.8 | 68.8 | 38.6 | 63.6 | 72.5 | 85.4 | 38.6 | 65.6 |
| LLaVA-Next | direct | 84.9 | 71.8 | 81.2 | 45.7 | 72.1 | 75.3 | 85 | 41.9 | 69.7 |
| + Cot ③ | CoT | 85.1 | 82.2 | 81.2 | 49.7 | 69.9 | 77 | 91.3 | 49.2 | 73.2 |
| LLaVA-Reasoner | direct | 85.4 | 76.1 | 82.9 | 50.6 | 73.1 | 79.4 | 90.4 | 44.3 | 72.8 |
| -SFT ④ | CoT | 86.2 | 83.0 | 81.8 | 51.6 | 71.1 | 78.5 | 92.7 | 50.6 | 74.4 |

Figure: Figure 5: The upper section displays the data sources used for the SFT experiments, while the lower section illustrates the data composition for model training.
Refer to caption: /html/2410.16198/assets/x6.png

## 4 SFT Experiments for Chain-of-thought Learning

In this section, we explore how SFT can enhance VLM reasoning by addressing two key research questions: (1) Can CoT reasoning be implicitly learned from short responses? and (2) How effectively can CoT be learned from GPT-4o distilled data? Additionally, we analyze the composition of CoT data across various reasoning capabilities and compare the performance of SOTA models with GPT-4o.

### 4.1 Training Setting

As shown in the upper part of [fig. 5](#S3.F5), we present the data composition for SFT. The training data includes CoT distillation (193k instances) from [table 1](#S2.T1) and corresponding short answers (193k). Additionally, for CoT data, we incorporate 16k visual math examples from G-LLaVA. To maintain general instruction-following capability as the base model, we include 2k randomly sampled instruction data from LLaVA pretraining  Liu et al. (). To ensure the SFT models can handle both direct and CoT prompts during inference, we sample a small set of format-aligned data—50 examples from each of the 9 datasets—resulting in 450 instances.

In the lower part of [fig. 5](#S3.F5), we outline the data composition for model training. Specifically, LLaVA-NeXT-Format ([fig. 5](#S3.F5) ①) serves as the baseline model, trained exclusively on format-aligned data to enforce the desired output format without learning any task-specific reasoning skills. In contrast, models in [fig. 5](#S3.F5) ② and ③ incorporate either direct or CoT datasets, enabling the model to be expert in one type of skill as well as following the both direct and CoT prompt styles. Finally, LLaVA-Reasoner-SFT ([fig. 5](#S3.F5) ④) represents the SFT model trained on both CoT and direct data, making it to be expert in both types of reasoning.

We use the LLaMA3-LLaVA-NeXT-8B architecture, initializing the weights with Open-LLaVA-NeXT. All Supervised Fine-Tuning (SFT) experiments are trained for 1 epoch with a learning rate of 5e-6 and a batch size of 32. The experiments are conducted on 8 H100 GPUs.

### 4.2 Evaluation Setting

We evaluate our method using a range of benchmark datasets, including A-OKVQA (Schwenk et al., ), ChartQA (Masry et al., ), DocVQA (Mathew et al., ), InfoVQA Mathew et al. (), TextVQA (Mathew et al., ), AI2D (Kembhavi et al., ), ScienceQA (Lu et al., ), and MathVista (Lu et al., ). We also conduct more evaluation on general datasets OCRBench (Liu et al., ), MMStar (Chen et al., ), and MMMU (Yue et al., ) in later sections.
The evaluation for A-OKVQA was implemented by us, while for the other datasets, we follow the evaluation protocols outlined in VLMEval (Duan et al., ).

For CoT evaluation, answers are extracted after the pattern "###Answer: " before sent to evaluation. More comparison with LLaMA3-LLaVA-NeXT-8B model is shown [appendix C](#A3) and evaluation on GPT-4o is shown in [appendix B](#A2).

### 4.3 Can reasoning be implicitly learnt from direct prediction?

[Table 2](#S3.T2) presents the performance of the models introduced in [fig. 5](#S3.F5). Since LLaVA-NeXT-8B training data contains very few CoT reasoning examples, CoT performance of ① lags behind direct prediction across most tasks. The only improvement is observed in ChartQA and MathVista with a modest gain of +1.0 in CoT performance, showing CoT is helpful for calculation related tasks.

When comparing model trained on direct only data (②) to that trained on format-aligned data (①), we observe an average gain of +5.6 in direct prediction accuracy (65.5 $\rightarrow$ 71.1) and a +2.9 improvement in CoT performance (62.7 $\rightarrow$ 65.6). Surprisingly, closer inspection of CoT performance in calculation-involved tasks, such as ChartQA and MathVista, reveals only marginal gains (+0.6 for ChartQA CoT) or even a performance drop (-1.7 on MathVista), which contrasts with the improvements seen on the two tasks in ①.
On text-rich tasks, positive gains (>1) are observed, with the most improvement seen in InfoVQA (+3.7). Significant gains are also evident in science-related tasks like AI2D (+5.1) and SQA (+11.0). Despite these improvements, CoT performance still trails behind direct prediction overall (CoT: 65.6 vs. direct: 71.1). This result suggests that training on direct only prediction may not effectively help with CoT prediction.

**Table 3: Effect of data composition on math reasoning. MV: MathVision, GL: G-LLaVA, MI: MathInstruct, MP: MathPlus.**
| Data Config | MathVista<br>(direct/CoT) |
| --- | --- |
| format only ① | 39.3/40.3 |
| MV | 41.0/43.4 |
| MV+GL | 43.2/44.9 |
| MV+GL+MP50k | 42.3/45.6 |
| MV+GL+MP100k | 43.0/44.9 |
| MV+GL+MI50k | 43.1/45.0 |
| MV+GL+MI100k | 43.7/46.3 |
| MV+GL+AI2D | 44.1/46.4 |
| MV+GL+SQA | 43.1/47.3 |
| MV+GL+ChartQA | 43.2/50.4 |

### 4.4 How Effective is CoT Reasoning Data?

When comparing the model trained on CoT-only data (③) with the one trained on format-aligned data (①), we observe improvements in both direct and CoT predictions. Direct prediction performance increases by an average of +4.2 (65.5 $\rightarrow$ 69.7), while CoT prediction improves significantly by +10.5 (62.7 $\rightarrow$ 73.2). Notably, the CoT performance of the model ③ surpasses its direct prediction (73.2 CoT vs. 69.7 direct). Significant gains are observed in calculation-intensive tasks like ChartQA and MathVista, with increases of +11.0 and +8.9 in CoT performance, respectively. Interestingly, for text-rich tasks such as DocVQA, InfoVQA, and TextVQA, the direct performance of model ③ (trained on CoT-only data) outperforms that of model ② (trained on direct-only data). This suggests that even for text-heavy tasks, reasoning processes, such as localizing information in documents or recognizing text in real-world scenarios, may benefit from CoT training. The skills learned from CoT training appear to generalize to direct prediction as well.

**Table 4: Effect of data composition on science related tasks.**
| Data Config | AI2D | SQA |
| --- | --- | --- |
| format only ① | 67.4 | 74.4 |
| AI2D | 76.3 | 76.6 |
| SQA | 66.9 | 90.4 |
| AI2D +SQA | 76.7 | 91.2 |
| AI2D +SQA +ChartQA | 77.4 | 91.4 |

When both CoT and direct data are combined (④), performance is further enhanced for both prediction types, with an average gain of +7.3 in direct prediction (65.5 $\rightarrow$ 72.8) and +11.7 in CoT prediction (62.7 $\rightarrow$ 74.4). This demonstrates that combining direct and CoT data yields the best overall performance. Interestingly, in model ④, for 3 out of 8 datasets (TextVQA, DocVQA, AI2D), direct prediction outperforms CoT prediction. We hypothesize that these tasks involve a significant proportion of concise fact extraction, where generating long-form CoT responses may not provide additional benefits or even hurts. Further validation of this hypothesis will be explored in future work.

### 4.5 Ablation Tests on Data Composition

#### Data Composition for Math.

In [table 3](#S4.T3), we examine the effectiveness of data composition on MathVista performance. We first include two visual math datasets: MathVision (MV) and G-LLaVA (GL). Including MV improves CoT performance by +3.1 over format only baseline ([fig. 5](#S3.F5) ①), while adding GL yields an additional gain of +1.5. Building on MV+GL, we incorporate several datasets that are potentially relevant to the task, including two math text-only datasets: MathPlus (MP) and MathInstruct (MI), two science datasets: SQA and AI2D, and ChartQA. Notably, ChartQA significantly boosts CoT performance (+5.5), while AI2D and SQA provide positive gains of +0.6 and +1.5, respectively. However, adding the math text datasets results in minimal improvement. Comparing inclusion of 100k MP vs 50k MP, more text data does not necessarily lead to better results. Therefore, we decided not to include them in training LLaVA-Reasoner-SFT.

#### Data Composition for Science Tasks with CoT Prediction.

In [table 4](#S4.T4), we evaluate the impact of data composition on science datasets, including AI2D and SQA. Our results show that combining SQA and AI2D provides additional gains on both datasets, indicating that they are mutually beneficial. Furthermore, adding ChartQA contributes positively to both datasets, with a notable improvement of +0.7 for AI2D.

**Table 5: Performance Comparison of GPT-4o, Cambrian-7b, and our SFT Model. For Cambrian, * indicates our replicated results, while others are adapted from Tong et al. (), ^† indicate CoT prompt used for evaluation. ‘Our-SFT’ refers to LLaVA-Reasoner-SFT.**
| Dataset | GPT-4o | Cambrian | Our-SFT |
| --- | --- | --- | --- |
|  | direct/cot | official | direct/cot |
| A-OK | 89.6/90.1 | 83.1* | 85.4/86.2 |
| ChartQA | 79.6/84.7 | 73.3 | 76.1/83.0 |
| DocVQA | 90.3/90.8 | 77.8 | 82.9/81.8 |
| InfoVQA | 72.4/72.8 | 45.7* | 50.6/51.6 |
| TextVQA | 78.1/75.4 | 71.7 | 73.1/71.1 |
| AI2D | 80.7/81.5 | 73.0 | 79.4/78.5 |
| SQA | 85.9/87.2 | 80.4 | 90.4/92.7 |
| MathVista | 54.8/63.4 | 49.0^† | 44.3/50.6 |
| OCRBench | 80.2/79.2 | 62.4 | 61.6/62.0 |
| MMStar | 55.1/64.7 | 50.3* | 51.6/54.0 |
| MMMU | 57.8/63.6 | 42.7 | 41.6/40.0 |
| Avg (of best) | 77.9 | 64.5 | 68.8 |

### 4.6 Comparing with SOTA model and GPT-4o

In [table 5](#S4.T5), we compare the performance of GPT-4o and a recent state-of-the-art model, Cambrian Tong et al. (). For GPT-4o, we include both direct and CoT predictions, following the prompt optimization steps outlined in Borchmann (), with the prompts detailed in [appendix B](#A2). For Cambrian, we report the numbers from Tong et al. () and replicated the results using the official checkpoint on MMStar, InfoVQA, and A-OKVQA. Specifically for Cambrian, CoT predictions were used for the MathVista dataset, while direct predictions were applied for the remaining datasets.

When compared to open-source models, GPT-4o outperforms on nearly all benchmark datasets, with the exception of SQA. Notably, significant improvements from CoT predictions are observed on tasks involving calculation or complex reasoning, such as ChartQA, MathVista, MMMU, and MMStar.

Cambrian-7B is trained on a dataset of 7 million open-source instruction-following examples. In contrast, our model, fine-tuned on fewer than 400k instruction examples, outperforms Cambrian-7B on most benchmark datasets, underscoring the effectiveness of incorporating CoT data. While we recognize the challenge of comparing against other models, such as One-Vision (Li et al., ), MiniCPM-V Yao et al. (), X-Composer Zhang et al. (), and InternVL Chen et al. (), due to differences in model architecture, training datasets, and evaluation pipelines, our primary focus is on studying the effectiveness of CoT learning rather than competing for state-of-the-art performance on visual-language tasks.

**Table 6: DPO experiment with LLaVA-Reasoner-SFT as the base policy model. We compare two DPO datasets: ⑤ RLAIF-V Yu et al. () and ⑥ our preference dataset comprising A-OKVQA, ChartQA, and math. The best CoT prediction is highlighted in orange. Our DPO dataset shows the better improvements in chain-of-thought reasoning.**
| Methods | Prompting | A-OK | ChartQA | DocVQA | InfoVQA | TextVQA | AI2D | SQA | MathVista | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-Reasoner | direct | 85.4 | 76.1 | 82.9 | 50.6 | 73.1 | 79.4 | 90.4 | 44.3 | 72.8 |
| -SFT ④ | CoT | 86.2 | 83.0 | 81.8 | 51.6 | 71.1 | 78.5 | 92.7 | 50.6 | 74.4 |
| LLaVA-Reasoner | direct | 85.6 | 76.1 | 83.1 | 50.7 | 73.3 | 79.6 | 91.1 | 44.1 | 73.0 |
| -RLAIF ⑤ | CoT | 86.7 | 83.0 | 82.4 | 50.8 | 71.4 | 79.1 | 92.9 | 50.8 | 74.6 |
| LLaVA-Reasoner | direct | 85.4 | 76.4 | 83.1 | 51.2 | 73.3 | 79.4 | 90.8 | 44.2 | 73.0 |
| -DPO-ours ⑥ | CoT | 87.0 | 84.2 | 82.7 | 52.7 | 71.5 | 79.5 | 92.6 | 52.1 | 75.3 |

## 5 RL Experiments for Enhanced Chain-of-thought Reasoning

In this section, we demonstrate the effectiveness of RL in further enhancing CoT reasoning. We employ the DPO algorithm, which is directly optimized using positive and negative pairs. By leveraging short-answer feedback ([section 3.3](#S3.SS3)), we construct preference pairs across three domains: A-OKVQA (real-world knowledge reasoning), ChartQA (chart interpretation), and math (MathVision and G-LLaVA). Although additional DPO data from other datasets could be incorporated, data scaling and balancing will be addressed in future work.

For the DPO dataset, we include 24.5k examples from ChartQA, 18.3k from A-OKVQA, and 22.0k from math domain, totaling 64.8k preference data pairs. We train LLaVA-Reasoner-SFT on this dataset using a learning rate of 5e-7, a batch size of 32, and for 1 epoch. We found an additional trick to truncate the responses up to 90 tokens to be helpful for DPO training. To compare the effectiveness of different DPO datasets, we include RLAIF-V Yu et al. (), which contains 80k DPO pairs representing the state-of-the-art dataset for aligning VLMs for reducing hallucinations.

### 5.1 Can DPO Calibrate Reasoning?

In [table 6](#S4.T6), we present the results of the DPO model optimized on top of LLaVA-Reasoner-SFT (④). Model ⑤ uses the SOTA RLAIF-V Yu et al. () data, while model ⑥ uses our dataset. We observe that Model ⑤ shows a slight improvement in both direct prediction (+0.2) and CoT prediction (+0.2), whereas model ⑥ demonstrates a greater improvement in CoT prediction (+1.1) with equal gains on direct prediction. Interestingly, though only 3 out of 8 datasets are selected to construct DPO pairs, gains are observed across 7 out of 8 datasets except for SQA with a slight decrease (92.9 $\rightarrow$ 92.6). These results suggest that DPO dataset constructed from model-generated rationales can effectively enhance reasoning accuracy and show generalization across tasks.

Figure: Figure 6: The figures illustrate the performance of the DPO model as a verifier on ChartQA, A-OKVQA, and MathVista. Compared to the model trained with RLAIF-V, the model trained on our reasoning data pairs consistently shows improvement in both best-of-N selection and weighted voting.
Refer to caption: /html/2410.16198/assets/x7.png

### 5.2 DPO as Verifier for CoT Reasoning Re-ranking

In [fig. 6](#S5.F6), we present the re-ranking results using the DPO model as a verifier, following the approach of Zhang et al. (); Hosseini et al. (); Lu et al. (). The DPO reward score is calculated as $\log\frac{\pi_{\text{dpo}}\left(y\mid x,\mathcal{V}\right)}{\pi_{\text{sft}}\left(y\mid x,\mathcal{V}\right)}$, where $\mathcal{V}$ represents the image, $x$ the question, and $y$ the candidate answer. We explore two re-ranking strategies: Best-of-N and Weighted Voting. A Majority Voting (or self-consistency) baseline is also included for comparison.

**Table 7: More DPO results on general evaluation benchmark datasets.**
| Methods | OCRBench | MMStar | MMMU | Avg |
| --- | --- | --- | --- | --- |
| SFT ④ | 62.0 | 54.0 | 40.1 | 52.0 |
| SFT+RLAIF ⑤ | 63.7 | 53.5 | 42.3 | 53.2 |
| SFT+DPO-ours ⑥ | 63.7 | 54.1 | 42.6 | 53.5 |

When trained with RLAIF-V data (⑤), the DPO model demonstrates improvements as both a generator and verifier on A-OKVQA, likely due to the dataset’s alignment with real-world images, which matches the nature of A-OKVQA. Interestingly, while model ⑤ does not show improvements as a generator on ChartQA, it still produces positive results in best-of-N re-ranking, indicating that the learned preferences can generalize across domains. However, weighted voting does not lead to any improvements, and no significant gains are observed in re-ranking for MathVision. In contrast, when trained with reasoning data pairs, LLaVA-Reasoner-DPO (⑥) shows improvements across both re-ranking metrics, underscoring the effectiveness of DPO on reasoning data pairs.

### 5.3 Additional DPO CoT Performance on General Datasets

Figure: Figure 7: The performance of the DPO verifier on the MMMU dataset with 988 multiple-choice questions. We observe that DPO trained on our reasoning dataset achieves consistent improvements in re-ranking metrics, while DPO trained with RLAIF does not show significant gains.
Refer to caption: /html/2410.16198/assets/x13.png

In [table 7](#S5.T7), we present the DPO CoT performance on OCRBench, MMStar, and MMMU. We observe that both DPO models outperform the SFT baseline, with our DPO model trained on CoT reasoning pairs showing slightly better results.

In [fig. 7](#S5.F7), we further explore the effectiveness of DPO on the MMMU dataset, which consists of challenging college-level subject questions. We provide re-ranking results for multiple-choice problems from the Dev+Val split (988/1050). First, the SFT model with self-consistency shows consistent improvements reaching 45.5 with 64 candidate votes. LLaVA-Reasoner-DPO, trained on reasoning data pairs, shows strong generalization on MMMU by excelling in both weighted voting and best-of-N voting during candidate re-ranking.
While the DPO model trained on RLAIF-V (⑤) improves CoT predictions, it does not achieve gains in the re-ranking metrics, indicating limitations in distinguishing correct from incorrect reasoning on more complex data. We hypothesize that, compared to ChartQA, the reasoning questions in MMMU are more challenging and span a broader range of subjects. The RLAIF-V dataset relies primarily on COCO image domain, which may not provide sufficient coverage, leading to weaker performance in re-ranking.

Figure: Figure 8: Credit assignment of the DPO model on a portion of the responses from the ChartQA and AI2D datasets. The DPO token-level reward is computed for each token, with the rewards normalized to have a mean of 0. Negative scores are highlighted in cool colors (blue), while positive scores are highlighted in warm colors (orange). We observe that the DPO model is particularly sensitive to the first mistakes or hallucinations introduced in the response.
Refer to caption: /html/2410.16198/assets/x15.png

### 5.4 DPO Credit Assignment

While the DPO model is trained on pairwise data, prior works (Rafailov et al., ; Lu et al., ) have shown that DPO policies can learn to predict token-level rewards from binary preference data. These experiments primarily focused on math reasoning with LLMs. In this work, we provide examples of credit assignment learned by the VLM DPO, as shown in [fig. 8](#S5.F8). The token-level DPO reward can be expressed as
$\log\frac{\pi_{\text{dpo}}\left(y_{i}\mid x,\mathcal{V}\right)}{\pi_{\text{sft}}\left(y_{i}\mid x,\mathcal{V}\right)}$, where $\mathcal{V}$ represents the image, $x$ the question, and $y_{i}$ the $i$-th token in the generated response. This reward reflects the relative confidence of the DPO model compared to the SFT model for a given token in a candidate response.

In [fig. 8](#S5.F8), negative scores are shown in cool (blue) colors, while positive scores are shown in warm (orange) colors, with rewards normalized to a mean of 0. On the left, we observe that the DPO model is particularly sensitive to errors during chart interpretation from the ChartQA dataset. For instance, when the response incorrectly lists “Lamb” as “Beef” in a chart reading task, the DPO model assigns a highly negative score to this mistake.

On the right, we present examples from the AI2D dataset. Here, a hallucination in the response, such as incorrectly stating that the left side of the moon is illuminated (the correct answer is the right side), receives a low score. Additionally, when external knowledge is required to correctly identify the moon’s phase as “Crescent” instead of “Gibbous,” the DPO model penalizes the incorrect “Gibbous” answer with a negative score.
This indicates that the DPO model is more sensitive to knowledge-based errors than the SFT model, explaining its superior performance on CoT reasoning tasks in datasets such as AI2D.

## 6 Conclusion

In this work, we aim to improve VLM CoT reasoning. First, we collect a CoT reasoning dataset ShareGPT-4o-Reasoning across a broad range of VQA tasks. We demonstrate that fine-tuning on this dataset significantly enhances reasoning performance. Additionally, we further improve these models using reinforcement learning with direct preference optimization, which strengthens their ability to reason and generalize to direct answer prediction tasks. Our results show that these approaches effectively enhance the reasoning capabilities of VLMs, paving the way for more robust and interpretable multimodal models.

## Appendix A ShareGPT-4o-Reasoning Data for VLM CoT Reasoning

### A.1 Prompt for GPT-4o Distillation

[Figure A.1](#A1.F1) and [fig. A.2](#A1.F2) illustrate the GPT-4o system (task) prompt and the GPT-4o distillation prompt. We employ the same prompt across all VQA datasets for data distillation. Specifically, the input to the prompt consists of an image, a question, and a short answer. The short answer serves as a reference for GPT-4o to generate a CoT reasoning followed by a final answer after ’### Answer’. We show a few more examples in the next subsections.

Figure: Figure A.1: GPT-4o system prompt for CoT distillation.

Figure: Figure A.2: GPT-4o prompt for CoT distillation.

### A.2 Filtering Mismatched Annotations in Distillation

In the GPT-4o prompt shown in [fig. A.2](#A1.F2), we treat the annotation as a reference answer and instruct GPT-4o to generate its own solution based on that reference. In [fig. A.3](#A1.F3) and [fig. A.4](#A1.F4), we illustrate cases where the GPT-4o-generated solution differs from the annotated answer. Upon human examination, we identified errors in the annotations. For example, in [fig. A.3](#A1.F3), there are issues such as incorrect text recognition (e.g., “dentist” misidentified as “heart”) and incorrect object identification (e.g., “beer” as “water”). In [fig. A.4](#A1.F4), the annotation errors involve incorrect calculations in the left figure and miscounting in the right figure.

To ensure consistency and avoid potential errors, we filtered out examples where the GPT-4o generated answer differs from the annotated answer. In ShareGPT-4o-Reasoning, we release the SFT CoT data along with the original distillation and filtered examples for reference.

Figure: Figure A.3: An example from the A-OKVQA dataset highlights cases where the annotated answer does not match the GPT-4o-generated answer. In these cases, the GPT-4o answers are correct, while the annotations contain labeling errors. In the left figure, the sign reads “dentist” (correctly identified by GPT-4o), and the answer should relate to ‘teeth,’ not ‘heart’ as in the annotation. In the right figure, the fridge contains beer, but the annotation incorrectly labels it as ‘water.’ Consequently, we filter out instances where the GPT-4o-generated answer does not match the annotated answers.
Refer to caption: /html/2410.16198/assets/x16.png

Figure: Figure A.4: Filtered examples from the ChartQA dataset are shown. In the left figure, GPT-4o correctly identifies ‘1917’ and ‘Sonic The Hedgehog’ and provides the correct summation, while the annotated answer incorrectly lists ‘204.41’, which is the value for ’Bad Boys for Life’ and is unrelated to the question. In the right figure, GPT-4o accurately ranks the numbers from highest to lowest, but the annotated answer incorrectly identifies ‘Spain’ as having the highest value, when it should be the third largest.
Refer to caption: /html/2410.16198/assets/x17.png

## Appendix B GPT-4o Evaluation and Prompt Optimization

In this section, we present the prompts used for GPT-4o on benchmark datasets, including both direct and Chain-of-Thought (CoT) predictions. Similar to the findings in Borchmann (), we observed that GPT-4o’s performance is highly sensitive to prompt phrasing. We explored several sets of prompts and selected the best-performing ones for reporting results. Specifically, we try to align our results with those reported in Li et al. (); Tong et al. (), Claude 3.5 Sonnet for Vision  (^2^22https://www.anthropic.com/news/claude-3-5-sonnet), among others.

#### Prompt Optimization

We follow the process outlined in Borchmann () to design effective GPT-4o prompts for the benchmark datasets. A random subset of 200 instances is selected as a development set to evaluate manually designed prompts. We manually inspect the predicted results and identify issues such as the model being overly cautious in declining answers, incorrect output formatting, or style mismatches with the ground truth labels. As an illustrative example, we detail the prompt optimization process using ChartQA, and apply similar techniques to the other datasets. Finally, we provide the prompts used for replicating our test results.

**Table B.1: Prompt optimization on ChartQA for direct prediction evaluated with relaxed accuracy.**
| # | Prompt | ChartQA (relaxed acc) |
| --- | --- | --- |
| 1 | {Question} | 2.7 |
| 2 | {Question}<br>Answer the question directly. | 32.3 |
| 3 | Answer the question. Do not write a full sentence, just provide a value.<br>Question: {Question} | 56.4 |
| 4 | Answer the question with following instruction:<br>1. Do not write a full sentence, just provide a value.<br>2. Don’t include any unit, i.e. 56 instead of 56 meters<br>Question: {Question} | 75.2 |
| 5 | Answer the question with following instruction:<br>1. Do not write a full sentence, just provide a value.<br>2. Don’t include any unit, i.e. 56 instead of 56 meters<br>3. Don’t include ’%’ sign, i.e. 56 instead of 56%<br>Question: {Question} | 80.3 |

We apply the prompts described in [table B.1](#A2.T1) to the development set and compare the predictions with the ground truth to optimize the prompts. Specifically, when using prompts #1 or #2, GPT-4o often generates full sentences instead of short answers. While prompt #3 produces a short answer, it often includes units or special tokens. To address this, we refined the instructions in prompt #4 by specifying that units should not be included in the final answer. This adjustment improved accuracy from 56.4 to 75.2. We also observed that the ground truth does not contain the % symbol, which could mismatch in evaluation, and we explicitly include this rule in prompt #5. Finally, we applied the tuned prompt to the test set, achieving an accuracy of $79.64$ reported in [table 5](#S4.T5).

**Table B.2: Prompt optimization on ChartQA for CoT prediction evaluated with relaxed accuracy.**
|  | System Prompt | ChartQA (relaxed acc) |
| --- | --- | --- |
|  | When provided with an image and a question, generate a rationale first and then derive an answer.<br>Your rationale should include detailed visual elements in order to derive the answer. |  |
| # | Prompt |  |
| 1 | Answer the question with following instruction:<br>1. Generate a rationale first and then derive an answer.<br>2. Don’t include any unit, i.e. 56 instead of 56 meters<br>3. Don’t include ’%’ sign, i.e. 56 instead of 56%<br>Question:<br>{question}<br># Output Format #<br><rationale><br>### Answer: <your answer> | 84.7 |
| 2 | Prompt #1, removing system prompt | 84.1 |

In [table B.2](#A2.T2), we first introduce output format instructions to guide GPT-4o in generating the correct CoT format, which aids in extracting the final answer. We reused the criteria from the direct prediction prompt to evaluate the results. Additionally, we found that including a system prompt typically leads to a 0.5-point increase in score across datasets, although it does not improve direct answer prediction. We hypothesize that the system prompt helps GPT-4o adhere more closely to the CoT output format. Finally, we applied the tuned prompt to the test set, achieving an accuracy of $84.72$ reported in [table 5](#S4.T5).

Following the prompt optimization steps outlined above, we provide the prompts used to replicate our GPT-4o test results in the next section.

### B.1 GPT-4o Prompts for Evaluation

[Table B.3](#A2.T3) and LABEL:tab:gpto_cot_prompts provide the optimized prompts for benchmark dataset evaluation. The tuning process does not garantee the prompt is optimal, but that roughly matches the reported value from previous papers Li et al. (); Tong et al. (), Claude 3.5 Sonnet for Vision  (^3^33https://www.anthropic.com/news/claude-3-5-sonnet), among others. We include the prompts for reference to replicate the GPT-4o results on benchmark datatsets.

**Table B.3: Prompts for direct prediction with GPT-4o on benchmark datasets.**
| Dataset | Prompt |
| --- | --- |
| A-OKVQA<br>AI2D<br>SQA<br>MMStar | Answer the question. Do not write a full sentence, just provide a letter choice.<br>question<br>{Question} |
| ChartQA | Answer the question with following instruction:<br>1. Do not write a full sentence, just provide a value.<br>2. Don’t include any unit, i.e. 56 instead of 56 meters<br>3. Don’t include ’%’ sign, i.e. 56 instead of 56%<br>Question: {Question} |
| DocVQA<br>TextVQA<br>InfoVQA<br>OCRBench | Answer the question. Do not write a full sentence, just provide a value.<br>Question: {question} |
| MathVista<br>MMMU | Answer the question. Do not write a full sentence, just provide a value or letter choice.<br>{question} |

**Table B.4: Prompts for CoT prediction with GPT-4o on benchmark datasets.**
| Dataset | CoT Prompt |
| --- | --- |
| system<br>prompt | When provided with an image and a question, generate a rationale first and then derive an answer.<br>Your rationale should include detailed visual elements in order to derive the answer. |
| A-OKVQA<br>AI2D<br>SQA<br>MMStar | Answer the question with following instruction:<br>1. Generate a rationale first and then derive an answer.<br>2. For your final answer, provide a letter choice.<br>Question:<br>{question}<br># Output Format #<br><rationale><br>### Answer: <your answer> |
| ChartQA | Answer the question with following instruction:<br>1. Generate a rationale first and then derive an answer.<br>2. Don’t include any unit, i.e. 56 instead of 56 meters<br>3. Don’t include ’%’ sign, i.e. 56 instead of 56%<br>Question:<br>{question}<br># Output Format #<br><rationale><br>### Answer: <your answer> |
| DocVQA<br>InfoVQA | # Objective #<br>You are provided with an image, a question. Your job is to generate a rationale first and then derive an answer.<br>###########<br># Question #<br>{question}<br>###########<br># Rationale Requirement #<br>1. Do not state an answer at the beginning. Explain descriptions of visual clue that help to derive the answer.<br>2. Conclude with ### Answer: <your answer><br>3. Your final answer should be a single word or phrase.<br>4. If possible, copy the answer from document. Don’t add or remove symbols, units, or titles.<br>###########<br># Output Style #<br><rationale><br>### Answer: <your answer><br>########### |
| TextVQA | # Objective #<br>You are provided with an image, a question. Your job is to generate a rationale first and then derive an answer.<br>###########<br># Question #<br>{question}<br>###########<br># Rationale Requirement #<br>1. Do not state an answer at the beginning. Explain descriptions of visual clue that help to derive the answer.<br>2. Conclude with ### Answer: <your answer><br>3. Your final answer should be a single word or phrase.<br>4. Output your answer in lower case.<br>###########<br># Output Style #<br><rationale><br>### Answer: <your answer><br>########### |
| OCRBench | Answer the question with following instruction:<br>1. Generate a rationale first and then derive an answer.<br>2. Your answer should be a single word or phrase.<br>Question:<br>{question}<br># Output Format #<br><rationale><br>### Answer: <your answer> |
| MathVista<br>MMMU | # Objective #<br>You are provided with an image, a question. Your job is to generate a rationale that logically derives an answer from the visual clues.<br>###########<br># Question #<br>{question}<br>###########<br># Rationale Requirement #<br>1. Do not state an answer at the beginning. Explain step by step logic to derive the answer.<br>2. Conclude with ### Answer: <your answer><br>###########<br>Example output style:<br><rationale><br>### Answer: <your answer><br>########### |

## Appendix C Baseline Evaluation

**Table C.1: Evaluation of VLM performance on benchmark datasets with direct and CoT inference.**
| Dataset | LLaVA-NeXT-8B | LLaVA-NeXT-Format |  |  |
| --- | --- | --- | --- | --- |
|  | direct | CoT | direct | CoT |
| A-OK | 85.9 | 44.5 | 85.8 | 84.3 |
| ChartQA | 68.6 | 52.8 | 70.2 | 71.2 |
| DocVQA | 78.4 | 57.1 | 75.7 | 67.0 |
| InfoVQA | 36.6 | 25.8 | 37.7 | 34.9 |
| TextVQA | 67.2 | 41.6 | 68.2 | 62.2 |
| AI2D | 73.0 | 70.0 | 71.5 | 67.4 |
| SQA | 77.4 | 75.8 | 75.4 | 74.4 |
| MathVista | 37.3 | 25.3 | 39.3 | 40.3 |
| OCRBench | 57.7 | 59.7 | 59.1 | 56.6 |
| MMStar | 47.8 | 45.7 | 44.7 | 46.7 |
| MMMU | 42.8 | 37.6 | 41.8 | 37.7 |
| Avg | 61.2 | 48.7 | 60.9 | 58.4 |

In this section, we provide evaluation details for our base model, which uses the LLaMA3-LLaVA-NeXT-8B architecture with weights initialized from Open-LLaVA-NeXT. We selected Open-LLaVA-NeXT weights because the data and training pipelines were fully available at the time of model development, allowing us to avoid reliance on the unreleased real user interactions referenced in Liu et al. (). The pretraining data for Open-LLaVA-NeXT consists of 1M image-text pairs, sourced from datasets such as ShareGPT4V, ALLaVA-Instruct-VFLAN-4V, DocVQA, SynDog-EN, ChartQA, DVQA, AI2D, and GeoQA+.

When evaluating LLaVA-NeXT-8B, we identified several issues, such as the inability to follow the CoT prompt, refusal to answer questions, and generating irrelevant reasoning. In [fig. C.1](#A3.F1), we present randomly sampled examples from LLaVA-NeXT-8B with a temperature setting of 1.0 on a ChartQA test case. These examples demonstrate the model’s difficulty in adhering to the CoT prompt. In the first example, the model declines to answer the question. In the second to fourth examples, the model provides an answer first, followed by an explanation, which doesn’t effectively use thought process to answer the question. In the final example, the model generates a descriptive response instead of reasoning through the question, ultimately failing to provide an answer. This illustrates the model’s inconsistent handling of the prompt structure.

[Table C.1](#A3.T1) presents the evaluation results for LLaVA-NeXT-8B. For CoT predictions, we use ChatGPT to extract a letter choice or short answer from the long-form model output, using the prompts shown in [fig. C.2](#A3.F2) and [fig. C.3](#A3.F3). However, due to LLaVA-NeXT-8B ’s inability to accurately follow the CoT format, its performance is significantly worse compared to direct predictions and our format-aligned model. For direct prediction, our LLaVA-NeXT-Format has similar performance as that of LLaVA-NeXT-8B.

In [fig. C.4](#A3.F4), we present the same example trained with our format-aligned data for CoT using only 450 examples. The model successfully follows the CoT format by verbalizing the thought process and providing a short answer after "### Answer:". This allows us to use a rule-based extractor to retrieve answers, which also improves CoT performance, as shown in [table C.1](#A3.T1). However, the example also demonstrates that, while our data induces the CoT process, the reasoning remains incorrect. Sampling 32 examples using the format in [fig. C.4](#A3.F4) resulted in 3 correct answers, yielding an accuracy of 9.4% for that case. As a result, only slight gain is observed on ChartQA for CoT prediction vs. direct prediction.

Based on the above experiments, we report LLaVA-NeXT-8B as the baseline in the paper to avoid the difficulty in CoT evaluation and answer extraction of LLaMA3-LLaVA-NeXT-8B baseline.

Figure: Figure C.1: Randomly sampled examples from LLaVA-NeXT-8B with temperature=1.0 for a test case in ChartQA reveal that the model struggles to effectively follow the CoT prompt. In Sample 1, the model refuses to answer the question. In Samples 2-4, the model generates an answer first, followed by an explanation. In the final sample, the model produces a description instead of reasoning through the question, without providing an answer.
Refer to caption: /html/2410.16198/assets/x18.png

Figure: Figure C.2: ChatGPT answer extraction prompt for multiple-choices questions.

Figure: Figure C.3: ChatGPT answer extraction prompt for short answer questions.

Figure: Figure C.4: Randomly sampled examples from LLaVA-NeXT-Format with a temperature setting of 1.0, evaluated on the same test case in ChartQA, show that after training on 450 format-aligned data, the model is able to follow the CoT prompt by verbalizing the thought process and providing a short answer.
Refer to caption: /html/2410.16198/assets/x19.png

## Appendix D Nearly Zero Data Learning for CoT Reasoning

**Table D.1: We study a self-taught reasoner with minimal CoT data (only 450 format-aligned examples). LLaVA-NeXT-Direct is used as the baseline, and our LLaVA-Next-STaR is trained with a rejection sampling method. The best CoT predictions are highlighted in orange, and the best direct predictions are highlighted in blue. Our rejection sampling method outperforms both CoT and direct prediction, with the exception of two data points.**
| Methods | Prompting | A-OK | ChartQA | DocVQA | InfoVQA | TextVQA | AI2D | SQA | MathVista |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-Next | direct | 86.4 | 73.7 | 78 | 45.4 | 71.9 | 78.8 | 91.5 | 43.2 |
| + Direct ② | CoT | 85.7 | 71.8 | 68.8 | 38.6 | 63.6 | 72.5 | 85.4 | 38.6 |
| LLaVA-Next | direct | 85.9 | 74.6 | 79.2 | 47.4 | 72.1 | 79.5 | 92.2 | 44.4 |
| -STaR | CoT | 85.9 | 77.9 | 75.8 | 44.0 | 25.1 | 76.6 | 86.8 | 42.0 |

In this section, we demonstrate how minimal CoT training data can enhance CoT reasoning capabilities. Specifically, we use only 450 CoT format-aligned examples alongside all available direct prediction data, with LLaVA-NeXT-Direct as the baseline. We apply rejection sampling fine-tuning (RFT) following (Sun et al., ; Setlur et al., ) to train a self-taught chain-of-thought reasoner, denoted as LLaVA-Next-STaR. From LLaVA-NeXT-Direct, we sample 32 CoT examples for each training instance and select those whose final predictions match the ground truth. Up to three positive examples are selected per question, resulting in a dataset of 260k RFT examples.

As shown in [table D.1](#A4.T1), RFT training improves both CoT reasoning and direct predictions overall, with the exception of two data points. Notably, TextVQA shows a significant drop in CoT performance, which we will explore further in future work. Notable (>3%) gain is observed on ChartQA, DocVQA, InfoVQA, AI2D and MathVista, and roughly 1% gain is observed on direct prediction on those datasets as well.

#### DPO Experiments

Prior to the RFT experiments, we conducted DPO experiments on the ChartQA dataset under the same conditions as described in [section 4](#S4). However, the improvements were modest, with a 72.3 (+0.5) gain in CoT prediction and a 74.2 (+0.5) gain in direct prediction. In contrast, RFT yielded a significant improvement, with 77.9 (+6.1) on CoT prediction and 74.6 (+0.9) on direct prediction. We hypothesize that for models with relatively weak CoT reasoning capabilities, RFT may be more effective in enhancing model performance, whereas DPO with preference modeling may be less impactful. We leave further analysis for future work.

## Appendix E SFT Ablation Experiments

**Table E.1: SFT Ablation Results: For each dataset, ‘-C’ indicates the inclusion of CoT data for training, and ‘-D’ indicates the inclusion of direct prediction data.**
| Methods | Prompt | A-OK | ChartQA | DocVQA | InfoVQA | TextVQA | AI2D | SQA | MathVista |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA-NeXT-Format | direct | 85.8 | 70.2 | 75.7 | 37.7 | 68.2 | 71.5 | 75.4 | 39.3 |
| cot | 84.3 | 71.2 | 67 | 34.9 | 62.2 | 67.4 | 74.4 | 40.3 |  |
| ChartQA-C+D | direct | 85 | 74.9 | 75.8 | 36.5 | 68.2 | 72.2 | 77.4 | 42.8 |
| cot | 84.4 | 81.7 | 69 | 32.2 | 63.3 | 68.6 | 74.9 | 41.7 |  |
| ChartQA-D | direct | 85.2 | 73.1 | 74.6 | 34.1 | 67.1 | 71.5 | 76.4 | 40.3 |
| cot | 84.3 | 71.8 | 62.4 | 31.8 | 58 | 66.3 | 74 | 35.5 |  |
| ChartQA-C | direct | 85.1 | 70.8 | 74.5 | 35 | 67.9 | 71.6 | 76.9 | 35.3 |
| cot | 84.9 | 81.4 | 67.2 | 32.2 | 61.5 | 68.8 | 76.6 | 40.1 |  |
| A-OK-C+D | direct | 86.2 | 69.2 | 75.4 | 37.7 | 67.3 | 70.7 | 77.5 | 38.8 |
| cot | 84.6 | 70.2 | 67.3 | 36 | 61.6 | 67.2 | 75.8 | 39.8 |  |
| A-OK-D | direct | 85.1 | 69 | 75.3 | 38.5 | 66.9 | 72.2 | 76.1 | 39.5 |
| cot | 84 | 67.7 | 66.5 | 34.8 | 61.1 | 68.4 | 76 | 39.9 |  |
| A-OK-C | direct | 84.4 | 69.4 | 75.8 | 37.4 | 67.9 | 69.2 | 77.3 | 34.6 |
| cot | 84.1 | 69.2 | 67.6 | 35.5 | 59.4 | 67.6 | 74.5 | 40.6 |  |
| DocVQA-C+D | direct | 85.5 | 69.5 | 80.7 | 40.4 | 68.8 | 72 | 77.5 | 41.1 |
| cot | 83.9 | 70.9 | 80 | 40.2 | 64.1 | 68.2 | 73.4 | 39.3 |  |
| DocVQA-D | direct | 85.5 | 66.5 | 77 | 39.1 | 68.2 | 70.8 | 76.3 | 41.9 |
| cot | 83.9 | 66 | 66.4 | 33.7 | 59.9 | 64.8 | 74.5 | 39.3 |  |
| DocVQA-C | direct | 85.2 | 69.1 | 79.1 | 37.5 | 68.5 | 72 | 76.7 | 33.8 |
| cot | 84.4 | 71.2 | 78 | 38.5 | 63.5 | 68.5 | 74.1 | 38 |  |
| InfoVQA-C+D | direct | 85.8 | 63.4 | 77.1 | 47.7 | 67.6 | 72.5 | 78.1 | 43.6 |
| cot | 85.3 | 65.4 | 72.6 | 47.5 | 62.4 | 69.4 | 74.6 | 37.8 |  |
| InfoVQA-D | direct | 85.7 | 56.7 | 75 | 45.4 | 67 | 72.5 | 77.5 | 42.8 |
| cot | 83.7 | 53 | 63.5 | 37.8 | 58.2 | 67 | 75 | 37 |  |
| InfoVQA-C | direct | 85.2 | 68.3 | 76.5 | 42.5 | 67.8 | 72.5 | 78.2 | 39 |
| cot | 83.7 | 63.4 | 71.1 | 46.3 | 59.9 | 67.4 | 74.3 | 37.6 |  |
| TextVQA-C+D | direct | 85.1 | 69.8 | 75.5 | 38.7 | 73 | 71.9 | 76.9 | 42.6 |
| cot | 84.6 | 68.9 | 70.5 | 36.3 | 70.9 | 67.6 | 76.6 | 36.1 |  |
| TextVQA-D | direct | 84.9 | 68.6 | 74.5 | 37.6 | 71.8 | 70.8 | 77 | 41.7 |
| cot | 84.4 | 63.3 | 64.2 | 33.2 | 64.2 | 66.1 | 73.6 | 38.2 |  |
| TextVQA-C | direct | 84.6 | 69.1 | 74.6 | 36.9 | 71.4 | 71.9 | 77.1 | 36.6 |
| cot | 84.7 | 68.2 | 69.5 | 36.9 | 70.3 | 67.8 | 75.1 | 37.1 |  |
| SQA-C+D | direct | 85.7 | 69 | 75 | 38.4 | 67.3 | 72.3 | 90.2 | 38.7 |
| cot | 83.1 | 71.2 | 66.5 | 35.6 | 58.9 | 66.9 | 90.4 | 40.8 |  |
| SQA-D | direct | 84.9 | 68.1 | 74.3 | 37 | 66.8 | 72.2 | 89.2 | 41.3 |
| cot | 83 | 68.4 | 67.5 | 33.8 | 62.1 | 68.7 | 81.9 | 39.8 |  |
| SQA-C | direct | 84 | 69.3 | 76 | 38.3 | 68.2 | 71.7 | 85 | 39.2 |
| cot | 82 | 69 | 65.3 | 34.4 | 58.3 | 66.6 | 88.8 | 39.4 |  |
| AI2D-C+D | direct | 85.2 | 69.6 | 75.8 | 39 | 67.6 | 78 | 78.4 | 40.1 |
| cot | 83.8 | 70.2 | 68 | 35.9 | 60.7 | 76.3 | 76.6 | 42.1 |  |
| AI2D-D | direct | 86.3 | 69.2 | 75.1 | 37.3 | 67.2 | 76.8 | 77.6 | 39.7 |
| cot | 82.7 | 67.6 | 66 | 33.7 | 61.4 | 71.7 | 74.4 | 38.3 |  |
| AI2D-C | direct | 84.4 | 69.6 | 75.9 | 37.7 | 68.2 | 75 | 76.3 | 39.1 |
| cot | 83.1 | 70.2 | 65.9 | 35.6 | 59.9 | 75.1 | 74.5 | 39.5 |  |
| math-C+D | direct | 85.3 | 68.5 | 75.5 | 37.8 | 67.3 | 71.7 | 77.4 | 42.7 |
| cot | 84.4 | 69.7 | 64.3 | 34.2 | 59.3 | 68.7 | 76.3 | 49 |  |
| math-C | direct | 85.2 | 68.1 | 75.6 | 38 | 67.4 | 72 | 77.5 | 40.5 |
| cot | 84.3 | 70.6 | 66.2 | 34.7 | 59.8 | 68.2 | 78.4 | 45.4 |  |
| math+ChartQA | direct | 85.3 | 70.9 | 75.7 | 36.8 | 67.8 | 71.7 | 78.3 | 41.9 |
| cot | 84.1 | 81.9 | 67 | 32.6 | 60.7 | 68.3 | 75.5 | 49.7 |  |
| LLaVA-Reasoner-SFT | direct | 85.4 | 76.1 | 82.9 | 50.6 | 73.1 | 79.4 | 90.4 | 44.3 |
| cot | 86.2 | 83.0 | 81.8 | 51.6 | 71.1 | 78.5 | 92.7 | 50.6 |  |

In LABEL:tab_apd:sft_ablation, we present additional ablation experiments on SFT across each dataset, using three settings: direct only, CoT only, and direct + CoT. Additionally, format-aligned data is incorporated during training to enable the model to follow the specific direct or CoT format during inference.

## Appendix F Additional DPO Experiments

**Table F.1: Truncating response length affects the final performance of DPO. No truncation leads to a decline in performance, while truncating to 90 tokens empirically yields the best results.**
| Data/Truncate Len | prompting | 70 | 90 | 110 | No Truncate | SFT baseline |
| --- | --- | --- | --- | --- | --- | --- |
| ChartQA | direct | 76.5 | 76.2 | 76.7 | 75.9 | 76.1 |
| CoT | 83.9 | 84.2 | 81.8 | 80.6 | 83.0 |  |
| A-OKVQA | direct | 85.2 | 85.2 | 85.3 | 85.1 | 85.4 |
| CoT | 86.7 | 86.9 | 86.3 | 85.7 | 86.2 |  |

#### Truncating Responses for DPO

In our initial experiments, we observed that truncating response length impacts the final performance of DPO. As shown in [table F.1](#A6.T1), no truncation results in a decline in performance, while truncating to 90 tokens empirically produces the best results. Consequently, we applied a 90-token truncation for the DPO experiments.

**Table F.2: Comparison of DPO with the RFT method. The upper part of the table presents the SFT baseline and the DPO model, while the lower part shows the ablation results of RFT trained on each of the A-OK, ChartQA, and math training datasets, as well as their combined results.**
| Methods | prompting | A-OK | ChartQA | MathVista |
| --- | --- | --- | --- | --- |
| SFT baseline | direct | 85.4 | 76.1 | 44.3 |
| CoT | 86.2 | 83.0 | 50.6 |  |
| LLaVA-Reasoner-DPO | direct | 85.4 | 76.4 | 44.2 |
| CoT | 87.0 | 84.2 | 52.1 |  |
| A-OKVQA | direct | 85.1 | 72.7 | 37.4 |
| -RFT | CoT | 87.7 | 0.0 | 32.5 |
| A-OKVQA | direct | 85.8 | 74.9 | 41.3 |
| -RFT+Format | CoT | 86.3 | 80.2 | 46.5 |
| ChartQA | direct | 85.4 | 75.0 | 42.6 |
| -RFT | CoT | 86.7 | 83.9 | 52.0 |
| ChartQA | direct | 85.9 | 75.8 | 44.4 |
| -RFT+Format | CoT | 85.5 | 83.4 | 50.6 |
| Math | direct | 85.3 | 76.0 | 32.4 |
| -RFT | CoT | 86.7 | 67.3 | 50.9 |
| Math | direct | 85.5 | 76.0 | 39.6 |
| -RFT+Format | CoT | 85.5 | 82.0 | 50.0 |
| Combined | direct | 85.3 | 75.4 | 37.8 |
| -RFT | CoT | 85.4 | 84.4 | 49.0 |
| Combined | direct | 85.0 | 75.5 | 43.0 |
| -RFT+Format | CoT | 86.6 | 83.1 | 47.1 |

#### DPO vs. RFT

Following [appendix D](#A4), we examine the impact of RFT and compare it to the DPO method.

In [table F.2](#A6.T2), for A-OKVQA, we observe that training with A-OKVQA RFT alone yields the best result for A-OKVQA; however, the model’s ability to generate short answers is entirely lost. When format-aligned data is added, there is a trade-off between performance on A-OKVQA and other datasets.

When the datasets are combined for training, we see improvements only on ChartQA, while performance on A-OKVQA and MathVista declines. This indicates that balancing RFT across datasets is challenging, especially when the SFT model already performs relatively well on basic tasks. In contrast, the DPO model demonstrates consistent gains across datasets, showing better generalization.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - What does format (1) "format alignment only" mean? How does it differ from (2) "direct responses only"? Does (1) involve finetuning on the same data as the other models?
- Similar to the last question, in Tables 3 and 4, which datasets are used to train the "format (1)" model?
- This method is bottlenecked by the performance of the teacher model (in this case, GPT-4o). In this sense, it is a distillation method rather than a method that could be used to advance the performance of any model (e.g. a frontier model).
soundness: 2
strengths: - The dataset of CoTs is likely to be useful in future work
- The results show that finetuning on CoTs is helpful in several tasks
- The results show marginal improvements from the DPO process
- There are several ablations on dataset composition and methods of training/evaluating the model, which are helpful
summary: This paper presents a method for training vision language models (VLMs) to output a chain of thought (CoT) before answering a question. The approach involves creating a dataset with CoTs generated by GPT-4o, finetuning an open-weight VLM on these chains of thought (and the associated answers), and further training the VLM using DPO, where the preferences come from the GPT-generated annotations. The first experiments evaluate the model trained by supervised finetuning on the CoT dataset to several baselines--the model trained w/o CoT, ablated versions where certain parts of the data are removed, and models from prior work. The later experiments evaluate the impact of the DPO procedure and compare it to prior work on applying DPO to VLMs.
weaknesses: - The improvements from DPO (in Tables 6, 7 and Figure 7) are quite small. (Are they statistically significant?)
- The related work section does not discuss the mentioned work in enough detail (and could probably benefit from inclusion of more related work). For instance, what are the differences between the method in this paper and methods in the prior work that finetunes VLMs on CoTs? The statement that "Shao et al. (2024) trains VLMs for chain-of-thought (CoT) reasoning in object localization tasks" does not seem entirely accurate; their CoTs may involve regions in the image, but they apply their models to VQA tasks.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: N/A
soundness: 2
strengths: A clear two-stage approach is proposed to improve the chain-of-thought reasoning and the final performance of vision-language models. The overall research goal and method are easy to understand.
summary: The paper focuses on enhancing chain-of-thought (CoT) reasoning in vision-language models (VLMs) to improve their interpretability and performance on complex tasks. The authors identify that current training methods often rely on datasets with short annotations and limited rationales, which restrict the models' ability to generalize to tasks requiring detailed reasoning.

To address these limitations, the authors propose a two-fold approach:

1. Data Distillation: They distill rationales from the GPT-4o model to enrich the training data, creating a comprehensive CoT dataset with 193,000 examples across various visual question answering (VQA) tasks.

2. Reinforcement Learning (RL): They apply reinforcement learning, specifically the Direct Preference Optimization (DPO) algorithm, to refine the reasoning abilities of the models by constructing positive (correct) and negative (incorrect) reasoning pairs based on model-generated outputs.
weaknesses: 1. The novelty in the proposed two-stage approach is quite limited, and similar strategies have been widely adopted in previous work. For the first stage that distills CoT reasoning chain from the teacher models GPT-4o, this kind of method has been implemented too many of times in different domains, different modality, and different purposes. For the second stage that prompt VLM to first generate the CoT solution and then check the correctness of the final answers by comparing with the ground-truth ones, it's first proposed in STaR [1] and has been widely extended in many following work. Although the authors leverage the DPO for further optimization, it is far from a shinning point to make the whole method novel. 
2. I doubt the scalability of the proposed method. For the first stage, it basically relies on a teacher model to provide supervision. For the second stage, it needs ground-truth annotation from existing supervised datasets, which makes the method less suitable for more open-domain and complex problems. 
3. The results in Table 2 and Table 6 are not impressive. The improvement is little and hard to tell whether it's because of distilling more GPT data than previous work. 


[1] STaR: Bootstrapping Reasoning With Reasoning. Eric Zelikman et al

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Given the concerns raised in the Weakness section, more discussion on each point summarized in lines 88-92 would help to showcase the novelty and contributions of the work. 

To be more clear, lines 88-92 claim the contribution is to (A) provide a GPT-generated dataset, (B) verify that this dataset can be used for SFT, and (C) use DPO after SFT through an approach similar to [3] is meaningful. This makes me feel that this work is more of a resource paper. I would like to learn more about the technical contributions.

Reference:

[3] RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold: https://arxiv.org/pdf/2406.14532
soundness: 3
strengths: 1. The overall design makes sense. As demonstrated by the benchmark results, this could be a possible choice for implementing vision-language applications.
2. The release of synthetic data generated by GPT-4o contributes to the VLM finetuning.
3. The failure analysis in this paper is insightful and potentially helpful for other VLM research.
summary: This work focuses on enhancing the CoT capabilities of VLMs to serve complex vision-language reasoning tasks. The proposed approach includes generating synthetic data, SFT for basic CoT capability, and DPO for further calibration of complex reasoning ability stages.

The method is overall reasonable. With results on 9 datasets, the authors demonstrate the effectiveness of the proposed method. The experiments are clear and the analysis is comprehensive. Overall, this work positively contributes to the applications that involve VLM reasoning.
weaknesses: The novelty is unclear.
- From the idea level, tuning for CoT ability is being actively explored in LLM-centric research (e.g., [1], [2], [3])
- For the overall design, the proposed method has recently been widely used. Including,
   - Step 1 directly leverages the commonly seen approach, i.e., knowledge distillation from a larger teacher model (GPT-4). (e.g., [4])
   - Two-stage tuning (SFT-RL) has been shown to be effective in improving reasoning ability in many LLM works, either through 
       - Pertaining (e.g., Llama-3) 
       - SFT on synthetic CoT data, then use RL to refine the LLM's reasoning abilities (e.g., [5]) 

Besides, one minor problem is missing related work in language-based reasoning. 
Specifically, "How does LLM research tackle the author-targeted challenges in pure language tasks?" This background discussion would be beneficial because this paper particularly studies the Vision-Language Model, which utilizes a vision encoder to project visual information onto language space; thus, it may share certain characteristics with its underlying LLM.

Reference:

[1] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. https://arxiv.org/pdf/2312.06585

[2] LogiCoT: Logical Chain-of-Thought Instruction Tuning. https://arxiv.org/pdf/2305.12147

[3] RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold: https://arxiv.org/pdf/2406.14532

[4] LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents. https://arxiv.org/pdf/2311.05437

[5] Tackling Vision Language Tasks Through Learning Inner Monologues. https://arxiv.org/pdf/2308.09970

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please answer my questions in “weaknesses” section.
soundness: 2
strengths: - The paper is well-written and thus easy to understanding.
- The motivation to equip VLMs with CoT capability is interesting.
- The two-stage method demonstrates improved accuracy across several ImageQA benchmark.
summary: This paper aims to equip vision-language models with the capability for chain-of-thought (CoT) reasoning. To achieve this, this paper first prompts GPT-4o to generate rationales as CoT training data for supervised fine-tuning of VLM. Then, Direct Preference Optimization (DPO) is used to optimize the VLM, where the positive data consists of generated rationales that lead to correct answer, while the negative data consists of generated rationales that lead to incorrect answer. The two-staged training method respectively improves the accuracy across several ImageQA benchmarks in both direct prompting and CoT prompting setting.
weaknesses: [Setting]
1. Lack of discussion on the reasons why the VLM lacks CoT reasoning capabilities, even when its LLM backbone possesses CoT capabilities. If the LLM backbone has already equipped with CoT capability, why does the VLM not retain this capability? On the other hand, if the LLM backbone itself lacks CoT capability, it raises the question of why improvements were not made directly to the CoT capabilities of the LLM.
2. Does llama3-8b have CoT capabilities (I ask this question because your base VLM is LLaMA3-LLaVA-NeXT-8B)? If not, why was the approach not to directly improve CoT capabilities within llama3-8b? If yes, why does llama3-8b seem to lack CoT capability here, given that its pre-training text-only corpus is 15 TB [1], which significantly larger than the image-caption pairs used for VLM pre-training?

[Method]
1. In the DPO optimization stage, how to ensure that the correctness of the predicted answer is genuinely due to the rationale provided by the model, rather than the VLM simply gets the correct answer from its parameteric knowledge and directly ignores the incorrect rationale?
2. What proportion of the VLM's correct answers are generated with an incorrect rationale?
3. The two-stage training design, in which knowledge distillation with a larger model is followed by reinforcement learning optimization, is similar to approaches in previous studies [2, 3]. It would be beneficial to clearly articulate how this training design differs from those outlined in prior works.


[1] The Llama 3 Herd of Models (arXiv 2408)

[2] Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (EMNLP 2022)

[3] RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering (ICLR 2024)