Improve Vision Language Model Chain-of-thought Reasoning

Authors: Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, Yiming Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Vision Language Model, Chain-of-thought Reasoning

Abstract:
Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. 
However, current training recipes lack robust CoT reasoning data, relying on datasets dominated by short annotations with minimal rationales. In this work, we first evaluate the CoT abilities of existing VLMs and show that training on short answers does not generalize well to reasoning tasks that require more detailed responses. To address this, we propose a two-fold approach. First, we distill rationales from GPT-4o model to enrich the training data and fine-tune VLMs, boosting their CoT performance. Second, we apply reinforcement learning to further calibrate reasoning quality by constructing positive (correct) and negative (incorrect) pairs of model-generated reasoning chains, based on the comparisons with annotated short answers. We then use the Direct Preference Optimization algorithm on this pairwise data to refine the model’s reasoning abilities. Our experiments demonstrate significant improvements in CoT reasoning on benchmark datasets and better generalization to direct answer prediction as well. This work emphasizes the importance of incorporating detailed rationales in training and leveraging reinforcement learning to strengthen the reasoning capabilities of VLMs.

Decision: reject
Reviewer scores: 6, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729840710800

thread_type:
official_review

<thread_object>
creator: reviewer_e36p

<content>
summary:
This paper aims to equip vision-language models with the capability for chain-of-thought (CoT) reasoning. To achieve this, this paper first prompts GPT-4o to generate rationales as CoT training data for supervised fine-tuning of VLM. Then, Direct Preference Optimization (DPO) is used to optimize the VLM, where the positive data consists of generated rationales that lead to correct answer, while the negative data consists of generated rationales that lead to incorrect answer. The two-staged training method respectively improves the accuracy across several ImageQA benchmarks in both direct prompting and CoT prompting setting.

strengths:
- The paper is well-written and thus easy to understanding.
- The motivation to equip VLMs with CoT capability is interesting.
- The two-stage method demonstrates improved accuracy across several ImageQA benchmark.

weaknesses:
[Setting]
1. Lack of discussion on the reasons why the VLM lacks CoT reasoning capabilities, even when its LLM backbone possesses CoT capabilities. If the LLM backbone has already equipped with CoT capability, why does the VLM not retain this capability? On the other hand, if the LLM backbone itself lacks CoT capability, it raises the question of why improvements were not made directly to the CoT capabilities of the LLM.
2. Does llama3-8b have CoT capabilities (I ask this question because your base VLM is LLaMA3-LLaVA-NeXT-8B)? If not, why was the approach not to directly improve CoT capabilities within llama3-8b? If yes, why does llama3-8b seem to lack CoT capability here, given that its pre-training text-only corpus is 15 TB [1], which significantly larger than the image-caption pairs used for VLM pre-training?

[Method]
1. In the DPO optimization stage, how to ensure that the correctness of the predicted answer is genuinely due to the rationale provided by the model, rather than the VLM simply gets the correct answer from its parameteric knowledge and directly ignores the incorrect rationale?
2. What proportion of the VLM's correct answers are generated with an incorrect rationale?
3. The two-stage training design, in which knowledge distillation with a larger model is followed by reinforcement learning optimization, is similar to approaches in previous studies [2, 3]. It would be beneficial to clearly articulate how this training design differs from those outlined in prior works.


[1] The Llama 3 Herd of Models (arXiv 2408)

[2] Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering (EMNLP 2022)

[3] RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering (ICLR 2024)

questions:
Please answer my questions in “weaknesses” section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730453208893

thread_type:
official_review

<thread_object>
creator: reviewer_tshb

<content>
summary:
This work focuses on enhancing the CoT capabilities of VLMs to serve complex vision-language reasoning tasks. The proposed approach includes generating synthetic data, SFT for basic CoT capability, and DPO for further calibration of complex reasoning ability stages.

The method is overall reasonable. With results on 9 datasets, the authors demonstrate the effectiveness of the proposed method. The experiments are clear and the analysis is comprehensive. Overall, this work positively contributes to the applications that involve VLM reasoning.

strengths:
1. The overall design makes sense. As demonstrated by the benchmark results, this could be a possible choice for implementing vision-language applications.
2. The release of synthetic data generated by GPT-4o contributes to the VLM finetuning.
3. The failure analysis in this paper is insightful and potentially helpful for other VLM research.

weaknesses:
The novelty is unclear.
- From the idea level, tuning for CoT ability is being actively explored in LLM-centric research (e.g., [1], [2], [3])
- For the overall design, the proposed method has recently been widely used. Including,
   - Step 1 directly leverages the commonly seen approach, i.e., knowledge distillation from a larger teacher model (GPT-4). (e.g., [4])
   - Two-stage tuning (SFT-RL) has been shown to be effective in improving reasoning ability in many LLM works, either through 
       - Pertaining (e.g., Llama-3) 
       - SFT on synthetic CoT data, then use RL to refine the LLM's reasoning abilities (e.g., [5]) 

Besides, one minor problem is missing related work in language-based reasoning. 
Specifically, "How does LLM research tackle the author-targeted challenges in pure language tasks?" This background discussion would be beneficial because this paper particularly studies the Vision-Language Model, which utilizes a vision encoder to project visual information onto language space; thus, it may share certain characteristics with its underlying LLM.

Reference:

[1] Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models. https://arxiv.org/pdf/2312.06585

[2] LogiCoT: Logical Chain-of-Thought Instruction Tuning. https://arxiv.org/pdf/2305.12147

[3] RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold: https://arxiv.org/pdf/2406.14532

[4] LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents. https://arxiv.org/pdf/2311.05437

[5] Tackling Vision Language Tasks Through Learning Inner Monologues. https://arxiv.org/pdf/2308.09970

questions:
Given the concerns raised in the Weakness section, more discussion on each point summarized in lines 88-92 would help to showcase the novelty and contributions of the work. 

To be more clear, lines 88-92 claim the contribution is to (A) provide a GPT-generated dataset, (B) verify that this dataset can be used for SFT, and (C) use DPO after SFT through an approach similar to [3] is meaningful. This makes me feel that this work is more of a resource paper. I would like to learn more about the technical contributions.

Reference:

[3] RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold: https://arxiv.org/pdf/2406.14532
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518902668

thread_type:
official_review

<thread_object>
creator: reviewer_yp44

<content>
summary:
The paper focuses on enhancing chain-of-thought (CoT) reasoning in vision-language models (VLMs) to improve their interpretability and performance on complex tasks. The authors identify that current training methods often rely on datasets with short annotations and limited rationales, which restrict the models' ability to generalize to tasks requiring detailed reasoning.

To address these limitations, the authors propose a two-fold approach:

1. Data Distillation: They distill rationales from the GPT-4o model to enrich the training data, creating a comprehensive CoT dataset with 193,000 examples across various visual question answering (VQA) tasks.

2. Reinforcement Learning (RL): They apply reinforcement learning, specifically the Direct Preference Optimization (DPO) algorithm, to refine the reasoning abilities of the models by constructing positive (correct) and negative (incorrect) reasoning pairs based on model-generated outputs.

strengths:
A clear two-stage approach is proposed to improve the chain-of-thought reasoning and the final performance of vision-language models. The overall research goal and method are easy to understand.

weaknesses:
1. The novelty in the proposed two-stage approach is quite limited, and similar strategies have been widely adopted in previous work. For the first stage that distills CoT reasoning chain from the teacher models GPT-4o, this kind of method has been implemented too many of times in different domains, different modality, and different purposes. For the second stage that prompt VLM to first generate the CoT solution and then check the correctness of the final answers by comparing with the ground-truth ones, it's first proposed in STaR [1] and has been widely extended in many following work. Although the authors leverage the DPO for further optimization, it is far from a shinning point to make the whole method novel. 
2. I doubt the scalability of the proposed method. For the first stage, it basically relies on a teacher model to provide supervision. For the second stage, it needs ground-truth annotation from existing supervised datasets, which makes the method less suitable for more open-domain and complex problems. 
3. The results in Table 2 and Table 6 are not impressive. The improvement is little and hard to tell whether it's because of distilling more GPT data than previous work. 


[1] STaR: Bootstrapping Reasoning With Reasoning. Eric Zelikman et al

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730601577129

thread_type:
official_review

<thread_object>
creator: reviewer_thp5

<content>
summary:
This paper presents a method for training vision language models (VLMs) to output a chain of thought (CoT) before answering a question. The approach involves creating a dataset with CoTs generated by GPT-4o, finetuning an open-weight VLM on these chains of thought (and the associated answers), and further training the VLM using DPO, where the preferences come from the GPT-generated annotations. The first experiments evaluate the model trained by supervised finetuning on the CoT dataset to several baselines--the model trained w/o CoT, ablated versions where certain parts of the data are removed, and models from prior work. The later experiments evaluate the impact of the DPO procedure and compare it to prior work on applying DPO to VLMs.

strengths:
- The dataset of CoTs is likely to be useful in future work
- The results show that finetuning on CoTs is helpful in several tasks
- The results show marginal improvements from the DPO process
- There are several ablations on dataset composition and methods of training/evaluating the model, which are helpful

weaknesses:
- The improvements from DPO (in Tables 6, 7 and Figure 7) are quite small. (Are they statistically significant?)
- The related work section does not discuss the mentioned work in enough detail (and could probably benefit from inclusion of more related work). For instance, what are the differences between the method in this paper and methods in the prior work that finetunes VLMs on CoTs? The statement that "Shao et al. (2024) trains VLMs for chain-of-thought (CoT) reasoning in object localization tasks" does not seem entirely accurate; their CoTs may involve regions in the image, but they apply their models to VQA tasks.

questions:
- What does format (1) "format alignment only" mean? How does it differ from (2) "direct responses only"? Does (1) involve finetuning on the same data as the other models?
- Similar to the last question, in Tables 3 and 4, which datasets are used to train the "format (1)" model?
- This method is bottlenecked by the performance of the teacher model (in this case, GPT-4o). In this sense, it is a distillation method rather than a method that could be used to advance the performance of any model (e.g. a frontier model).
</content>
</thread_object>

</thread>


Metareview: